Connecting Pipelines to Third-Party Services

This section presents the method of connecting a Pipeline to a third-party service using the credentials for that service. These credentials are saved as Secrets within the HERE platform. HERE Platform supports two types of third-party credentials:

1. AWS
2. Custom

AWS credentials are located at a file location readable to pipeline code. The location of the file can be read from environment variable AWS_SHARED_CREDENTIALS_FILE in the pipeline code. Custom credentials are located at the default path: /dev/shm/identity/.here/credentials and this file is readable from the pipeline code. To keep the pipelines functional, the third-party credentials are automatically refreshed every 12 hours. If the credentials have changed and need to be consumed immediately, the pipeline will need to be manually re-activated.

The example below shows the way of connecting a Pipeline to AWS S3. Similar approach can be used for other services and credentials.

Here are the steps required to connect a Pipeline to AWS S3:

1. Identify the location of AWS credentials. These are generally found in ~/.aws/.

   Note

   The AWS credentials need to be of the AWS Key/Secret form. (AWS IAM role is not supported at this time). Contact your AWS admin/manager to create the key/secret and setup the access. To reduce the security risk, it is recommended to grant minimal privileges to this new identity.

2. Identify the location of HERE credentials to be used for connecting the OLP CLI to the HERE platform. (Setup HERE Credentials)
3. Use the OLP CLI's create secret command with the --grant-read-to and --credentials parameters, to create a new third-party secret for AWS and to grant read permission for the secret to the HRN of the HERE App or User that will be used to run the Pipeline.
4. Within the Pipeline code, create a new client for S3:

    S3Client s3client = S3Client.builder()
    .build();
   

5. The S3Client class is provided by AWS Java SDK. These steps were tested with below AWS Java SDK dependency. AWS SDK will load the credentials from the file located at AWS_SHARED_CREDENTIALS_FILE set by platform.

    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <version>2.17.90</version>
    </dependency>
   

6. (Optional) To list the contents of an S3 bucket, include the following within the Pipeline code:

    for (ListIterator iter = objects.listIterator();iter.hasNext();) {
        S3Object s3Object = (S3Object) iter.next();
        LOGGER.info("Aws object key is {}", s3Object.key());
        count++;
    }
   

7. Create a new Pipeline Version with the new JAR file that contains the code to connect to and use AWS S3.
8. Activate the new Pipeline Version using the HERE APP that was used while creating the third-party secret in Step #3 above.

See Also

• Manage Third-Party Secrets