Workflow Steps
Build Artifacts and Execute Tests Locally
To build artifacts and execute tests locally, follow the steps below. Pre Submit and Post Submit Verification require the first four steps below, but only Post-Submit verification requires Steps 5 and 6.
-
Initialize. Cleans the workspace to start from a clean build.
-
Clone your source code Git repository. Retrieves the code you want to test into the clean workspace.
-
Build the package. Compiles the source code and creates the artifact. All the following testing steps use this artifact. The artifact created in this step must be immutable during the entire process.
-
Run local tests. Tests the artifact as defined for those steps that do not require HERE platform resources.
-
If the package passes the local tests, publish the JAR to your Artifact Repository. Publishes the artifact to an Artifact Repository Manager together with the tests. This example uses Artifactory as the Artifact Repository Manager. This step is executed only in Post Submit Verification.
-
If the package passes the local tests, trigger verification in test environment. Deploys the artifact to the test environment. This step is executed only in Post Submit Verification.
jenkins/step1.groovy
contains a reference implementation for the steps above. To examine the code, see the Build and Test Implementation.
To run the test job in the platform, follow the steps below. This step uses the App-Y (Group-Y) credentials pair.
All artifacts created by this job have their job number included as a suffix in their IDs to prevent resources created by different jobs from conflicting with each other.
-
Initialize. Downloads artifacts, parses deployment file and downloads OLP CLI.
-
Prepare test catalogs. Creates the input and output catalogs used by the pipeline, shares them with Group-Y and generates the file pipeline-config.conf
to configure the pipeline version. The Input catalog is filled with test partitions.
-
Create pipeline. Creates a pipeline, pipeline template and pipeline version necessary to run test in pipelines.
-
Activate pipeline. Activates a pipeline and waits for the pipeline to start running.
-
Wait for pipeline completion. Waits until the pipeline job completes. Fails the test if the pipeline does not complete successfully.
-
Run product acceptance tests. Downloads output partitions from the output catalog and executes product acceptance tests.
-
Deploy to production environment. Triggers the production deployment job.
-
Clean up. Removes test artifacts.
jenkins/step2.groovy
contains a reference implementation for the steps above. To examine the code, see the Validation Implementation.
Deploy to Production
To deploy to production on the platform, follow the steps below.
-
Initialize. Downloads artifacts, parses deployment file and downloads OLP CLI.
-
Create new pipeline version. Creates a new pipeline version.
-
Activate/upgrade pipeline. Upgrades the pipeline to the new pipeline version.
-
Clean up. Removes created artifacts in case of deployment failure.
jenkins/step3.groovy
contains a reference implementation for the steps above. To examine the code, see the Deployment Implementation.
Deployment Descriptions
When a job deploys a pipeline to the platform, the deployment is associated with a set of parameters, with most properties following the parameters of the corresponding CLI commands and their parameters. The example below is for a single deployment.
sdk_version=1.6.1.2
group_id=GROUP-9a42655d-ed42-4a0a-95ae-6f5a428aff4b ////please, specify here your group id
pipeline_name="CICD Auto Sensor Learning Processor P2"
pipeline_description="Reads from Archive layer and writes learnings to output catalog"
pipeline_type=batch-2.0.0
pipeline_template_name="Auto Sensor Learning Processor"
pipeline_template_description="Auto Sensor Learning Processor"
input_catalog_ids=archive-catalog
class_name=com.here.platform.examples.p2.Main
input-catalogs.archive-catalog.method=use
input-catalogs.archive-catalog.hrn=hrn:here:data:::auto-sensor-1-archive
output_catalog.method=use
output_catalog.hrn=hrn:here:data:::auto-sensor-2-learnings
The table below provides the descriptions for these properties.
property | description |
sdk_version | Version of the SDK used to download OLP CLI |
group_id | Group ID used to create pipeline templates and pipelines. Your Jenkins App ID must belong to this group. |
pipeline_name | Name of the pipeline created |
pipeline_description | Description of the pipeline created |
pipeline_type | The type of pipeline - batch-2.0.0 or stream-2.0.0 |
pipeline_template_name | Name of the pipeline template created |
pipeline_template_description | Description of the pipeline template created |
input_catalog_ids | IDs of input catalogs as used in the Data Processing Library |
class_name | Pipeline template entry point |
input_catalogs.*.method | "use" - use existing catalog or "create" - create a catalog inside the job |
input_catalogs.*.hrn | HRN of the catalog when method is "use" |
input_catalogs.*.id | Hint for creating the catalog HRN when method is "create" |
input_catalogs.*.name | Name of the catalog to create |
input_catalogs.*.config | Name of the file with catalog configuration located in deployments/ directory |
output_catalogs.*.method | "use" - use existing catalog or "create" - create a catalog inside the job |
output_catalogs.*.hrn | HRN of the catalog when method is "use" |
output_catalogs.*.id | Hint for creating the catalog HRN when method is "create" |
output_catalogs.*.name | Name of the catalog to create |
output_catalogs.*.config | Name of the file with catalog configuration located in deployments/ directory |