Managing Pipelines

Topics

• List of Pipelines
• Upgrading a Pipeline Version
• When a Pipeline Version Fails to Run
• Viewing Information About a Pipeline
  • View Pipeline Version Details
  • View Operations History
  • Share Access to a Pipeline
  • Copy a Pipeline Version
  • Copy a Pipeline Version as a New Pipeline
  • View Job History
  • Delete a Pipeline

List of Pipelines

Using the CLI or API, you can get a list of pipelines showing all of the pipelines that you have access to, their ID, and other relevant metadata. To do this in the UI, navigate to the Pipelines tab and find a list of pipelines in their various states, with a summary of relevant metadata for each. You can sort, filter, and search this list. An example is shown here.

To get further information on any of these Pipeline Versions, click on the pipeline name.

↑ Top

Upgrading a Pipeline Version

To upgrade a pipeline version, you replace it with a new pipeline version. The process for doing this using the CLI is described in the discussion of the Pipeline Lifecycle and applies to both batch and stream pipelines. Doing an upgrade from the platform portal is easier.

In the example shown in Figure 2, you can see four variants of the same Compute Bus Delays batch Pipeline Version. The goal here is to upgrade the Running Pipeline Version 3 to the Ready Version 4. Click on the Upgrade button for Version 4. This initiates the upgrade process that, when concluded, allows the next scheduled job to run with Pipeline Version 4.

The example in Figure 3 is a variation on the scenario in Figure 2. In this case, the goal is to upgrade the Running batch Pipeline Version 3 of Compute Bus Delays to the Scheduled Version 4. As in the previous case, clicking on the Upgrade button for Version 4 initiates the upgrade process. But the Upgrade button is not visible for V4. That is because V4 is in a Scheduled mode. You will have to use the Deactivate button to make the Upgrade function is available. Then, the Upgrade can be initiated. Another difference here is that Version 3 is running a batch job, so it will continue to run to the completion of that job. At that point, Version 3 is replaced by Version 4 and the next scheduled job runs with Pipeline Version 4. The alert message at the top of the screen confirms this action.

In the next example (Figures 4 to 7), the goal is to upgrade the Running Stream Pipeline Version 3 of Compute Bus Delays to Version 4. Here are the steps to follow:

Step 1 - Pause the Running Pipeline Version by clicking on its Pause button

Step 2 - The Stream Processing is Paused

Step 3 - Click on the Upgrade button to start the upgrade

Step 4 - Upgrade Confirmed

When the running pipeline version was paused, the state of the current job was saved. Then, when the upgrade was complete, the job picked up where it left off. To avoid this behavior and start a new job, Cancel the job instead of pausing it and proceed as shown above.

↑ Top

When a Pipeline Version Fails to Run

If there is a problem and a Pipeline Version fails to run, an alert message, like the one shown below in red, is displayed at the top of the screen.

↑ Top

Viewing Information About a Pipeline

The Admin Menu provides access to several management functions. Select a specific Pipeline Version from the versions list. Then, click on the ellipses at the far right side of that pipeline as shown below to open the menu.

View Pipeline Version Details

To view all of the information about a specific Pipeline Version, open its Admin Menu and select View Version Details. A modal information window opens that shows you all of the pipeline version configuration parameters, as well as any relevant meta information. Figure 11 shows an example of this window for a batch pipeline and Figure 12 shows an example for a stream pipeline.

↑ Top

View Operations History

Click View operations on the Admin Menu and the system will display a list of historical operations executed by this pipeline version.

↑ Top

Share Access to a Pipeline

BYO data sent to HERE Workspace is private by default. All data sharing requires your explicit action, regardless of which interface you are working with.

Furthermore, PLM has both pipelines and pipeline templates. Both resources must be created either in a group or project, and cannot be in a group and a project at the same time. The following access rules apply to groups and projects:

• If the pipelines and templates are created in a group, all members of the group have full access to all pipeline and pipeline template operations. Access to a pipeline automatically grants full access to all pipeline versions under that pipeline.
• Members of the ‘Home’ project of a pipeline and pipeline template have full access to all pipeline and template operations, but only Project Admins can make pipeline templates available to be linked from other projects.
• The only action allowed on linkable pipeline templates is ‘readResource’. Once the pipeline template has been made linkable, it can be linked, unlinked, and so on. Once linked to a project, pipelines in that project can use that pipeline template to create pipeline versions to activate and run them.

Copy a Pipeline Version

You can copy a Pipeline Version. It will be identical except that it will be assigned a new Pipeline Version ID. The procedure is described here:

1. Navigate to a specific Pipeline Version in the versions list.

2. Click on the ellipses button and select Copy as new version from the menu.

   

3. You will then see a Version Configuration Page with all of the parameters from the original Pipeline Version prepopulated. You can edit any parameter on the page, but some parameter combinations may initiate the creation of a new template.

   

↑ Top

Copy a Pipeline Version as a New Pipeline

You can create a new pipeline based on a single existing pipeline version. The procedure is described here:

1. Navigate to a specific Pipeline Version in the versions list.

2. Click on the ellipses button and select Copy as new pipeline from the menu.

   

3. You can then change the name, description, and share group before saving the new pipeline. Click Next when your changes are made.

   

4. A pipeline detail page similar to that when copying a Pipeline Version is displayed with a single version based on the original copied version. You can change any of the configuration items as needed for your new pipeline version. Click Save version when ready to commit your changes.

↑ Top

View Job History

Click the View Jobs menu item for a specific Pipeline Version and the service will display a list of jobs executed by this Pipeline Version.

↑ Top

Delete a Pipeline

It may be necessary to completely remove a pipeline from the pipeline service. This is equivalent to deleting the pipeline instance and everything associated with it (pipeline JAR file, templates, Pipeline Versions). Perform this action as follows.

Step 1: Go to the Pipeline List and select the Pipeline to delete

1. The pipeline must be in the Ready state.
2. Select the pipeline.
3. Click on the More menu.
4. From the More menu, click on Delete Pipeline.

Step 2: Confirm Pipeline Deletion

Step 3: Completion status confirmation is displayed

↑ Top