Managing Pipelines

Topics

List of Pipelines

Using the CLI or API, you can get a list of pipelines showing all of the pipelines that you have access to, their ID, and other relevant metadata. To do this in the UI, navigate to the Pipelines tab and find a list of pipelines in their various states, with a summary of relevant metadata for each. You can sort, filter, and search this list. An example is shown here.

Screen capture of a typical list of available pipelines under the Pipelines tab.
Figure 1. List of all Available Pipelines

To get further information on any of these Pipeline Versions, click on the pipeline name.

↑ Top

Upgrading a Pipeline Version

To upgrade a pipeline version, you replace it with a new pipeline version. The process for doing this using the CLI is described in the discussion of the Pipeline Lifecycle and applies to both batch and stream pipelines. Doing an upgrade from the platform portal is easier.

In the example shown in Figure 2, you can see four variants of the same Compute Bus Delays batch Pipeline Version. The goal here is to upgrade the Running Pipeline Version 3 to the Ready Version 4. Click on the Upgrade button for Version 4. This initiates the upgrade process that, when concluded, allows the next scheduled job to run with Pipeline Version 4.

screenshot of upgrading a scheduled pipeline version
Figure 2. Upgrade a Scheduled Pipeline Version

The example in Figure 3 is a variation on the scenario in Figure 2. In this case, the goal is to upgrade the Running batch Pipeline Version 3 of Compute Bus Delays to the Scheduled Version 4. As in the previous case, clicking on the Upgrade button for Version 4 initiates the upgrade process. But the Upgrade button is not visible for V4. That is because V4 is in a Scheduled mode. You will have to use the Deactivate button to make the Upgrade function is available. Then, the Upgrade can be initiated. Another difference here is that Version 3 is running a batch job, so it will continue to run to the completion of that job. At that point, Version 3 is replaced by Version 4 and the next scheduled job runs with Pipeline Version 4. The alert message at the top of the screen confirms this action.

screenshot of upgrading a running pipeline version
Figure 3. Upgrade a Running Batch Pipeline Version

Hint: New Pipeline Version Required

Before you can upgrade an existing Pipeline Version, you must have a new Pipeline Version to replace it with. You can create this from the same pipeline JAR file, but you may want to create a new Template with modified parameter values.

In the next example (Figures 4 to 7), the goal is to upgrade the Running Stream Pipeline Version 3 of Compute Bus Delays to Version 4. Here are the steps to follow:

Step 1 - Pause the Running Pipeline Version by clicking on its Pause button

screenshot of pausing the running streamed pipeline version to upgrade it
Figure 4. Pause a Running Streaming Pipeline Version

Note

Although the upgrade process applies to both batch and stream pipelines, the behavior is somewhat different. When you pause a pipeline, a stream pipeline responds quickly and a batch pipeline can take some time to pause. This is because the batch pipeline only pauses when the currently running job is done. Thus, the delay to pause is primarily a function of the size and completeness of the currently running job. The stream pipeline just saves its location in the current job so it can resume at that point.

Step 2 - The Stream Processing is Paused

screenshot of the running streamed pipeline version being paused
Figure 5. Upgrade a Running Streaming Pipeline Version

Step 3 - Click on the Upgrade button to start the upgrade

screenshot of the running streamed pipeline version upgrade submission
Figure 6. Upgrade a Running Streaming Pipeline Version

Step 4 - Upgrade Confirmed

screenshot of the completed running streamed pipeline version upgrade
Figure 7. Upgrade a Running Streaming Pipeline Version

When the running pipeline version was paused, the state of the current job was saved. Then, when the upgrade was complete, the job picked up where it left off. To avoid this behavior and start a new job, Cancel the job instead of pausing it and proceed as shown above.

↑ Top

When a Pipeline Version Fails to Run

If there is a problem and a Pipeline Version fails to run, an alert message, like the one shown below in red, is displayed at the top of the screen.

screenshot of failure to run alert message
Figure 8. Failure to Run Alert

Hint

For additional information, see the Logs, Monitoring and Alerts User Guide .

↑ Top

Viewing Information About a Pipeline

The Admin Menu provides access to several management functions. Select a specific Pipeline Version from the versions list. Then, click on the ellipses at the far right side of that pipeline as shown below to open the menu.

Screenshot of pipeline version admin menu
Figure 9. Pipeline Version Admin Menu

View Pipeline Version Details

To view all of the information about a specific Pipeline Version, open its Admin Menu and select View Version Details. A modal information window opens that shows you all of the pipeline version configuration parameters, as well as any relevant meta information. Figure 11 shows an example of this window for a batch pipeline and Figure 12 shows an example for a stream pipeline.

screenshot of a batch pipeline version details displays
Figure 10. Batch Pipeline Version Details Display

Hint

Dismiss the display window by clicking on the "X" in the top right corner.

screenshot of a streaming pipeline version detail display
Figure 11. Streaming Pipeline Version Details Display

↑ Top

View Operations History

Click View operations on the Admin Menu and the system will display a list of historical operations executed by this pipeline version.

Hint

Dismiss the display window by clicking on the "X" in the top right corner.

screenshot of a job history for a selected pipeline version
Figure 12. View Job Details for a Pipeline Version

↑ Top

Copy a Pipeline Version

You can copy a Pipeline Version. It will be identical except that it will be assigned a new Pipeline Version ID. The procedure is described here:

  1. Navigate to a specific Pipeline Version in the versions list.
  2. Click on the ellipses button and select Copy as new version from the menu.

    Screenshot of Copy new version menu.
    Figure 13. Copy New Pipeline Version
  3. You will then see a Version Configuration Page with all of the parameters from the original Pipeline Version prepopulated. You can edit any parameter on the page, but some parameter combinations may initiate the creation of a new template.

    Screenshot of the Version Configuration Page
    Figure 14. Pipeline Version Configuration Page

↑ Top

Copy a Pipeline Version as a New Pipeline

You can create a new pipeline based on a single existing pipeline version. The procedure is described here:

  1. Navigate to a specific Pipeline Version in the versions list.

  2. Click on the ellipses button and select Copy as new pipeline from the menu.

    Screenshot of Copy as new pipeline menu.
    Figure 15. Copy as New Pipeline
  3. You can then change the name, description, and share group before saving the new pipeline. Click Next when your changes are made.

    Screen capture of new pipeline description dialog.
    Figure 16. Describe New Pipeline
  4. A pipeline detail page similar to that when copying a Pipeline Version is displayed with a single version based on the original copied version. You can change any of the configuration items as needed for your new pipeline version. Click Save version when ready to commit your changes.

↑ Top

View Job History

Click the View Jobs menu item for a specific Pipeline Version and the service will display a list of jobs executed by this Pipeline Version.

Hint

Dismiss the display by clicking on the "X" in the top right corner.

screenshot of a job history for a selected pipeline version
Figure 17. View Job Details for a Pipeline Version

↑ Top

Delete a Pipeline

It may be necessary to completely remove a pipeline from the pipeline service. This is equivalent to deleting the pipeline instance and everything associated with it (pipeline JAR file, templates, Pipeline Versions). This action is performed as follows.

Note: Catalogs Not Included

This procedure does not affect any Catalogs loaded into the pipeline service.

Step 1: Go to the Pipeline List and select the Pipeline to delete

  1. The pipeline must be in the Ready state.
  2. Select the pipeline.
  3. Click on the More menu.
screenshot of pipeline list screen
Figure 18. Select Pipeline to Delete

Step 2: From the More menu, click on Delete Pipeline

screen capture of more menu
Figure 19. Open More Menu

Step 3: Confirm Pipeline Deletion

Warning: Irrevocable Action

This action cannot be reversed. The entire pipeline instance and everything associated with it will be removed.

screenshot of delete pipeline confirmation dialog
Figure 20. Confirm or Cancel Pipeline Deletion

Step 4: Upon completion, a status confirmation will be displayed

screenshot of pipeline delete completion message**
Figure 21. Message confirming Pipeline Deletion

↑ Top

results matching ""

    No results matching ""