Batch pipelines have a useful tool for monitoring and inspecting batch jobs' execution. The Spark framework includes a Web Console that is active for all Spark jobs in the Running state. It is called the Spark UI and can be accessed directly from within OLP. This article explains how to access the Spark UI and some of its basic functions.
Spark UI is automatically started when a batch pipeline job is initiated. It can be accessed from the OLP Portal or from a terminal.
The best way to reach the Spark UI is through the OLP Portal. Figure 1 shows a typical display for a running batch pipeline. On the right side of the screen you can see two links:
See Log and
Open Spark UI.
Alternately, OLP CLI command
pipeline job show can be used to get the
pipeline UI URL for the Spark UI for a Batch pipeline.
Note: Logging in
If you have not already logged into the Spark UI in your current session, you will be requested to do so using the normal OLP sign-in dialog. Use your normal OLP credentials. Cookies are used to maintain your current session information.
Open Spark UI link in the Portal or the
pipeline UI URL link in the CLI opens the Spark UI in your browser as illustrated in figure 2.
There are 5 tabs that allow access to different categories of information. These include:
- Jobs -- This tab shows the job and its stages and tasks with the current state.
- Stages -- This tab shows details of a selected stage.
- Storage -- This tab shows information about server storage available for the job.
- Environment -- This tab shows run-time information for the job.
- Executors -- This tab shows information about executor status and resource allocations.
Clicking on the linked pipeline job description opens the details page as shown in Figure 3.
The Executors tab also has useful information as shown in Figure 4.
This version of the Spark UI has been modified to make it compatible with OLP. This means that there are some functions that might be available in a native Spark environment that are not available here. When referring to other Spark UI documentation references, you may see discussions of features that are missing here. Those missing features are not compatible with OLP.