Pipeline Patterns

HERE platform pipelines are designed to accommodate specific usage patterns. The available patterns are illustrated below, starting with the simplest pattern and progressing to more complex use cases. Additional information is provided throughout the Developer Guide.

General Pattern

This is the general pattern for using a pipeline.

general pipeline pattern
Figure 1. Basic pipeline pattern

Note: Data sources and sinks

A specific catalog layer can serve as a source or a sink, but never both at once. The type of catalog layer that may be used depends on the type of pipeline it is used with. For example, a streamed layer cannot be used with a batch pipeline.

Multiple Inputs

You can have multiple inputs, but only one output from a pipeline.

multiple input pattern
Figure 2. Multiple inputs pattern

Stream Processing Pattern

You can use the pipeline to process continuous data streams using Apache Flink. [stream-to-stream]

stream processing pattern
Figure 3. Stream processing pattern
  • The data catalog is defined in the configuration files.
  • The layer used is defined in the code.

Note

You may use the same data catalog for a stream pipeline's input and output as long as separate layers are being used for the data source and data sink.

Batch Processing Pattern

This is a typical batch processing pattern using Apache Spark. [versioned-to-versioned]

Batch Processing Pattern
Figure 4. Batch processing pattern

Volatile Pattern

This is a typical pattern using volatile layers.

Volatile Pattern
Figure 5. Volatile pattern

Index Pattern

These are typical patterns using index layers.

index pattern 1
Figure 6. Index Layer Sink 1
index pattern 2
Figure 7. Index Layer Sink 2

Note: Index Layer Limits of Use

PIPELINE Source SINK
Batch Yes Yes
Stream No Yes

Advanced Patterns

A more advanced pattern uses a catalog's volatile layer as reference data.

In this case, the output catalog uses a stream layer.

Advanced Patterns
Figure 8. Advanced pattern

Info

The stream layer here typically uses a windowing function.

But in this case, the output catalog is only interested in a "data snapshot," so the volatile layer is used.

using a volatile layer
Figure 9. Volatile layer pattern

Alternatively, you can use the output catalog's versioned layer, perhaps for aggregating data over a window of time. This approach could also be useful for archiving data, with or without processing enhancement. Also, it could be useful for historical analysis in a Notebook.

using a versioned layer
Figure 10. Versioned layer pattern

Or, you can use the output catalog's Index layer, perhaps for organizing historical data by event time.

advanced index layer pattern
Figure 11. Index Layer Pattern 1

Another pattern combines input data from a versioned data set with data from an index layer.

advanced index layer pattern 2
Figure 12. Index Layer Pattern 2

For examples of pipeline implementation, see GitHub.
For detailed step-by-step instructions on configuring and running pipelines, see Developer Tutorials.

results matching ""

    No results matching ""