Filter-Forward Message Router

Overview

Users can use Filter-Forward Message Router to efficiently read messages from multiple input streams and apply their own custom rules to filter and forward messages. This customizable pipeline template reads messages from one or more input streams and applies user-defined business rules to filter or forward the incoming messages. There are three types of filters available (in the order applied):

  1. Size Filter: Messages over a specified threshold will be shunted to an output stream layer and no further filtering applied. Messages under the threshold will go through any additional filters.
  2. Schema Filter: Messages of various supported schema types will be forwarded to different outputs (a streaming layer or a following content filter route).
  3. Content Filter: Messages matching the user-defined rules will be forwarded to the configured output layer(s). This filter requires the schema filter to determine message type beforehand. Only message types that are configured in the schema filter will have the content filter applied (if configured).

The configuration is loaded once on start up, it will not be updated while the pipeline is running.

Structure

Application Flow Diagram
Figure 1. Application Flow Diagram

Legend: Legend Diagram

Prerequisites

This pipeline template expects the user to provide one or more catalogs with streaming layers as input. The pipeline-config.conf file in config folder must be updated with the input catalog HRNs. The pipeline template expects the config/layer.conf file to be updated with the appropriate input layer(s). The user must also provide a versioned layer where the user-defined rules are uploaded, this can be in any catalog the group has access to. The pipeline template also expects the user to provide an output catalog with the layers referenced in the uploaded configuration.

The user-defined rules can define a size filter, a schema filter and a content filter. The content filter expressions are in Java Expression Language. See the config/filter-config.conf file for an example configuration.

Execution

In order to deploy and run this pipeline template, you will need the Wizard Deployer. The Wizard executes interactively, asking questions about the application, and expects the user to provide needed answers.

Follow the Wizard's documentation instructions and set up the needed parameters, then follow these steps:

  1. Execute the script as ./wizard.sh
  2. Follow the prompts and provide needed answers

Verification

In Platform Portal select the Pipelines tab where you can see your pipeline deployed and running. Once your pipeline has reached a Running state, it should start applying the user-defined rules and forwarding the input messages to the appropriate streams. You can retrieve the output data by using one of the following options:

Cost Estimation

Executing this pipeline template will incur the following costs:

Storage-Stream and Stream TTL

Cost will depend on the configuration of the output Stream layer selected by the user, such as Throughput IN, Throughput OUT, and TTL.

Data Transfer IO

Cost will depend on amount of:

  • user-defined rules read from a single partition in a Versioned layer on a start-up of this pipeline template (done once per deployment)
  • amount of data published to the output Stream layer(s)
Compute Core and Compute RAM

Cost will depend on the frequency configuration selected by the user. If high throughput is needed, more workers will be used for deployment.

Log Search IO

Cost will depend on the log level set for the execution of this pipeline template. To minimize this cost, the user can set log level to WARN.

Support

If you need support with this pipeline template, please contact us.

results matching ""

    No results matching ""