Users can use Filter-Forward Message Router to efficiently read messages from multiple input streams and apply their own custom rules to filter and forward messages. This customizable pipeline template reads messages from one or more input streams and applies user-defined business rules to filter or forward the incoming messages. There are three types of filters available (in the order applied):
The configuration is loaded once on start up, it will not be updated while the pipeline is running.
This pipeline template expects the user to provide one or more catalogs with streaming layers as input. The pipeline-config.conf file in config folder must be updated with the input catalog HRNs. The pipeline template expects the config/layer.conf file to be updated with the appropriate input layer(s). The user must also provide a versioned layer where the user-defined rules are uploaded, this can be in any catalog the group has access to. The pipeline template also expects the user to provide an output catalog with the layers referenced in the uploaded configuration.
The user-defined rules can define a size filter, a schema filter and a content filter. The content filter expressions are in Java Expression Language. See the config/filter-config.conf file for an example configuration.
In order to deploy and run this pipeline template, you will need the Wizard Deployer. The Wizard executes interactively, asking questions about the application, and expects the user to provide needed answers.
Follow the Wizard's documentation instructions and set up the needed parameters, then follow these steps:
In Platform Portal select the Pipelines tab where you can see your pipeline deployed and running. Once your pipeline has reached a Running state, it should start applying the user-defined rules and forwarding the input messages to the appropriate streams. You can retrieve the output data by using one of the following options:
Executing this pipeline template will incur the following costs:
Cost will depend on the configuration of the output Stream layer selected by the user, such as Throughput IN, Throughput OUT, and TTL.
Cost will depend on amount of:
Cost will depend on the frequency configuration selected by the user. If high throughput is needed, more workers will be used for deployment.
Cost will depend on the log level set for the execution of this pipeline template. To minimize this cost, the user can set log level to WARN.
If you need support with this pipeline template, please contact us.