Select a Streaming Connector: Kafka or HTTP

Stream layers are implemented as Kafka clusters.

To read from or write to a stream layer, your application must use one of the following connector types:

• Direct Kafka is the preferred connector type since it directly communicates with the underlying Kafka cluster. It is the default connector.
• HTTP Connector is an HTTP wrapper on top of Kafka, and therefore implies a communication overhead.

If required, you can provide a different connector type for the producer and the consumer. For example, your application can use Direct Kafka to read from a stream layer and the HTTP connector to write to a different stream layer.

Direct Kafka

To use the Direct Kafka connector type, you must provide the following dependency:

<dependencies>
    <dependency>
        <groupId>com.here.platform.data.client</groupId>
        <artifactId>kafka-support_2.12</artifactId>
        <version>1.18.173</version>
    </dependency>
</dependencies>

libraryDependencies += "com.here.platform.data.client" %% "kafka-support" % "1.18.173"

Note that, if your application uses flink-support, you do not need to add this dependency as the dependency is already included in flink-support. For more details, refer to the Dependencies for Stream Pipelines section.

To enable Direct Kafka support, you must provide the following configuration:

here.platform.data-client.stream.connector {
  producer = "kafka-connector"
  consumer = "kafka-connector"
}

Producers and consumers can be configured independently. For more information, see the Data Client configuration file in the Configuration section.

HTTP Connector

If you cannot use Kafka Direct, for example because your application needs to access data in the Marketplace or is running behind a proxy, use the HTTP connector.

If you cannot use Kafka Direct, for example because your application is running behind a proxy, use the HTTP connector. To use it, the following configuration must be set:

here.platform.data-client.stream.connector {
  producer = "http-connector"
  consumer = "http-connector"
}

Producers and consumers can be configured independently. For more information, see the Data Client configuration file in the Configuration section.