Get data and partition metadata from a stream layer

You can read messages from a stream layer if you subscribe to the layer. The messages contain data and the following partition metadata:

  • Data handle
  • ID
  • Data size
  • Compressed data size
  • Checksum
  • Timestamp

To get data and partition metadata from the stream layer:

  1. Get an access key ID and access key secret.

    For instructions, see Authenticate to the HERE platform using client credentials.

  2. Create the OlpClientSettings instance.

    For instructions, see Create platform client settings.

  3. Create the StreamLayerClient object with the HERE Resource Name (HRN) of the catalog that contains the layer, layer ID, and platform client settings from step 2.

    const streamLayerClient = new StreamLayerClient({
      catalogHrn: HRN.fromString("your-catalog-hrn"),
      layerId: "your-layer-id",
      settings: olpClientSettings
    });
    
  4. Create the SubscribeRequest object with the serial or parallel subscription type.

    • If your application should read smaller volumes of data using a single subscription, use the serial subscription type.

      const subscribeRequest = new SubscribeRequest().withMode("serial");
      
    • If your application should read large volumes of data in a parallel manner, use the parallel subscription type and subscription ID.

      const subscribeRequest = new SubscribeRequest()
        .withMode("parallel")
        .withSubscriptionId("your-subscription-id");
      
  5. Call the subscribe method with the subscribeRequest parameter.

    const subscribtionId = await streamLayerClient.subscribe(subscribeRequest);
    

    You receive a subscription ID from the requested subscription to the selected layer.

  6. Call the poll method with the subscription ID.

    const messages = await streamLayerClient.poll(
      new PollRequest().withMode("serial").withSubscriptionId(subscribtionId)
    );
    

    You get messages with the layer data and partition metadata. The poll method also commits the offsets, so you can continue polling new messages.

    Example:

    {
    "messages": [
        {
        "metaData": {
            "partition": "314010583",
            "checksum": "ff7494d6f17da702862e550c907c0a91",
            "compressedDataSize": 152417,
            "dataSize": 250110,
            "data": "iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAABhQTFRFvb29AACEAP8AhIKEPb5x2m9E5413aFQirhRuvAMqCw+6kE2BVsa8miQaYSKyshxFvhqdzKx8UsPYk9gDEcY1ghZXcPbENtax8g5T+3zHYufF1Lf9HdIZBfNEiKAAAAAElFTkSuQmCC",
            "dataHandle": "",
            "timestamp": 1517916706
        },
        "offset": {
            "partition": 7,
            "offset": 38562
        }
        }
    ]
    }
    

    If the data size is less than 1 MB, the data field is populated. If the data size is greater than 1 MB, you get a data handle that points to the object stored in the blob store.

  7. If the data size is greater than 1 MB, call the getData method with the Messages object.

    const data = await streamLayerClient.getData({
      metaData: {
        partition: "314010583",
        checksum: "ff7494d6f17da702862e550c907c0a91",
        compressedDataSize: 152417,
        dataSize: 250110,
        data:
          "iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAABGdBTUEAALGPC/xhBQAAABhQTFRFvb29AACEAP8AhIKEPb5x2m9E5413aFQirhRuvAMqCw+6kE2BVsa8miQaYSKyshxFvhqdzKx8UsPYk9gDEcY1ghZXcPbENtax8g5T+3zHYufF1Lf9HdIZBfNEiKAAAAAElFTkSuQmCC",
        dataHandle: "",
        timestamp: 1517916706
      },
      offset: {
        partition: 7,
        offset: 38562
      }
    });
    

You get data from the requested partition.

results matching ""

    No results matching ""