Stream

The OLP CLI supports the following:

  • get partitions from a local stream layer
  • put partitions to a local stream layer

local catalog layer stream get

Subscribes to the specified local stream layer and downloads or prints every partition that is uploaded to the layer during the subscription period.

olp local catalog layer stream get <catalog HRN> <layer ID> [command options]

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer.

Optional parameters:

  • --output <path to dir> The output directory to store data received from the stream. Files are created with the names of partitions received from the stream. If the specified directory doesn't exist, it is automatically created. If no directory is specified, partitions from the stream are printed to the standard output separated by a delimiter.
  • --delimiter <character> A delimiter used for printing stream data to the standard output; empty by default.
  • --limit <max number of paritions> The maximum number of partitions to be downloaded.
  • --timeout <duration in seconds> The subscription timeout duration in seconds (infinite by default).
  • --consumer-group <consumer ID> The name of a consumer, or a group if the consumer is distributed.
  • --quiet Displays the absolute path to the output directory, if available. Otherwise, the partitions' contents are printed from the stream to the standard output.

Example:

The command below prints all partitions from stream-layer1 in the catalog1 using the default empty delimiter.


olp local catalog layer stream get hrn:local:data:::catalog1 stream-layer1

local catalog layer stream put

Publishes the content from the filesystem into a local catalog's stream layer.

olp local catalog layer stream put <catalog HRN> <layer ID> [command options]

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer.

Optional parameters:

  • --input <path to dir> The input directory from where to read partitions. Relative paths are supported. Partitions are populated by walking the file tree rooted at the given input directory. The file tree is traversed depth-first and in an alphabetical order. If not specified, the partitions are taken from the standard input and split by the delimiter.
  • --delimiter <character> A delimiter used for reading partitions from the standard input.
  • --limit <max number of partitions> The maximum number of partitions to be published.
  • --quiet Displays empty output with no additional information.

Example:

The command below publishes all partitions from /path/to/directory.


olp local catalog layer stream put hrn:local:data:::catalog1 stream-layer1 --input /path/to/directory

results matching ""

    No results matching ""