local catalog layer partition

The OLP CLI supports the following:

• list partitions in a local layer
• get partitions from a local layer
• put partitions in a local layer
• delete partitions in a local layer

local catalog layer partition list

Lists the names of all partitions in the specified catalog layer; either versioned, volatile, or index.

olp local catalog layer partition list <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

• --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted.

  Note

  --filter <filter query> is optional for versioned and volatile layers, where it is used as a prefix to filter partitions by partition name.

Optional parameters:

• --json Displays the command result in JSON format.
• --quiet Displays partition names, each on a new line.

Optional parameters for a versioned layer:

• --version <catalog version> The catalog's metadata version.

Optional parameters for a volatile layer:

• --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.

Example:

The command below lists all partitions from layer1 in the catalog catalog1.

olp local catalog layer partition list hrn:local:data:::catalog1 layer1

local catalog layer partition get

Downloads the partitions that are specified with the --partitions parameter in the local catalog layer.

olp local catalog layer partition get <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer to use; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

Required parameters for an index layer:

• --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted. This parameter is optional for versioned and volatile layers, where it is used as a prefix to filter the partitions by partition name.

Optional parameters:

• --partitions <partitionID1 partitionID2 ...> The partitions to download. If only one partition is specified and no output folder is provided, the partition content is retrieved into the standard output. If the partition content is retrieved into the standard output, the --modified-since parameter is used, and there are no modifications after the date provided, then the command execution finishes with the exit code 4.
• --all Downloads all partitions in the layer.
• --output <path to dir> The output directory to store the downloaded partitions. If the specified directory doesn't exist, it is automatically created. If no directory is specified, the default value is <current dir>/<catalog HRN>.
• --quiet Displays the absolute path to the output directory.

Optional parameters for a versioned layer:

• --version <catalog version> The catalog's metadata version.
• --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.

Optional parameters for a volatile layer:

• --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss, but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.
• --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.

Example:

The command below retrieves the partition content into the standard output if you specify only one partition.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1 partition2

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1 partition2

Example:

The command below downloads partition1 from layer1 in catalog1.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1 > output.txt

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1 > output.txt

Example:

The command below downloads partition1 and partition2 from version 1 of versioned-layer in catalog1 to the specified output directory.

olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer \
 --partitions partition1 partition2 --output <path_for_partitions> --version 1

olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer ^
--partitions partition1 partition2 --output <path_for_partitions> --version 1

Example:

To customize the path where the partition is saved, concatenate the relative or the absolute path with the partition name, using a colon (:).

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition_name:path/to/partition

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition_name:path/to/partition

Example:

The command below retrieves all partitions from the index layer that were uploaded on January 1, 2020, between 00:01:00 AM and 00:05:00 AM with locationTag equal to 4242.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output <path_for_partitions>

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output <path_for_partitions>

local catalog layer partition put

Publishes the specified files to the specified local partitions.

olp local catalog layer partition put <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer to upload partition(s) to; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

Required parameters for an index layer:

• --index-fields <bool|int|string|heretile|timewindow>:<index-key>:<index-value> A list of index fields for this publication. The <timewindow> index field definition is required and should be the same as was specified in the layer's index definitions.

  If the --index-fields parameter is not passed and the --input parameter is passed, the application tries to read the index configuration from the index.properties file located in the --input path. The format of the index.properties file is the following:

  • fileMask A regular expression used to filter files in the input directory.
  • indexType An index type name with these acceptable values: bool, int, string, heretile, timewindow. You must define timewindow that should be the same as the one specified in the layer index-definitions.
  • indexKey An index name. This key is used for listing index partitions.
  • indexValue The value of the index. The value of timewindow should be the same as the one specified in the layer index-definitions.

  The following example sets the timewindow index myIndex to 50000 for all the files whose names start with file and one digit like file1.bin, file2.txt, and the like. Add the boolean index key myBoolIndex set to true to the file1.bin file.

  fileMask=file[0-9].*
  indexType=timewindow
  indexKey=myIndex
  indexValue=50000
  
  fileMask=file1.bin
  indexType=bool
  indexKey=myBoolIndex
  indexValue=true
  

  An empty line between index definitions is not required but it improves the readability of the config. The values for all configuration keys (fileMask, indexType, indexKey, indexValue) are required.

  If both of the --index-fields and --input parameters are passed, then the configurations from index.properties are ignored.

Optional parameters:

• --partitions <partition ID>:<path to file> The partitions and files to upload.
• --input <path to dir> The input directory from which to upload the files; supports relative paths; hidden files are skipped.
• --dependency <catalog-hrn:version[:direct]> The list of the dependencies for this publication. direct is optional, and its default value is true.
• --quiet Displays empty output with no additional information.

Optional parameters for a versioned layer:

• --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Optional parameters for a volatile layer:

• --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Optional parameters for an index layer:

• --metadata <key1=val1 key2=val2 ...> List of all user-defined fields that can store extra information. If the --input parameter is provided, the metadata applies to all the files located in the input directory.

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively from the folder relative to the current folder.

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:path/to/file1 partition2:path/to/file2

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:path/to/file1 partition2:path/to/file2

If you specify only one partition, the command retrieves the partition content into the standard output. For more information, see olp local catalog layer partition get.

OLP CLI lets you chain both get and put commands when you retrieve content into the standard output. This means that you can omit the path to the partition and use a hyphen (-) instead.

Example:

olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 \
 --partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 \
 versioned-layer1 --partitions partition1:-

olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 ^
--partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 ^
versioned-layer1 --partitions partition1:-

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively to the index layer with the --index-fields parameter equal to timewindow:ingestionTime:1577836960 int:locationTag:4242

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:<path/to/file1> partition2:<path/to/file2>
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:path/to/file1 partition2:path/to/file2
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

local catalog layer partition delete

Deletes partitions from an index layer by the --filter query. Note that this command does not support data deletion from versioned and volatile layers.

olp local catalog layer partition delete <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer to delete partitions from.
• --filter <filter query> The query string required to query data that should be deleted from the index layer. The <filter query> value must be written in the Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside <filter query> should be escaped or quoted.

Optional parameters:

• --timeout <duration in seconds> The timeout duration of the delete request in seconds. If not specified, the default is 3600.
• --quiet Displays empty output with no additional information.

Example:

The command below deletes partitions with keys testIndex==3600000 and env==4242.

olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 \
 --filter "testIndex==3600000;env==4242"

olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 ^
--filter "testIndex==3600000;env==4242"