Data Workflows

The HERE platform organizes data according to three key concepts: catalog, layer, and partition. The platform hosts a number of catalogs, each with one or more layers, which are divided into partitions.

For more information on these concepts, see the Data API.

The OLP CLI provides tools for managing data, such as:

  • List all catalogs to which you have access
  • Create a new catalog and layer
  • Upload data to a catalog

For more details, see Data Commands.

List All Catalogs

To list the catalogs to which you have access, enter the catalog list command.

olp catalog list

The command lists the HRNs of the catalogs to which you have access, each in a new line.

Create a New Catalog and Layer

Follow the steps below to create a new catalog and layer.

  1. Enter the catalog create command.

    Linux
    Windows
    olp catalog create first-catalog-example-id first-catalog-example \
       --summary "This catalog contains a sample layer containing data in CSV format"
    olp catalog create first-catalog-example-id first-catalog-example ^
       --summary "This catalog contains a sample layer containing data in CSV format"
  2. To add a layer to your catalog, enter the catalog layer add command.

    Linux
    Windows
    olp catalog layer add hrn:here:data:::first-catalog-example-id example-layer-id \
       "Example layer" --versioned --summary "This layer contains simple data in CSV format" \
       --description "A longer description of the layer contents" --content-type=text/csv
    olp catalog layer add hrn:here:data:::first-catalog-example-id example-layer-id ^
       "Example layer" --versioned --summary "This layer contains simple data in CSV format" ^
       --description "A longer description of the layer contents" --content-type=text/csv
  3. To verify the catalog configuration, enter the catalog show command.

    
        olp catalog show hrn:here:data:::first-catalog-example-id --json
        

    The command displays the following results.

    
        {
            "summary": "This catalog contains sample layers containing simple text",
            "owner": {
                "creator": {"id": "mzLcb1rL8nskvDQpCFEF"},
                "organisation": {"id": "olp-here"}
            },
            "billingTags": [],
            "hrn": "hrn:here:data:::first-catalog-example-id",
            "created": "2018-03-26T12:48:10.315Z",
            "name": "first-catalog-example",
            "layers": [{
                "summary": "This layer contains simple data in CSV format",
                "volume": {
                     "volumeType": "Durable"
                },
                "layerType": "Versioned",
                "billingTags": [],
                "name": "Example layer",
                "description": "A longer description of the layer contents",
                "partitioning": {
                     "scheme": "generic"
                },
                "id": "example-layer-id",
                "contentType": "text/csv",
                "tags": []
            }],
            "description": "",
            "notifications": {
                "enabled": false
             },
            "marketplaceReady": false,
            "id": "first-catalog-example-id",
            "version": -1,
            "tags": []
        }
        

Upload Data to a Catalog

  1. Create a file with the details you want to upload, such as partition-0.csv in the following example.

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA
    
  2. Enter the catalog layer partition put command.

    Linux
    Windows
    olp catalog layer partition put hrn:here:data:::first-catalog-example-id \
       example-layer-id --partitions testPart0:<path_to_file>/partition-0.csv
    olp catalog layer partition put hrn:here:data:::first-catalog-example-id ^
       example-layer-id --partitions testPart0:<path_to_file>\partition-0.csv

    The following lines appear.

    Uploading the files into partitions.
    
    100% [=====================================================================>] 1/1, ETA: 00:00:00
    The data was successfully added to the partitions: [testPart0]
    
  3. To verify the partition, enter the catalog layer partition list command.

    
       olp catalog layer partition list hrn:here:data:::first-catalog-example-id example-layer-id
       

    The console displays the following text.

    Available partitions:
    
    partition           size                checksum
    testPart0           23                  665d2817c51cbccd4312eb22699163c05b9d5a0a
    
    Total size: 23 B
    
  4. To retrieve the partition-0.csv file from the olpcli-example-catalog-id, enter the catalog layer partition get command.

    Linux
    Windows
    olp catalog layer partition get \
       hrn:here:data:::first-catalog-example-id example-layer-id --partitions testPart0
    olp catalog layer partition get ^
       hrn:here:data:::first-catalog-example-id example-layer-id --partitions testPart0

    The console should contain the text inside the partition-0.csv file:

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA
    

To get a full list of the available commands, enter olp --help.

results matching ""

    No results matching ""