Geovisualization Developer's Guide

Uploading Data with a Recipe

Ingesting data with a recipe is a method for creating a spatial index of your geometry data well suited to large and complex data sources. It involves more steps than directly importing data from a file or URL, but will result in more reliable imports and performant queries and visualizations.

Note: Read this guide for more guidance on the best formatting for the file you upload.

To ingest using a recipe, take the following steps.

Create an empty dataset

Create an empty dataset with a POST request including your APP_ID, APP_CODE and a name and description in the body:

https://datalens.api.here.com/v1/datasets?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}

{
  "name": "Name of your new dataset",
  "description": "Description of your new dataset, can be empty"
}
        

You can also use the datasets endpoint in the 04. Datasets CRUD folder of our Postman collection.

You will receive a response body that contains the new dataset ID.
Note: The supported protocols for the URL are HTTP, HTTPS and SFTP. All csv, csv.gz or zip files in that directory are downloaded and added to the dataset. The dataset must have a predefined schema and all files in the directory must conform to it.

Create an empty recipe

A receipe defines how your data source should be represented in the Geovisualization dataset query language…

POST to the following endpoint with the recipe code as the request body:

https://datalens.api.here.com/v1/recipes?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}
  
{
  "source": {
    "url": "https://s3-eu-west-1.amazonaws.com/apps.cit.datalens.api.here.com/documentation/windfarms_fivestates.csv",
    "is_folder": false
  },
  "format": {
    "delimiter": ",",
    "line_terminator": "\n",
    "quote_char": "\"",
    "encoding": "utf-8",
    "header_line": true,
    "bom": "UTF8",
    "skip_lines": 1
  },
  "schema": {
    "columns": [
      {
        "alias": "lat",
        "type": "number",
        "name": "lat",
        "index": 0
      },
      {
        "alias": "lon",
        "type": "number",
        "name": "lon",
        "index": 1
      },
      {
        "alias": "output",
        "type": "number",
        "name": "output",
        "index": 2
      },
      {
        "alias": "state",
        "type": "string",
        "name": "state",
        "index": 3
      }
    ],
    "geometries": [
      {
        "type": "point",
        "x": "lon",
        "y": "lat"
      }
    ]
  }
}
        
You will receive a response body that contains the new recipe ID.

Import your data

POST to the following endpoint with the recipe ID as the request body and the dataset ID received in the previous step:

https://datalens.api.here.com/v1/datasets/{DATASET_ID}?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}          

{
  "source": {
    "recipe_id": "{RECIPE_ID}"
  }
}
      
You can also use the datasets/_ID_ endpoint in the 04.1. Dataset Import folder of our Postman collection.

Check progress of an upload

You can check the progress of the data ingestion by sending a GET request to the following resource, with the relevant {DATASET_ID}:
https://datalens.api.here.com/v1/datasets/{DATASET_ID}/progress?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}

Updating a recipe and dataset

To change a recipe a dataset is based upon, you first change the recipe used to import the dataset and then append any new data.

There are specific endpoints for updating each section of the recipe definition, the source, the format and schema. For example to change the schema, send a PUT request to the following resource, with the relevant {RECIPE_ID}:


https://datalens.api.here.com/v1/recipes/{RECIPE_ID}/schema?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}          

{
  "schema": {…}
}
      
And now append your new data by sending a POST request to the following endpoint with the {RECIPE_ID} as the request body and the {DATASET_ID} as a parameter:

https://datalens.api.here.com/v1/datasets/{DATASET_ID}?app_id={YOUR_APP_ID}&app_code={YOUR_APP_CODE}          

{
  "source": {
    "recipe_id": "{RECIPE_ID}"
  }
}