Tutorials / Using the HERE Data Hub CLI for Advanced Searching
Last Updated: June 24, 2020

Introduction

The HERE Data Hub CLI, is a command line tool for interacting with HERE Data Hub spaces. Using the Data Hub CLI you can create and manage your Projects and easily upload and manage your datasets.

In this tutorial we’ll go over: - Configure your HERE Data Hub account using the CLI - Uploading Shapefile data to a Data Hub space - Use Property Search to query records - Create a space using a Schema Definition file - Create Rule-Based Tags - Load CSV data into a space using a Schema Definition file - Virtualize multiple spaces - Property Search Extra Credit

Pre-requisites and Installing the Data Hub CLI

To get stared you’ll need a HERE developer account. You can sign up at: developer.here.com. Some parts of this tutorial will require advanced features of Data Hub. To use these you’ll need a HERE Developer account with access to the Data Hub add-on plan. For more information about Pro account features see: HERE CLI.

To install the Data Hub CLI, open a command prompt and type npm install @here/cli. Additionally use -g to install globally. Your command line should be look similar to this:

living-with-the-land$ npm install -g @here/cli
npm WARN deprecated text-encoding@0.6.4: no longer maintained
/Users/carousel-of-progress/.npm-global/bin/here -> /Users/carousel-of-progress/.npm-global/lib/node_modules/@here/cli/bin/here.js
+ @here/cli@1.2.0
added 401 packages from 300 contributors in 9.747s

Configure your HERE Data Hub account using the CLI

Next, we need to make sure your existing account is configured for command line use. Doing this links your work on the Data Hub site to anything you’ll do on the command line.

Type here configure account and you’ll be prompted for the email address used for your HERE Developer account and the password.

living-with-the-land$ here configure account
Welcome to the HERE XYZ! 
Please review the terms and conditions https://developer.here.com/terms-and-conditions

? Enter (A)ccept or (D)ecline to proceed A
prompt: Email:  carousel-of-progress@gmail.com
prompt: Password:  
Secrets verified successfully
Default App Selected - cV1JVtCdMWM60dkKY4Fb

If you have previously used any of the functionality of HERE Data Hub through the web interface, like HERE Studio, you can now see that activity.

Type here xyz list and your existing spaces will be displayed by space id. The space id is how we’ll manage all of our data in the Data Hub CLI.

living-with-the-land$ here xyz list
┌────────────┬──────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│ id         │ title                                    │ description                                                  │
├────────────┼──────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ dhGZiRwZ   │ its__a_small_world.geojson               │ small data for a small world                                 │
│ ZQuY98FS   │ epcot_monorail                           │ monorail ridership data                                      │
└────────────┴──────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Uploading Shapefile data to an Data Hub Space

For this tutorial we want to visually analyze some property data. Our parcel data comes from the City of Poquoson, Virginia GIS Website. The city GIS office provides polygon data that only contains the geometry for each parcel and a single attribute for the account number of the parcel. Separately, the city’s Assessment office provides assessment data in CSV format. It has over 60 columns of information about owner, zoning, previous assessments, and much more. To keep it simple, we’ll only use the year built and total value columns.

First, we’ll upload the parcel geometries that came in shapefile format. We’re going to use two options: - f to define the shapefile to upload, only using the shp extension. HERE Data Hub will automatically take care of uploading the additional parts of the shapefile using the same name and location. - i to define our unique id, in our parcels this is the account number.

living-with-the-land$ here xyz upload -f parcel.shp -i ACCTNUM
No space ID specified, creating a new XYZ space for this upload.
? Enter a title for the new space:  poquoson parcels
? Enter a description for the new space :  property geometries
xyzspace 'xJ0JgMnP' created successfully
parcel.prj file exists, using this file for crs transformation
uploaded 100.00%
data upload to xyzspace 'xJ0JgMnP' completed
==========================================================
                     Upload Summary                     
==========================================================
Total 5389 features
┌──────────────┬───────┐
│ GeometryType │ Count │
├──────────────┼───────┤
│ Polygon      │ 5254  │
│ MultiPolygon │ 135   │
└──────────────┴───────┘

Total unique tag Count : 1
Unique tag list  :["parcel"]
┌─────────┬───────┐
│ TagName │ Count │
├─────────┼───────┤
│ parcel  │ 5389  │
└─────────┴───────┘

5389 features uploaded to XYZ space 'xJ0JgMnP' in 56.819 seconds, at the rate of 95 features per second

As you can see in the output, HERE Data Hub recognized the prj part of our shapefile and successfully uploaded our data. Now we can visually check our geometries using Space Invader. The command below will open our new space parcel data in a browser window.

living-with-the-land$  here xyz show z8JrJrg5 -v

Use Property Search to query records

Property Search is a function of the Data Hub CLI that allows you to search your data. This feature is available for columns that are indexed. Indexing automatically happens for small to medium size datasets (up to 10k features). You can also add indexing to a column by using:

living-with-the-land$  here xyz config xJ0JgMnP --searchable --add
? Enter the property name to make searchable ( create index on ) :  

The Data Hub CLI will prompt you for the column name you’d like to index. To see what indexes exist on your space by use:

living-with-the-land$  here xyz config xJ0JgMnP --searchable
All the properties of your space are searchable by default currently since your space size (feature count) is less than 10,000
Manually configured settings will take effect once the space size (feature count) is more than 10,000
┌──────────────────────────────────────────────────────────────┬──────────────────────┬──────────────────────┐
│ propertyName                                                 │ mode                 │ searchable           │
├──────────────────────────────────────────────────────────────┼──────────────────────┼──────────────────────┤
│ ACCTNUM                                                      │ auto                 │ true                 │
│ Shape.STAr                                                   │ auto                 │ true                 │
│ Shape.STLe                                                   │ auto                 │ true                 │
└──────────────────────────────────────────────────────────────┴──────────────────────┴──────────────────────┘

Property Search through the Data Hub CLI has a few input requirements:

  • wrap the entire search in double quotes
  • you must prefix your property names with p. example: p.YEARBUILT
  • available operators are =,!=,>,>=,<,<=
  • comma separate values to search multiple values of a property, this replaces an or
  • use the + to signify an and

Here’s an example of searching our parcel geometries by the Account Number:

living-with-the-land$ here xyz show xJ0JgMnP --search "p.ACCTNUM=1003"
┌─────────────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────┐
│ id                      │ geometry.type           │ tags                    │ createdAt               │ updatedAt               │
├─────────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────┤
│ 1003                    │ Polygon                 │ parcel                  │ 2020-2-5 17:53          │ 2020-2-5 17:53          │
└─────────────────────────┴─────────────────────────┴─────────────────────────┴─────────────────────────┴─────────────────────────┘

We can also add the -w or -v to visually inspect a search result. Lets search for three separate account numbers and display the results on a map:

living-with-the-land$ here xyz show xJ0JgMnP --search "p.ACCTNUM=83,85,89" -w`

Create a space using a Schema Definition file

A Schema Definition file is a json file that identifies the columns in your data, and the information about them, like if a column is a numeric data type. We can create a space using a Schema Definition file to ensure that all of the data loaded into the space meets the schema we have defined, basically validating your data. Using Schema Definition files is a PRO feature of the HERE Data Hub CLI.

We’re going to create a schema definition file for the assessment data we got from the city Assessor’s office. Here’s what the json file looks like:

{
  "definitions": {},
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://example.com/root.json",
  "type": "object",
  "title": "The Root Schema",
  "required": [
    "type",
    "id",
    "geometry",
    "properties"
  ],
  "properties": {
    "type": {
      "$id": "#/properties/type",
      "type": "string",
      "title": "The Type Schema",
      "default": "",
      "examples": [
        "Feature"
      ],
      "pattern": "^(.*)$"
    },
    "id": {
      "$id": "#/properties/id",
      "type": "string",
      "title": "The Id Schema",
      "default": "",
      "examples": [
        "ec15403776cc33e75dbd348321912964"
      ],
      "pattern": "^(.*)$"
    },
    "geometry": {
      "$id": "#/properties/geometry",
      "type": "object",
      "title": "The Geometry Schema",
      "required": [
        "type",
        "coordinates"
      ],
      "properties": {
        "type": {
          "$id": "#/properties/geometry/properties/type",
          "type": "string",
          "title": "The Type Schema",
          "default": "",
          "examples": [
            "Point"
          ],
          "pattern": "^(.*)$"
        },
        "coordinates": {
          "$id": "#/properties/geometry/properties/coordinates",
          "type": "array",
          "title": "The Coordinates Schema",
          "items": {
            "$id": "#/properties/geometry/properties/coordinates/items",
            "type": "number",
            "title": "The Items Schema",
            "default": 0.0,
            "examples": [
              -76.394111528,
              37.116341946
            ]
          }
        }
      }
    },
    "properties": {
      "$id": "#/properties/properties",
      "type": "object",
      "title": "The Properties Schema",
      "required": [
        "ACCTNUM",
        "YEARBUILT",
        "TOTALVALUE"
      ],
      "properties": {
        "ACCTNUM": {
          "$id": "#/properties/properties/properties/ACCTNUM",
          "type": "integer",
          "title": "The Acctnum Schema",
          "default": 0,
          "examples": [
            5258
          ]
        },
        "YEARBUILT": {
          "$id": "#/properties/properties/properties/YEARBUILT",
          "type": "integer",
          "title": "The Yearbuilt Schema",
          "default": 0,
          "examples": [
            2009
          ]
        },
        "TOTALVALUE": {
          "$id": "#/properties/properties/properties/TOTALVALUE",
          "type": "integer",
          "title": "The Totalvalue Schema",
          "default": 0,
          "examples": [
            222800
          ]
        }
      }
    }
  }
}

You can use a service like jsonschema.net to create a Schema Definition file from your data. Now lets create a new space using these options: - t the title of our space - d a description for our space - s the schema definition file

living-with-the-land$ here xyz create -t "poquoson assessment" -d "poquoson property assessment data" -s schema_definition.json
xyzspace '7CYxUQUJ' created successfully

We can confirm this worked by using --schema as part of a config call to your space like:

living-with-the-land$ here xyz config 7CYxUQUJ --schema
{
  "definitions": {},
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "#/items",
  "type": "object",
  "title": "The Items Schema",
  "properties": {
     "ACCTNUM": {
        "$id": "#/items/properties/ACCTNUM",
        "type": "integer",
        "title": "The Acctnum Schema",
        "default": 0
     },
     "YEARBUILT": {
        "$id": "#/items/properties/YEARBUILT",
        "type": "integer",
        "title": "The Yearbuilt Schema",
        "default": 0
     },
     "TOTALVALUE": {
        "$id": "#/items/properties/TOTALVALUE",
        "type": "integer",
        "title": "The Totalvalue Schema",
        "default": 0
     },
     "LATITUDE": {
        "$id": "#/items/properties/LATITUDE",
        "type": "number",
        "title": "The Latitude Schema",
        "default": 0
     },
     "LONGITUDE": {
        "$id": "#/items/properties/LONGITUDE",
        "type": "number",
        "title": "The Longitude Schema",
        "default": 0
     }
  }
}

We’ve verified that we created a space with our desired schema, and when we can load data into the space, any data not valid according to the schema we defined will not be loaded.

Create Rule-Based Tags

Another config option for our space is Rule Based tagging. This feature allows us to add tags to our data based on rules we define. Based on the schema for our new space, we know that we have YEARBUILT and TOTALVALUE columns in our dataset. We can use rule based tagging to categorize the values of these columns, which can help us analyze our data. Lets create a few rules based on the YEARBUILT column.

We create Rule Based tags while using the config command. To get help for HERE Data Hub CLI config features you can always run:

living-with-the-land$ here xyz config -h

Just like the help describes, we use --tagrules and --add to create Rule Based tags. The HERE Data Hub CLI will first walk you through naming your tag, then prompt you to define the condition for your rule. This uses the same syntax as the Property Search feature we went through above.

Lets create these value groups: - boomers = 0 - 1965 - genx = 1965-1980 - millennials = 1980-1995 - genz = 1995-2015 - genalpha = 2015-2020

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules --add 
Starting to add a new synchronous rule to automatically tag features..
? Enter a tag name you would like to assign :  boomers
Please enter condition(s) for the auto tagging your features with  `boomers` e.g. "f.id == 123 || (p.country=='USA' & p.count<=100)"
? condition :   p.YEARBUILT<1965
tagrules updated successfully!

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules --add 
Starting to add a new synchronous rule to automatically tag features..
? Enter a tag name you would like to assign :  genx
Please enter condition(s) for the auto tagging your features with  `genx` e.g. "f.id == 123 || (p.country=='USA' & p.count<=100)"
? condition :  p.YEARBUILT>=1965 & p.YEARBUILT<1980

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules --add  
Starting to add a new synchronous rule to automatically tag features..
? Enter a tag name you would like to assign :  millennials
Please enter condition(s) for the auto tagging your features with  `millennials` e.g. "f.id == 123 || (p.country=='USA' & p.count<=100)"
? condition :   p.YEARBUILT>=1980 & p.YEARBUILT<1995
tagrules updated successfully!

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules --add 
Starting to add a new synchronous rule to automatically tag features..
? Enter a tag name you would like to assign :  genz
Please enter condition(s) for the auto tagging your features with  `genz` e.g. "f.id == 123 || (p.country=='USA' & p.count<=100)"
? condition :  p.YEARBUILT>=1995 & p.YEARBUILT<2015
tagrules updated successfully!

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules --add 
Starting to add a new synchronous rule to automatically tag features..
? Enter a tag name you would like to assign :  genalpha
Please enter condition(s) for the auto tagging your features with  `genalpha` e.g. "f.id == 123 || (p.country=='USA' & p.count<=100)"
? condition :   p.YEARBUILT>=2015
tagrules updated successfully!

We can verify that everything we just did works by using --tagrules when running the config for our space.

living-with-the-land$ here xyz config 7CYxUQUJ --tagrules
┌─────────────────────────────────────┬───────┬─────────────────────────────────────────────────────────────────────────────┐
│ tag_name                            │ mode  │ auto_tag_condition                                                          │
├─────────────────────────────────────┼───────┼─────────────────────────────────────────────────────────────────────────────┤
│ boomers                             │ sync  │ p.YEARBUILT<1965                                                            │
│ genalpha                            │ sync  │ p.YEARBUILT>=2015                                                           │
│ genx                                │ sync  │ p.YEARBUILT>=1965 & p.YEARBUILT<1980                                        │
│ genz                                │ sync  │ p.YEARBUILT>=1995 & p.YEARBUILT<2015                                        │
│ millennials                          │ sync  │ p.YEARBUILT>=1980 & p.YEARBUILT<1995                                        │
└─────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────────┘

Load CSV data into a space using a Schema Definition file

We’ve set the schema for data to be validated against and we’ve created rules on a column to categorize the data when we upload it - so let’s get to that part! Using the same options, as when we loaded our parcel geometries. - -f for the file name - -i for the naming ACCTNUM as our unique identifier

living-with-the-land$ here xyz upload 7CYxUQUJ -f poq_assessment_data.csv -i ACCTNUM
uploaded 100.00%
data upload to xyzspace '7CYxUQUJ' completed
==========================================================
                     Upload Summary                     
==========================================================
Total 5389 features
┌──────────────┬───────┐
│ GeometryType │ Count │
├──────────────┼───────┤
│ Point        │ 5389  │
└──────────────┴───────┘

Total unique tag Count : 1
Unique tag list  :["poq_assessment_data"]
┌────────────────────┬───────┐
│ TagName            │ Count │
├────────────────────┼───────┤
│ poq_assessment_data │ 5389  │
└────────────────────┴───────┘

5389 features uploaded to XYZ space 'JiLeIOFz' in 39.667 seconds, at the rate of 136 features per second

If we use the command describe, we can take a deep dive into our attributes and tags.

living-with-the-land$  here xyz describe 7CYxUQUJ 
Operation may take a while. Please wait.....
==========================================================
                     Summary for Space 7CYxUQUJ
==========================================================
Total 5388 features
┌──────────────┬───────┐
│ GeometryType │ Count │
├──────────────┼───────┤
│ Point        │ 5388  │
└──────────────┴───────┘

Total unique tag Count : 6
Unique tag list  :["poq_assessment_data","genz","millennials","boomers","genx","genalpha"]
┌────────────────────┬───────┐
│ TagName            │ Count │
├────────────────────┼───────┤
│ poq_assessment_data│ 5388  │
│ boomers            │ 1978  │
│ genx               │ 1324  │
│ millennials        │ 1062  │
│ genz               │ 863   │
│ genalpha           │ 161   │
└────────────────────┴───────┘

Features created from 2020-2-5 23:20 to 2020-2-5 23:21
Features updated from 2020-2-5 23:20 to 2020-2-5 23:21

WOW! We can see the year built values sorted by count, and a final tally of how many unique values exist. We can even use the show command with -t to show features in a particular tag, and then -v to auto open in Space Invader.

here xyz show 7CYxUQUJ -t genalpha -v

Virtualize multiple spaces

We have parcel geometries loaded in one space, and assessment data loaded in another. Now we want to see if we can merge these spaces together so we can visualize the assessment data by parcel geometry.

The virtual space command comes with two main options, group or associate, each with their own purpose. Group smashes multiple spaces together and presents them as one space. Associate takes features from one space and merges them into the second, kind of like a join. For this example, we’ll use associate by using the -a command.

living-with-the-land$ here xyz vs -a 7CYxUQUJ,xJ0JgMnP 
virtual xyzspace 'epVjE8Dq' created successfully

We now have a new space with both spaces merged together. This is all possible because we set the account number as the unique identifier, by specifying the -i property when we uploaded each space. Now, let’s take a peek:

living-with-the-land$ here xyz show epVjE8Dq -v

It worked! We can see the assessment data is attached to the parcel geometries. And finally, we can use list to see what our spaces look like now:

here xyz list
┌────────────┬──────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│ id         │ title                                    │ description                                                  │
├────────────┼──────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ xJ0JgMnP   │ poquoson parcels                         │ property geometries                                          │
│ 7CYxUQUJ   │ poquoson assessment                      │ poquoson property assessment data                            │
│ epVjE8Dq   │ XYZ Virtual Space, uUMH0m8p -> z8JrJrg5  │ association of uUMH0m8p (poquoson assessment) and z8JrJrg5   │
│            │ (associate)                              │ (poquoson parcels)                                           │
└────────────┴──────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Virtual Space Extra Credit

A new CLI command, here xyz join, simplifies the virtual space creation process. If you have a CSV with unique IDs, you can use join to associate its properties with geometries in an existing space, if the feature IDs match.

here xyz join spaceWithGeometries -f my.csv -k csvColumn

This command is especially useful for dealing with census tables.

Note that the join command shares many of the CSV options available in upload. More details are available in the online help via here xyz join -h.

Property Search Extra Credit

Property Search is a pretty awesome way to query your data straight from the command line. Here’s some additional examples of searches you can do.

search for account number 1003
living-with-the-land$ here xyz show xJ0JgMnP --search "p.ACCTNUM=1003"
search parcels built after the year 2000 and have a total value over $1,000,000
living-with-the-land$ here xyz show 7CYxUQUJ --search "p.YEARBUILT>2000+p.TOTALVALUE>1000000"
search parcels built in 1989 OR 1990 AND have a total value less than $200,000
 here xyz show 7CYxUQUJ --search "p.YEARBUILT=1989,1990+p.TOTALVALUE<200000" 

And, hey!, we can add -w to any of the searches above and visualize results as well!