Introduction

The HERE Data Hub CLI is a Node.js command line interface which works not only with HERE Data Hub, but eventually, with any HERE API. Right now, it allows you to interact with Data Hub to create and manage your projects and easily upload and manage your datasets. The Data Hub CLI now supports advanced features to better enable sophisticated HERE Studio projects. These advanced features require the use of the Data Hub Add-on plan (details of which are covered later in this tutorial).

In this tutorial we'll go over:

  • How to configure your HERE account using the CLI
  • Upload CSV data to a HERE Data Hub space
  • Preview a space on a map
  • Create Hexbins
  • Options for previewing Hexbins
  • Share your HERE Data Hub space
  • Additional CLI options

Pre-requisites and Installing the Data Hub CLI

To get stared, you'll need a HERE developer account. You can sign up at: developer.here.com. Some parts of this tutorial will require PRO features, to use these, you'll need a HERE Developer account with access to Pro features. For more information about Pro account features see: HERE CLI.

To install the Data Hub CLI, open a command prompt and type npm install @here/cli, additionally use -g to install globally. Your command line should be look similar to this:

carousel-of-progress$ npm install -g @here/cli
npm WARN deprecated text-encoding@0.6.4: no longer maintained
/Users/carousel-of-progress/.npm-global/bin/here -> /Users/carousel-of-progress/.npm-global/lib/node_modules/@here/cli/bin/here.js
+ @here/cli@1.2.0
added 401 packages from 300 contributors in 9.747s

How to configure your HERE account using the CLI

Next, we need to make sure your existing account is configured for command line use. Doing this links your work on the Data Hub site to anything you'll do on the command line.

Type here configure account and you'll be prompted for the email address used for your HERE Developer account and the password.

carousel-of-progress$ here configure account
Welcome to the HERE XYZ! 
Please review the terms and conditions https://developer.here.com/terms-and-conditions

? Enter (A)ccept or (D)ecline to proceed A
prompt: Email:  carousel-of-progress@gmail.com
prompt: Password:  
Secrets verified successfully
Default App Selected - cV1JVtCdMWM60dkKY4Fb

If you have previously used any of the functionality of HERE through the web interface, like HERE Studio, you can now see that activity.

Type here xyz list and your existing spaces will be displayed by space id. The space id is how we'll manage all of our data in the Data Hub CLI.

carousel-of-progress$ here xyz list
┌────────────┬──────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│ id         │ title                                    │ description                                                  │
├────────────┼──────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ dhGZiRwZ   │ space_moutain.geojson                    │ out of this world data                                       │
│ ZQuY98FS   │ haunted_mansion.geojson                  │ super scary data                                             │
└────────────┴──────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Upload CSV data to an HERE Data Hub Space

For this tutorial we want to upload data to a space using the Data Hub CLI. One awesome thing about the Data Hub CLI is that we can upload data in many forms. This includes shapefiles, csv, and geojson to name a few. I've found some data on dockless vehicle ridership in Louisville, KY. The city collects data from the major service providers and aggregates it to a block level to protect rider privacy. More information on this dataset is available at this link.

The ridership data is available for download in a csv format. Normally, you might use the Add Delimited Text Layer in QGIS to make a shapefile or geojson from this CSV file. With the Data Hub CLI we can upload the raw CSV that contains latitude and longitude values. We don't even have to rename the columns! Our dataset has both start and end lat/long values - for this demo, we'll just be using the start values.

Here's what our data looks like:

We'll use several options when uploading this dataset:

  • -f to define the location of the file on our local machine
  • -s to stream the upload, a faster upload approach that will help avoid any Node memory errors
  • -x to define the X or the longitude field in my dataset
  • -y to define the Y or the latitude field in my dataset
  • -p define attribute columns to use has tags - in a comma separated list.

We can format our upload like:

carousel-of-progress$ here xyz upload -f downloads/DocklessTripOpenData.csv -s -p StartDate,TripDuration,TripDistance,DayOfWeek

After executing this command you will be prompted for a few things:

  • Enter a title for the new space - the name for your space - note, this is not the space id.
  • Enter a description for the new space - a short description for your space.
  • Select property which should be be used for Latitude - arrow key up/down to interactively choose the latitude field in your dataset.
  • Select property which should be be used for Longitude - arrow key up/down to interactively choose the longitude field in your dataset.

Here's what your CLI should look like as you proceed through the options:

carousel-of-progress$ here xyz upload -f downloads/DocklessTripOpenData.csv -x StartLongitude -y StartLatitude -s -p StartDate,TripDuration,TripDistance,DayOfWeek
No space ID specified, creating a new XYZ space for this upload.
? Enter a title for the new space:  louisville_dockless
? Enter a description for the new space :  open dockless rideshare data
xyzspace '7pdp8Llz' created successfully
? Select property which should be be used for Latitude StartLatitude : 38.253
new Latitude field selected - StartLatitude
? Select property which should be be used for Longitude StartLongitude : -85.756
new Longitude field selected - StartLongitude
uploaded feature count :434382, failed feature count :0
434382 features uploaded to XYZ space '7pdp8Llz' in 228.135 seconds, at the rate of 1904 features per second

And here is a sped up gif of the action:

Our dataset was successfully uploaded! Before we dive in we can check out our data using the show command, but we'll want to limit the output on the command line using -l so tons of records don't display.

A few things to note:

  • id - a unique id field created during the upload process
  • geometry.type - the geometry type of your features
  • tags - the attributes from our data we specifically included in our upload command
carousel-of-progress$ here xyz show 7pdp8Llz -l 2
┌─────────────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────┐
│ id                      │ geometry.type           │ tags                    │ createdAt               │ updatedAt               │
├─────────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────┤
│ 0ceb818abcd9f73fd9b2fcd │ Point                   │ startdate@2018-11-22,   │ 2020-2-1 14:57          │ 2020-2-1 14:57          │
│ a0e334001               │                         │ tripduration@8,         │                         │                         │
│                         │                         │ tripdistance@0.33,      │                         │                         │
│                         │                         │ dayofweek@5,            │                         │                         │
│                         │                         │ docklesstripopendata    │                         │                         │
│ f9ab4ecce7e1d774cf73a25 │ Point                   │ startdate@2018-11-16,   │ 2020-2-1 14:57          │ 2020-2-1 14:57          │
│ bdf90ddf6               │                         │ tripduration@5,         │                         │                         │
│                         │                         │ tripdistance@0.47,      │                         │                         │
│                         │                         │ dayofweek@6,            │                         │                         │
│                         │                         │ docklesstripopendata    │                         │                         │
└─────────────────────────┴─────────────────────────┴─────────────────────────┴─────────────────────────┴─────────────────────────┘

Preview a space on a map

Now that our data is contained in a space, we can preview it, visually, and we have a few options for that as well.

Adding -w will open up the data as geojson in your default browser and is good for smaller datasets.

carousel-of-progress$ here xyz show 7pdp8Llz -w

carousel-of-progress$ here xyz show 7pdp8Llz -v

Adding -v will open up the data as geojson in your default browser using Space Invader and is great for a dataset like mine with hundreds of thousands features.

Space Invader also provides some search, filter, and cartographic options. In this example, I've symbolized my point size by the trip duration, which is a numeric field from 0-3100.

Alternatively, we can log in to HERE Studio and work with our data in there as well.

Create Hexbins

So we have a lot of points, and they load pretty fast. If we could visually summarize them, we can begin to see some patterns. Enter Hexbins, a pro feature of the Data Hub CLI that will summarize our data in "bins", sized and grouped by our choosing, and ultimately rendered faster than single point features.

We have many options when creating hexbins at the command line:

  • -z defines appropriately sized hexbins per zoom, you can specify a single zoom like 8 or range like 8-13
  • -c lets you define the comma separated sizes of hexbins you want, like 500,5000,50000 or 500 meters, 5,000 kilometers, or 50,000 kilometers wide
  • -a using -a followed by a property will add average and sum values per bin to your output. ex -a DayOfWeek
  • -p lets us define a property to group by, which will also add subcounts for - so if we chose DayOfWeek from our data, it would output a group per hexbin for each day of the week with corresponding counts.

Lets start simple:

carousel-of-progress$ here xyz hexbin 7pdp8Llz -z 9-12
Creating hexbins for the space data
hexbin creation done for feature count - 434582.
No hexbin space found, creating new hexbin space
xyzspace 'urcQSgMb' created successfully
uploading the hexagon grids to space with size 1600 / zoom Level 9
uploaded 100.00%
data upload to xyzspace 'urcQSgMb' completed
==========================================================
                     Upload Summary                     
==========================================================
....
...
..
.
Total unique tag Count : 7
Unique tag list  :["centroid_200","cell_200","centroid","zoom12","zoom12_centroid","7pdp8llz","hex34333JMKDVCrooon9"]
┌──────────────────────┬───────┐
│ TagName              │ Count │
├──────────────────────┼───────┤
│ centroid_200         │ 1695  │
│ cell_200             │ 1695  │
│ centroid             │ 1695  │
│ zoom12               │ 1695  │
│ zoom12_centroid      │ 1695  │
│ 7pdp8llz             │ 1695  │
│ hex34333JMKDVCrooon9 │ 1695  │
└──────────────────────┴───────┘

1695 features uploaded to XYZ space 'urcQSgMb' in 8.673 seconds, at the rate of 195 features per second
hexbins written to space urcQSgMb from points in source space 7pdp8Llz

Our command line console will appear to do a lot - it is! We see the output of hexbin creation for each of our zoom levels with some nice statistics. It's important to note that our hexbins have a new space id separate from our point data.

We can also preview our data, like before. This time we can add -t , to let us specify the appropriate hexbin zoom level to preview.

carousel-of-progress$ here xyz show urcQSgMb -v -t zoom11_hexbin

Once we zoom into our data set in Space Invader, you can see we've got zoom 11 resolution hexbins of our data, automatically symbolized by the total count of feature. Now lets run the hexbin creation again, but this time include the DayOfWeek property from our data.

carousel-of-progress$ here xyz hexbin 7pdp8Llz -z 9-12 -p DayOfWeek

Using Space Invader again, we can now analyze the ridership counts for each Day of the Week.

You can also use Space Invader to access hexbins from their source space.

here xyz show urcQSgMb -v

In the info panel on the top right, you'll notice there a new message:

hexbins available: mode 0
urcQSgMb, zoom 9,10,11,12

Click on the mode and it will toggle through hexbins, centroids, and back to the original points. As you zoom, Space Invader will automatically change the tags and bring up the appropriate hexbins or centroids. If you want to override this (say you want to look at z12 centroids at z8 to get more of a heatmap effect) you should use the hexbin space itself.

Protip: you can press x on your keyboard to toggle through the hexbin options.

More Hexbin Options

Since we love giving you options with Data Hub, there's yet another way to do hexbins! If you have an Data Hub Pro account, you can dynamically generate H3 hexbins via the Data Hub API endpoint using the clustering parameter.

There are different options available in the CLI hexbins and the API clustering hexbins: in the latter, there is no built-in color property. An API hexbin only knows about itself and doesn't have a global view of min and max data ranges like the CLI hexbins do, and there are no subcounts by property. On the other hand, you get API clustering on the fly without having to wait. Each H3 hexbin has a globally unique ID, and you can get statistics if you cluster by a quantitative property. The resolution (width) of H3 hexbins are fixed by tile zoom level, while you can view the CLI hexbins of any width at any zoom level. You can export the CLI hexbins as a geojson file, while the API hexbins are only available online.

You can toggle these by clicking on the clustering toggle in the info panel.

Launch Live Viewer

output
Figure 1. output

Protip: to access the count of an H3 hexbin, you need to choose aggregation.property, or if you have chosen a property to cluster by, aggregation.propertyname.qty

More details on the clustering option are available at here.xyz/api.

Share Your Data Hub Space

You can share your fancy new hexbins with all your hexbin loving friends, straight from the command line. This feature of the Data Hub CLI allows you to share your work with other users with HERE Developer accounts, which empowers them to use any number of HERE tools to analyze and visualize your data.

carousel-of-progress$ here xyz config 7pdp8Llz --shared true
setting the space SHARED
space config updated successfully!

Now your hexbins are shared - anyone using your shared data will do so with their own account and tokens.

Additional CLI Options

There's a ton of cli options available to you - here's a small sample of them:

List all of the spaces you have have now:

carousel-of-progress$ here xyz list
┌────────────┬──────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│ id         │ title                                    │ description                                                  │
├────────────┼──────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ dhGZiRwZ   │ space_moutain.geojson                    │ out of this world data                                       │
│ ZQuY98FS   │ haunted_mansion.geojson                  │ super scary data                                             │
│ 7pdp8Llz   │ louisville_dockless                      │ open dockless rideshare data                                 │
│ urcQSgMb   │ hexbin space of louisville_dockless      │ hexbin space created for source spaceId - 7pdp8Llz , title - │
│            │                                          │ louisville_dockless                                          │
└────────────┴──────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Get basic information for a single space:

carousel-of-progress$ here xyz config 7pdp8Llz 
=========== SPACE CONFIG INFO ===========
┌────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────┐
│ property                       │ value                                                                                      │
├────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ id                             │ 7pdp8Llz                                                                                   │
│ title                          │ louisville_dockless                                                                        │
│ description                    │ open dockless rideshare data                                                               │
│ owner                          │ HERE-84ed6fd5-7387-440c-8d93-fede73d3806f                                                  │
│ cid/app_id                     │ cV1JVtCdMWM60dkKY4Fb                                                                       │
│ client                         │ {"hexbinSpaceId":"urcQSgMb"}                                                               │
│ shared                         │ false                                                                                      │
│ storageparam                   │ null                                                                                       │
│ storageid                      │ psql                                                                                       │
└────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────┘

Get more than basic information about a space - warning, this will print out a lot of information to the CLI.

carousel-of-progress$ here xyz config 7pdp8Llz --stats
=========== SPACE MAIN STATS INFO ===========
┌────────────────────────────────┬────────────────────────────────┬────────────┐
│ property                       │ value                          │ estimated  │
├────────────────────────────────┼────────────────────────────────┼────────────┤
│ BBox                           │ -85.845,38.175,-85.661,38.272  │ true       │
│ Size                           │ 606.8 MB                       │ true       │
│ Feature Count                  │ 434651                         │ true       │
│ Geometry Types                 │ Point                          │ true       │
│ Properties Searchable          │ PARTIAL                        │            │
└────────────────────────────────┴────────────────────────────────┴────────────┘

=========== FEATURES' TAGS STATS INFO ===========
Estimated : true
┌────────────────────────────────────────────────────┬────────────┐
│ key                                                │ count      │
├────────────────────────────────────────────────────┼────────────┤
│ docklesstripopendata                               │ 434651     │
│ dayofweek@7                                        │ 80845      │
│ dayofweek@6                                        │ 69544      │
│ dayofweek@1                                        │ 66501      │
│ dayofweek@5                                        │ 64762      │
│ dayofweek@4                                        │ 58677      │
│ dayofweek@3                                        │ 56939      │


....
...
..
.

│ tripduration@96                                    │ 434        │
│ tripduration@99                                    │ 434        │
└────────────────────────────────────────────────────┴────────────┘

=========== FEATURES' PROPERTIES STATS INFO ===========
Estimated : true
┌────────────────────────────────────────────────────┬─────────────────┬────────────┐
│ key                                                │ count           │ searchable │
├────────────────────────────────────────────────────┼─────────────────┼────────────┤
│ DayOfWeek                                          │ 434651          │ true       │
│ EndDate                                            │ 434651          │ true       │
│ EndLatitude                                        │ 434651          │ true       │
│ EndLongitude                                       │ 434651          │ true       │
│ EndTime                                            │ 434651          │ true       │
│ HourNum                                            │ 434651          │ true       │
│ StartDate                                          │ 434651          │ true       │
│ StartTime                                          │ 434651          │ true       │
│ TripDistance                                       │ 434651          │ false      │
│ TripDuration                                       │ 434651          │ false      │
│ TripID                                             │ 434651          │ false      │
└────────────────────────────────────────────────────┴─────────────────┴────────────┘

See the raw json of your spaces endpoint:

carousel-of-progress$ here xyz config 7pdp8Llz -r
{ id: '7pdp8Llz',
  title: 'louisville_dockless',
  description: 'open dockless rideshare data',
  storage: { id: 'psql', params: null },
  owner: 'HERE-84ed6fd5-7387-440c-8d93-fede73d3806f',
  client: { hexbinSpaceId: 'urcQSgMb' },
  cid: 'cV1JVtCdMWM60dkKY4Fb',
  createdAt: 1580586969081,
  updatedAt: 1580596017725,
  contentUpdatedAt: 1580587149410 }

Be sure to visit the HERE CLI site for more info!

results matching ""

    No results matching ""