HERE Map Content (HMC) Cartography Filter

Overview

HERE Map Content (HMC) Cartography Filter narrows down cartography data from HERE Map Content catalog to the categories requested by user, e.g. lakes, cities or universities/colleges. This pipeline template searches tiles in HMC Cartography layer for places belonging to the requested category and then saves these places along with their location details in a Versioned layer of an output catalog. This catalog can be later used as an input by other pipelines. Please see Supported Categories section for the list of all categories that can be extracted from Cartography layer. During deployment of this pipeline template you will need to provide a comma separated list of the categories you wish to extract, e.g.: "5,17,48".

Structure

HERE Map Content (HMC) Cartography Filter operates as a batch pipeline that reads data from HMC catalog (Cartography layer), searches for requested places and publishes this information to a Versioned layer. Output schema is the same as schema used in Cartography layer of HMC catalog: com.here.schema.rib:cartography_v2. Data is stored in protobuf format.

Application Flow Diagram
Figure 1. Application Flow Diagram

Legend: Legend Diagram

Prerequisites

  • This pipeline template writes output to a Versioned layer of the catalog. You can use your existing output layer or let the Wizard script create a new catalog/layer for you. Please refer to the section "Execution" below for further details.
  • If you are planning to use an existing catalog/layer please make sure that your output catalog is shared with the same GROUP that will be used to deploy this pipeline template.
  • Confirm that your local credentials (~/.here/credentials.properties) are added to the same group.
  • You chose the categories you wish to extract and noted their indices.

Execution

In order to deploy and run this pipeline template, you will need the Wizard Deployer. The Wizard executes interactively, asking questions about the application, and expects the user to provide needed answers. Assuming you followed the Wizard's documentation instructions and set up the needed parameters beforehand, follow these steps:

  1. Execute the script as ./wizard.sh
  2. Follow the prompts and provide needed answers

You can use your existing output layer or let the Wizard script create a new catalog/layer for you. If using an existing catalog, make sure it is shared with GROUP_ID which will be used for this deployment.

PLEASE NOTE: During deployment with Wizard script you will be asked to provide a bounding box of the area you wish to process by supplying four coordinates. You will also be asked to describe the size of the area by picking one of the presented options. Pick the one that best defines the size of your area (approximately). The answer to this question helps to define the number of workers(cores) required for processing and will affect the processing time.

Verification

In Platform Portal select the Pipelines tab where you should be able to see your pipeline deployed and running. After your pipeline finishes and your data is published, search Splunk logs for "published_tile_counter" and you will be able to see the message like this:

com.here.platform.data.processing.driver.impl.DriverImpl  - published_tile_counter=75

If you wish to compare amount of original places vs. filtered places per partition, you may change the logging level for com.here.platform.pipelines.filter.hmc.CompilerConfig class to DEBUG before activating pipeline version and search logs for "Original partition" and "Filtered partition" after the job finishes running. You will see messages like this:

com.here.platform.pipelines.filter.hmc.Compiler  - ===> Original partition 20095877: locationCount=178 placesCount=178
com.here.platform.pipelines.filter.hmc.Compiler  - ===> Filtered partition 20095877: locationCount=3 placesCount=3
Supported Categories
0. AIRCRAFT_ROADS("here:cm:carto-category:1907403")
1. AIRPORT("here:cm:carto-category:1900403")
2. ALLOTMENT("here:cm:carto-category:900151")
3. AMUSEMENT_PARK("here:cm:carto-category:2000460")
4. ANIMAL_PARK("here:cm:carto-category:2000461")
5. BAY_HARBOUR("here:cm:carto-category:507116")
6. BEACH("here:cm:carto-category:509998")
7. BUILT_UP_AREA("here:cm:carto-category:900156")
8. CANAL_WATER_CHANNEL("here:cm:carto-category:500414")
9. CARTOGRAPHIC_COUNTRY_BOUNDARY("here:cm:carto-category:908000")
10. CARTOGRAPHIC_DISPUTED_COUNTRY_BOUNDARY("here:cm:carto-category:908004")
11. CARTOGRAPHIC_DISPUTED_STATE_BOUNDARY("here:cm:carto-category:908005")
12. CARTOGRAPHIC_SETTLEMENT_BOUNDARY("here:cm:carto-category:908003")
13. CARTOGRAPHIC_STATE_PROVINCE_BOUNDARY("here:cm:carto-category:908001")
14. CEMETERY("here:cm:carto-category:2000420")
15. CITY("here:cm:carto-category:900101")
16. CONGESTION_ZONE("here:cm:carto-category:9997004")
17. COUNTRY("here:cm:carto-category:907196")
18. COUNTY("here:cm:carto-category:900170")
19. DISPUTED_COUNTRY_BOUNDARY("here:cm:carto-category:907197")
20. DISPUTED_STATE_BOUNDARY("here:cm:carto-category:909997")
21. ENVIRONMENTAL_ZONE("here:cm:carto-category:9997010")
22. FLOOD_PRONE_AREA("here:cm:carto-category:600102")
23. GLACIER("here:cm:carto-category:509997")
24. GOLF_COURSE("here:cm:carto-category:2000123")
25. HOSPITAL("here:cm:carto-category:2000408")
26. HURRICANE_PRONE_AREA("here:cm:carto-category:600101")
27. INDUSTRIAL_COMPLEX("here:cm:carto-category:2000200")
28. INTERMITTENT_RIVER("here:cm:carto-category:500413")
29. ISLAND("here:cm:carto-category:509999")
30. LAKE("here:cm:carto-category:500421")
31. LIGHT_RAIL("here:cm:carto-category:1800203")
32. MILITARY_BASE("here:cm:carto-category:900108")
33. NATIONAL_FOREST("here:cm:carto-category:900152")
34. NATIVE_AMERICAN_RESERVATION("here:cm:carto-category:900107")
35. NEIGHBOURHOOD_BOUNDARY("here:cm:carto-category:908002")
36. OCEAN("here:cm:carto-category:500116")
37. PARK_CITY_COUNTY("here:cm:carto-category:900150")
38. PARK_STATE("here:cm:carto-category:900130")
39. PARK_IN_WATER("here:cm:carto-category:900140")
40. PARK_MONUMENT_NATIONAL("here:cm:carto-category:900103")
41. PARKING_GARAGE("here:cm:carto-category:1700216")
42. PARKING_LOT("here:cm:carto-category:1700215")
43. PEDESTRIAN_ZONE("here:cm:carto-category:900158")
44. RAILROAD("here:cm:carto-category:1800201")
45. RAIL_YARD("here:cm:carto-category:9997007")
46. RESIDENTIAL_AREA("here:cm:carto-category:900160")
47. RIVER("here:cm:carto-category:500412")
48. ROAD_NETWORK("here:cm:carto-category:9999999")
49. SEAPORT_HARBOUR("here:cm:carto-category:9997008")
50. SHOPPING_CENTRE("here:cm:carto-category:2000124")
51. SPORTS_COMPLEX("here:cm:carto-category:2000457")
52. STATE("here:cm:carto-category:909996")
53. SUBWAY_LINE("here:cm:carto-category:1800202")
54. TSUNAMI_PRONE_AREA("here:cm:carto-category:600103")
55. UNDEFINED_TRAFFIC_AREA("here:cm:carto-category:900159")
56. UNIVERSITY_COLLEGE("here:cm:carto-category:2000403")
57. WATER_BOUNDARY("here:cm:carto-category:9997021")
58. WOODLAND("here:cm:carto-category:900202")

Verification

In Platform Portal select the Pipelines tab where you should be able to see your pipeline deployed and running. After your pipeline finishes and your data is published, you can find your output catalog under Data tab and inspect your data visually or query/retrieve your data programmatically using one of the following options:

Cost Estimation

Executing this pipeline will incur the following costs:

Storage-Blob

Cost will depend on the amount of data that will be published to a Versioned layer as an output from execution.

Data Transfer IO

Will depend on the amount of:

  • amount of data read from HMC catalog - this will depend on the bounding box you specified
  • amount of data published to your output layer
    Metadata
    Will depend on the amount and size of partitions(metadata) stored in the Versioned layer.
    Compute Core and Compute RAM
    Depending on the option you choose for your bounding box during deployment this pipeline will use 1, 2 or 3 workers to process. 3 workers are expected to process entire world in under 1 hour.
    Log Search IO
    Depends on the log level set for the execution of this pipeline. To minimize this cost user can set log level to WARN.

Support

If you need support with this pipeline, please contact us.

results matching ""

    No results matching ""