Objectives: Generate, build, and publish JSON schema to the platform
Complexity: Beginner
Time to complete: 20 min
Prerequisites: Get your credentials, Verify your credentials
The HERE platform allows you to define schemas that are used for sharing and transferring data between other platform users and pipelines. A schema is a collection of related messages that describe the structure of a partition in a catalog layer. It is a high-level construct that consists of a set of artifacts.
This tutorial demonstrates how to generate and build a JSON schema and publish it to the platform. It shows how to get information about a published JSON schema and how to download a JSON schema.
The tutorial covers the following topics:
To follow this tutorial, you need to have the CLI installed and have both platform and repository credentials set up on your local machine.
To verify that you set up your platform credentials correctly, make sure that the Verify your credentials tutorial returns the expected result.
Generate JSON schema
There are two ways to generate a JSON schema: using the OLP CLI and using a Maven Archetype.
In this tutorial, we will use the OLP CLI to generate a JSON schema.
The OLP CLI provides the olp schema generate
command that generates a new Maven
project for a schema.
Let's generate a JSON schema project in the current directory with the com.here.platform.tutorial
group ID and generate_json_schema
artifact ID:
olp schema generate com.here.platform.tutorial generate_json_schema
Note
The group ID should usually include a team/product hierarchy to the group ID, such as com.some-company.automated-driving.schema
. This way, different teams can manage their schemas independently.
When you deploy a schema to the Artifact Service, its group ID is being reserved by your organization, and no other realm can reuse it in this case. So, specifying generic group ID values like com.example
, com.here.schema
, and com.here.example
, may make it impossible to upload the schema to the Artifact Service repository and, thus, using it in catalogs or sharing it with other users.
The OLP CLI should return the following output:
Schema has been successfully generated in /home/user/generate_json_schema.
Once the JSON schema is generated, we can take a look at the structure of the generated project.
JSON schema structure
The olp schema generate
command generates a project with the following structure:
generate_json_schema/
├── ds
│ ├── pom.xml
│ └── src
│ ├── assembly
│ │ └── json.xml
│ └── main
│ └── resources
│ ├── renderers
│ │ └── ReadMe.txt
│ └── ResourcesReadMe.txt
├── json
│ ├── pom.xml
│ └── src
│ ├── assembly
│ │ └── json.xml
│ └── main
│ ├── json
│ │ └── com
│ │ └── here
│ │ └── platform
│ │ └── tutorial
│ │ └── generate_json_schema
│ │ └── v1
│ │ └── generate_json_schema.json
│ └── resources
│ └── description.md
├── pom.xml
└── schema.yml
From the structure above, we can see that the generated project contains the ds
, json
modules, schema.yml
and pom.xml
files.
The ds
module contains a data schema (ds) bundle. The ds/src/main/resources/renderers
folder is the place where you can put a GeoJSON renderer that becomes bundled with your schema once published to the HERE platform. The renderer is written in JavaScript and is used for visualizing partition data on top of the base map in the HERE platform portal. For more information, see the Implement a GeoJSON Renderer documentation.
The json
module contains the definitions JSON messages for the schema. The json/src/main/json/com/here/platform/tutorial/generate_json_schema/v1/generate_json_schema.json
file is the main JSON file that describes the data format:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "mainMessageId",
"definitions": {
"MainMessage": {
"type": "object",
"properties": {
"lat": {
"description": "Latitude",
"type": "integer"
},
"lon": {
"description": "Longitude",
"type": "integer"
}
},
"required": [
"lat", "lon"
]
}
}
The default main JSON file describes the data format including two required properties lat
and lon
of the integer
type.
The content of the main JSON file must conform to the JSON syntax, contain a property with reference to a JSON schema and follow that schema specification. These requirements are checked by validators.
The json/src/main/resources/description.md
file is an empty file where you can put detailed schema documentation in the Markdown format.
The pom.xml
file in the schema project root contains the main schema details, such as name
, description
, version
. If you want to override the default values in the pom.xml
, specify the name
, description
or version
parameters when running the olp schema generate
command.
Declarations of other schemas that the given schema is dependent on are also included in pom.xml
:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.here.platform.tutorial</groupId>
<artifactId>generate_json_schema_v1</artifactId>
<version>1.0.0</version>
<packaging>pom</packaging>
<name>JSON Schema</name>
<description>JSON Schema project for OLP</description>
<inceptionYear>2022</inceptionYear>
<distributionManagement>
<repository>
<id>HERE_PLATFORM_ARTIFACT</id>
<layout>here</layout>
<url>here+artifact-service://artifact-service</url>
</repository>
</distributionManagement>
<parent>
<groupId>com.here.platform</groupId>
<artifactId>sdk-standalone-bom_2.12</artifactId>
<version>2.43.1</version>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<here.plugin.version>2.1.18</here.plugin.version>
<artifact.wagon.version>2.0.11</artifact.wagon.version>
</properties>
<repositories>
<repository>
<id>HERE_PLATFORM_ARTIFACT</id>
<layout>here</layout>
<url>here+artifact-service://artifact-service</url>
</repository>
</repositories>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>verify</phase>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<inherited>false</inherited>
<executions>
<execution>
<id>copy-schema</id>
<phase>package</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echoproperties destfile="${project.build.directory}/build.properties"/>
<copy file="${project.basedir}/schema.yml" overwrite="true" todir="${project.build.directory}">
<filterset begintoken="${" endtoken="}" filtersfile="${project.build.directory}/build.properties"/>
</copy>
</target>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.0.0-M2</version>
<executions>
<execution>
<id>enforce-property</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireProperty>
<property>project.groupId</property>
<message>Project groupId must be specified.</message>
<regex>^[a-zA-Z_]+[a-zA-Z0-9_-]*(\.[a-zA-Z0-9_]+[a-zA-Z0-9_-]*)+$</regex>
<regexMessage>Project groupId must be specified as reverse domain name.</regexMessage>
</requireProperty>
</rules>
<fail>true</fail>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.6</version>
<inherited>false</inherited>
<executions>
<execution>
<id>attach-artifacts</id>
<phase>package</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${project.build.directory}/schema.yml</file>
<type>yml</type>
<classifier>here-schema</classifier>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
<version>1.1.0</version>
<configuration>
<flattenMode>bom</flattenMode>
<outputDirectory>${project.build.directory}</outputDirectory>
<flattenedPomFilename>${project.artifactId}-${project.version}.pom</flattenedPomFilename>
</configuration>
<executions>
<execution>
<id>flatten</id>
<phase>process-resources</phase>
<goals>
<goal>flatten</goal>
</goals>
</execution>
<execution>
<id>flatten.clean</id>
<phase>clean</phase>
<goals>
<goal>clean</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<extensions>
<extension>
<groupId>com.here.platform.artifact</groupId>
<artifactId>artifact-wagon</artifactId>
<version>${artifact.wagon.version}</version>
</extension>
</extensions>
</build>
<modules>
<module>json</module>
<module>ds</module>
</modules>
</project>
The schema.yml
file in the root of the project is a descriptor file that indicates that your project is a schema project:
name: "${project.name}"
summary: "${project.description}"
layerSchema: true
type: "json"
The name
and summary
properties define the name
and description
of your schema, respectively. By default, these values are taken from pom.xml
file. The general rule of thumb is not to change these properties in the schema.yml
file and define them only in pom.xml
.
The layerSchema
property defines whether this schema should be used as a standalone schema. Setting it to false
means that the schema will only be used as a dependency for another schema and will not be visible on the platform nor listed in the OLP CLI.
Note
The schema won't be visible on the platform if the project was built and deployed without the schema.yml included in the root folder.
Update JSON schema project
Once we have generated a JSON schema project, we can update it taking into account our needs.
In this tutorial, we will update the json/src/main/json/com/here/platform/tutorial/generate_json_schema/v1/generate_json_schema.json
main JSON file by providing additional properties and describing this schema in the documentation
file.
Let's start with updating the main JSON file.
We are going to add a new optional property zoom_level
with title Zoom Level
, description The relative vertical stacking of multiple points with the same latitude and longitude.
, and type integer
.
The updated main JSON file looks as follows:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "mainMessageId",
"definitions": {
"MainMessage": {
"type": "object",
"properties": {
"lat": {
"description": "Latitude",
"type": "integer"
},
"lon": {
"description": "Longitude",
"type": "integer"
},
"zoom_level":{
"description": "The relative vertical stacking of multiple points with the same latitude and longitude.",
"type": "integer"
}
},
"required": [
"lat", "lon"
]
}
}
}
The next step is to add documentation about our schema. You can add prepared documentation when generating a new schema project using the olp schema generate
OLP CLI command with the --documentation
parameter. There are two ways to pass your documentation to the schema project. You can pass the URL to the documentation if you want to refer to external documentation or specific HERE documentation, or you can specify the path to a Markdown file if you want to include your own schema description.
If you already have a generated schema project, and you want to update or add documentation, you need to update the json/src/main/resources/description.md
file, or to add/update the docUrl
field in the schema.yml
file.
Let's add the following documentation to the json/src/main/resources/description.md
file:
This schema was generated for the `Publish JSON schema to the Platform` tutorial.
The main json file describes the data format including the following properties required and optional parameters:
Required properties:
`lat` - the measurement of distance north or south of the Equator.
`lon` - the measurement east or west of the prime meridian.
Optional properties:
`zoom_level` - a number that defines how large or small the contents of a map appear in a map view.
Once our schema project is updated, we need to validate and package the project to make sure that our changes have not broken the project itself.
Validate and package JSON schema
The OLP CLI provides the olp schema package
command to build and validate the schema project.
The olp schema package
command can be useful when you have updated the schema project and want to make sure that your project is valid before deploying it to the platform. You don't need to use this command if you do not change the schema.
During the schema building, validate plugins verify that the JSON schema project contains a valid main JSON file. There are two JSONSyntax
and JSONSchema
validators. The JSONSyntax
validator verifies that content of the file is a syntactically correct JSON. The JSONSchema
validator verifies that JSON contains a $schema
property which defines the schema specification and follows that specification. The validation results you can find in the json/target/json/META-INF/validation-results.json
file after olp schema package
command is executed.
Let's build an updated JSON schema project using the following OLP CLI command:
olp schema package --input /home/user/generate_json_schema
The OLP CLI should return the following output:
Schema has been built successfully.
The message above indicates that your JSON schema project is valid and the main JSON message was validated by the Maven plugins. The following json/target/json/META-INF/validation-results.json
file should be generated:
{
"passedValidators" : [ "JSONSyntax", "JSONSchema" ]
}
If your schema project has an invalid JSON file, the olp schema package
command should return the following output with a path to the log file where you can find additional information about what went wrong with your project:
ERROR: Schema packaging failed.
Log file path: /tmp/olpcli2445382190186474494.log
To disable validators in the schema project, you can set jsonSyntax
and jsonSchema
configuration to false
in the json/pom.xml
and ds/pom.xml
files:
<plugin>
<groupId>com.here.platform.schema.maven_plugins</groupId>
<artifactId>json-validate-plugin</artifactId>
<version>${here.plugin.version}</version>
<executions>
<execution>
<id>validate</id>
<phase>compile</phase>
<goals>
<goal>validate</goal>
</goals>
</execution>
</executions>
<configuration>
<jsonSyntax>false</jsonSyntax>
<jsonSchema>false</jsonSchema>
</configuration>
</plugin>
Note
It is not recommended to disable plugins, as this may cause schemes to malfunction.
Once the project is validated and packaged, we can publish it to the platform.
Let's deploy the JSON schema to the platform, using the olp schema put
OLP CLI command:
olp schema put --input /home/user/generate_json_schema
The OLP CLI should return the following output:
WARNING: Please wait while the command is completed. This may take several minutes.
Schema hrn:here:schema::realm:com.here.tutorial:generate_json_schema_v1:1.0.0 has been created.
Please use 'olp schema show' command to get more information.
Once the schema is deployed, you can see its details in the OLP CLI or portal.
Get details about published schema
Let's get more information about the deployed schema using the olp schema show
command:
olp schema show hrn:here:schema::realm:com.here.test:test_schema_v1:1.0.0
The OLP CLI should return the following information:
Details of the JSON schema:
schema version 1.0.0
artifact id generate_json_schema_v1
group id com.here.platform.tutorial
type json
summary JSON Schema project for HERE Platform
Schema validation results:
module JSON syntax JSON schema
json true true
ds true true
Artifacts:
hrn:here:artifact::realm:com.here.platform.tutorial:generate_json_schema_v1_json:1.0.0
hrn:here:artifact::realm:com.here.platform.tutorial:generate_json_schema_v1_ds:1.0.0
From the output above, we get general information about the deployed schema, particularly, schema version
, group id
, artifact id
, schema type
, and schema summary
. Schema validation results indicate which plugins were used for which modules. And the last information in the output is artifacts.
When you deploy a JSON schema, two artifacts are deployed as well: _json
and _ds
.
The _json
artifact contains a zip file containing definition JSON messages for the schema. The artifact contains the main JSON file and validation results in JSON format. The optional file is description.md
. This file contains documentation.
Note
The documentation.md
file can be overridden by the docUrl
property in the schema.yml
file. The docUrl
has higher priority, so if both documentation. md and docUrl specified - the docUrl
will be used.
To override the documentation.md
file when generating a schema, use the --doc-url <URL to the documentation>
parameter with the olp schema generate
command.
The _ds
artifact contains a zip file containing a data schema (ds) bundle with the JSON messages. To download the _ds
artifact, see the Download schema section.
Download schema
To download a _ds
package with JSON messages, the OLP CLI provides the olp schema get
command.
olp schema get hrn:here:schema::realm:com.here.test:test_schema_v1:1.0.0 --output ~/Desktop/ds_package
The OLP CLI should return the following output:
Schema package was successfully downloaded to ~/Desktop/ds_package/generated_schema_v1_ds-1.0.0.zip
The downloaded zip file contains the following structure:
generated_schema_v1_ds-1.0.0/
├── com
│ └── here
│ └── platform
│ └── tutorial
│ └── generate_json_schema
│ └── v1
│ └── generate_json_schema.json
├── META-INF
│ ├── layer.manifest.json
│ └── validation-results.json
├── renderers
│ └── ReadMe.txt
└── ResourcesReadMe.txt
From the project structure above, we can see that the _ds
package contains the main generate_json_schema.json
message, validation-results.json
validation results, and a layer.manifest.json
manifest that declares the top-level message.
Use Maven archetype
There is an additional option to generate, validate and upload schemas. Since a JSON schema project is a Maven project, you can use Maven commands to work with schemas. For this purpose, the SDK for Java & Scala provides the json_archetype
that allows you to generate a JSON schema project using Maven.
Note
In order to execute the commands listed below, you need to have Maven
installed on your local machine.
To generate a JSON schema project with the json_archetype
, use the following Maven command that does the same as the olp schema generate com.here.platform.tutorial generate_json_schema
command in the Generate JSON schema section:
mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema \
-DarchetypeArtifactId=json_archetype \
-DarchetypeVersion=2.1.117 \
-DartifactId=generate_json_schema \
-Dversion=1.0.0 \
-DgroupId=com.here.platform.tutorial \
-DmajorVersion=1
mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema ^
-DarchetypeArtifactId=json_archetype ^
-DarchetypeVersion=2.1.117 ^
-DartifactId=generate_json_schema ^
-Dversion=1.0.0 ^
-DgroupId=com.here.platform.tutorial ^
-DmajorVersion=1
To validate and build generated schema, use the mvn install
command from the root of the project which is similar to the olp schema package
command.
To deploy the schema without using the olp schema put
command, you can run the mvn deploy
command from the root of the generated project.
Once you deployed the schema, you can inspect deployed schema using the OLP CLI commands mentioned in this tutorial.
Conclusion
In this tutorial, you have practiced generating, building, and publishing a JSON schema to the platform using the OLP CLI and Maven and got acquainted with the JSON schema structure and the OLP CLI commands that allow you to work with JSON schemas.
- For more information on how to work with schemas using the OLP CLI, see the OLP CLI Guide.