Publish JSON schema to the platform

Objectives: Generate, build, and publish JSON schema to the platform

Complexity: Beginner

Time to complete: 20 min

Prerequisites: Get your credentials, Verify your credentials

The HERE platform allows you to define schemas that are used for sharing and transferring data between other platform users and pipelines. A schema is a collection of related messages that describe the structure of a partition in a catalog layer. It is a high-level construct that consists of a set of artifacts.

This tutorial demonstrates how to generate and build a JSON schema and publish it to the platform. It shows how to get information about a published JSON schema and how to download a JSON schema.

The tutorial covers the following topics:

• Generate JSON schema
• JSON schema structure
• Update JSON schema project
• Validate and package JSON schema
• Publish JSON schema to the platform
• Get details about published schema
• Download schema
• Use Maven Archetype

To follow this tutorial, you need to have the CLI installed and have both platform and repository credentials set up on your local machine.

To verify that you set up your platform credentials correctly, make sure that the Verify your credentials tutorial returns the expected result.

Generate JSON schema

There are two ways to generate a JSON schema: using the OLP CLI and using a Maven Archetype.

In this tutorial, we will use the OLP CLI to generate a JSON schema.

The OLP CLI provides the olp schema generate command that generates a new Maven project for a schema.

Let's generate a JSON schema project in the current directory with the com.here.platform.tutorial group ID and generate_json_schema artifact ID:

olp schema generate com.here.platform.tutorial generate_json_schema

The OLP CLI should return the following output:

Schema has been successfully generated in /home/user/generate_json_schema.

Once the JSON schema is generated, we can take a look at the structure of the generated project.

JSON schema structure

The olp schema generate command generates a project with the following structure:

generate_json_schema/
├── ds
│   ├── pom.xml
│   └── src
│       ├── assembly
│       │   └── json.xml
│       └── main
│           └── resources
│               ├── renderers
│               │   └── ReadMe.txt
│               └── ResourcesReadMe.txt
├── json
│   ├── pom.xml
│   └── src
│       ├── assembly
│       │   └── json.xml
│       └── main
│           ├── json
│           │   └── com
│           │       └── here
│           │           └── platform
│           │               └── tutorial
│           │                   └── generate_json_schema
│           │                       └── v1
│           │                           └── generate_json_schema.json
│           └── resources
│               └── description.md
├── pom.xml
└── schema.yml

From the structure above, we can see that the generated project contains the ds, json modules, schema.yml and pom.xml files.

The ds module contains a data schema (ds) bundle. The ds/src/main/resources/renderers folder is the place where you can put a GeoJSON renderer that becomes bundled with your schema once published to the HERE platform. The renderer is written in JavaScript and is used for visualizing partition data on top of the base map in the HERE platform portal. For more information, see the Implement a GeoJSON Renderer documentation.

The json module contains the definitions JSON messages for the schema. The json/src/main/json/com/here/platform/tutorial/generate_json_schema/v1/generate_json_schema.json file is the main JSON file that describes the data format:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "mainMessageId",
    "definitions": {
        "MainMessage": {
            "type": "object",
            "properties": {
                "lat": {
                    "description": "Latitude",
                    "type": "integer"
                },
                "lon": {
                    "description": "Longitude",
                    "type": "integer"
                }
            },
            "required": [
                "lat", "lon"
            ]
        }
    }

The default main JSON file describes the data format including two required properties lat and lon of the integer type.

The content of the main JSON file must conform to the JSON syntax, contain a property with reference to a JSON schema and follow that schema specification. These requirements are checked by validators.

The json/src/main/resources/description.md file is an empty file where you can put detailed schema documentation in the Markdown format.

The pom.xml file in the schema project root contains the main schema details, such as name, description, version. If you want to override the default values in the pom.xml, specify the name, description or version parameters when running the olp schema generate command.

Declarations of other schemas that the given schema is dependent on are also included in pom.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.here.platform.tutorial</groupId>
  <artifactId>generate_json_schema_v1</artifactId>
  <version>1.0.0</version>
  <packaging>pom</packaging>

  <name>JSON Schema</name>
  <description>JSON Schema project for OLP</description>

  <inceptionYear>2022</inceptionYear>

  <distributionManagement>
    <repository>
      <id>HERE_PLATFORM_ARTIFACT</id>
      <layout>here</layout>
      <!-- This custom url serves to invoke the HERE wagon extension that handles OAUTH
           and re-writes Maven URIs for the HERE secure repository.-->
      <url>here+artifact-service://artifact-service</url>
    </repository>
  </distributionManagement>

  <parent>
    <groupId>com.here.platform</groupId>
    <artifactId>sdk-standalone-bom_2.12</artifactId>
    <version>2.43.1</version>
  </parent>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <here.plugin.version>2.1.18</here.plugin.version>
    <artifact.wagon.version>2.0.11</artifact.wagon.version>
  </properties>

  <repositories>
    <repository>
      <id>HERE_PLATFORM_ARTIFACT</id>
      <layout>here</layout>
      <url>here+artifact-service://artifact-service</url>
    </repository>
  </repositories>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-source-plugin</artifactId>
          <version>3.0.1</version>
          <executions>
            <execution>
              <id>attach-sources</id>
              <phase>verify</phase>
              <goals>
                <goal>jar-no-fork</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.7</version>
        <inherited>false</inherited>
        <executions>
          <execution>
            <id>copy-schema</id>
            <phase>package</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <target>
                <echoproperties destfile="${project.build.directory}/build.properties"/>
                <copy file="${project.basedir}/schema.yml" overwrite="true" todir="${project.build.directory}">
                  <filterset begintoken="${" endtoken="}" filtersfile="${project.build.directory}/build.properties"/>
                </copy>
              </target>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-enforcer-plugin</artifactId>
        <version>3.0.0-M2</version>
        <executions>
          <execution>
            <id>enforce-property</id>
            <goals>
              <goal>enforce</goal>
            </goals>
            <configuration>
              <rules>
                <requireProperty>
                  <property>project.groupId</property>
                  <message>Project groupId must be specified.</message>
                  <regex>^[a-zA-Z_]+[a-zA-Z0-9_-]*(\.[a-zA-Z0-9_]+[a-zA-Z0-9_-]*)+$</regex>
                  <regexMessage>Project groupId must be specified as reverse domain name.</regexMessage>
                </requireProperty>
              </rules>
              <fail>true</fail>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>1.6</version>
        <inherited>false</inherited>
        <executions>
          <execution>
            <id>attach-artifacts</id>
            <phase>package</phase>
            <goals>
              <goal>attach-artifact</goal>
            </goals>
            <configuration>
              <artifacts>
                <artifact>
                  <file>${project.build.directory}/schema.yml</file>
                  <type>yml</type>
                  <classifier>here-schema</classifier>
                </artifact>
              </artifacts>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>flatten-maven-plugin</artifactId>
        <version>1.1.0</version>
        <configuration>
          <flattenMode>bom</flattenMode>
          <outputDirectory>${project.build.directory}</outputDirectory>
          <flattenedPomFilename>${project.artifactId}-${project.version}.pom</flattenedPomFilename>
        </configuration>
        <executions>
          <execution>
            <id>flatten</id>
            <phase>process-resources</phase>
            <goals>
              <goal>flatten</goal>
            </goals>
          </execution>
          <execution>
            <id>flatten.clean</id>
            <phase>clean</phase>
            <goals>
              <goal>clean</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
    <extensions>
      <extension>
        <groupId>com.here.platform.artifact</groupId>
        <artifactId>artifact-wagon</artifactId>
        <version>${artifact.wagon.version}</version>
      </extension>
    </extensions>
  </build>
  <modules>  
    <module>json</module>
    <module>ds</module>
  </modules>
</project>

The schema.yml file in the root of the project is a descriptor file that indicates that your project is a schema project:

name: "${project.name}"
summary: "${project.description}"
layerSchema: true
type: "json"

The name and summary properties define the name and description of your schema, respectively. By default, these values are taken from pom.xml file. The general rule of thumb is not to change these properties in the schema.yml file and define them only in pom.xml.

The layerSchema property defines whether this schema should be used as a standalone schema. Setting it to false means that the schema will only be used as a dependency for another schema and will not be visible on the platform nor listed in the OLP CLI.

Update JSON schema project

Once we have generated a JSON schema project, we can update it taking into account our needs.

In this tutorial, we will update the json/src/main/json/com/here/platform/tutorial/generate_json_schema/v1/generate_json_schema.json main JSON file by providing additional properties and describing this schema in the documentation file.

Let's start with updating the main JSON file.

We are going to add a new optional property zoom_level with title Zoom Level, description The relative vertical stacking of multiple points with the same latitude and longitude., and type integer.

The updated main JSON file looks as follows:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "mainMessageId",
    "definitions": {
        "MainMessage": {
            "type": "object",
            "properties": {
                "lat": {
                    "description": "Latitude",
                    "type": "integer"
                },
                "lon": {
                    "description": "Longitude",
                    "type": "integer"
                },
                "zoom_level":{
                    "description": "The relative vertical stacking of multiple points with the same latitude and longitude.",
                    "type": "integer"
                }
               },
            "required": [
                "lat", "lon"
            ]
        }
    }
}

The next step is to add documentation about our schema. You can add prepared documentation when generating a new schema project using the olp schema generate OLP CLI command with the --documentation parameter. There are two ways to pass your documentation to the schema project. You can pass the URL to the documentation if you want to refer to external documentation or specific HERE documentation, or you can specify the path to a Markdown file if you want to include your own schema description.

If you already have a generated schema project, and you want to update or add documentation, you need to update the json/src/main/resources/description.md file, or to add/update the docUrl field in the schema.yml file.

Let's add the following documentation to the json/src/main/resources/description.md file:

This schema was generated for the `Publish JSON schema to the Platform` tutorial.

The main json file describes the data format including the following properties required and optional parameters:

Required properties:

`lat` - the measurement of distance north or south of the Equator.
`lon` - the measurement east or west of the prime meridian.

Optional properties:

`zoom_level` - a number that defines how large or small the contents of a map appear in a map view.

Once our schema project is updated, we need to validate and package the project to make sure that our changes have not broken the project itself.

Validate and package JSON schema

The OLP CLI provides the olp schema package command to build and validate the schema project.

The olp schema package command can be useful when you have updated the schema project and want to make sure that your project is valid before deploying it to the platform. You don't need to use this command if you do not change the schema.

During the schema building, validate plugins verify that the JSON schema project contains a valid main JSON file. There are two JSONSyntax and JSONSchema validators. The JSONSyntax validator verifies that content of the file is a syntactically correct JSON. The JSONSchema validator verifies that JSON contains a $schema property which defines the schema specification and follows that specification. The validation results you can find in the json/target/json/META-INF/validation-results.json file after olp schema package command is executed.

Let's build an updated JSON schema project using the following OLP CLI command:

olp schema package --input /home/user/generate_json_schema

The OLP CLI should return the following output:

Schema has been built successfully.

The message above indicates that your JSON schema project is valid and the main JSON message was validated by the Maven plugins. The following json/target/json/META-INF/validation-results.json file should be generated:

{
  "passedValidators" : [ "JSONSyntax", "JSONSchema" ]
}

If your schema project has an invalid JSON file, the olp schema package command should return the following output with a path to the log file where you can find additional information about what went wrong with your project:

ERROR: Schema packaging failed.
Log file path: /tmp/olpcli2445382190186474494.log

To disable validators in the schema project, you can set jsonSyntax and jsonSchema configuration to false in the json/pom.xml and ds/pom.xml files:

      <plugin>
        <groupId>com.here.platform.schema.maven_plugins</groupId>
        <artifactId>json-validate-plugin</artifactId>
        <version>${here.plugin.version}</version>
        <executions>
          <execution>
            <id>validate</id>
            <phase>compile</phase>
            <goals>
              <goal>validate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <jsonSyntax>false</jsonSyntax>
          <jsonSchema>false</jsonSchema>
        </configuration>
      </plugin>

Once the project is validated and packaged, we can publish it to the platform.

Publish schema to the platform

Let's deploy the JSON schema to the platform, using the olp schema put OLP CLI command:

olp schema put --input /home/user/generate_json_schema

The OLP CLI should return the following output:

WARNING: Please wait while the command is completed. This may take several minutes.

Schema hrn:here:schema::realm:com.here.tutorial:generate_json_schema_v1:1.0.0 has been created.

Please use 'olp schema show' command to get more information.

Once the schema is deployed, you can see its details in the OLP CLI or portal.

Get details about published schema

Let's get more information about the deployed schema using the olp schema show command:

olp schema show hrn:here:schema::realm:com.here.test:test_schema_v1:1.0.0

The OLP CLI should return the following information:

Details of the JSON schema:
schema version           1.0.0
artifact id              generate_json_schema_v1
group id                 com.here.platform.tutorial
type                     json
summary                  JSON Schema project for HERE Platform
Schema validation results:
module         JSON syntax                   JSON schema
json           true                          true
ds             true                          true

Artifacts:
hrn:here:artifact::realm:com.here.platform.tutorial:generate_json_schema_v1_json:1.0.0
hrn:here:artifact::realm:com.here.platform.tutorial:generate_json_schema_v1_ds:1.0.0

From the output above, we get general information about the deployed schema, particularly, schema version, group id, artifact id, schema type, and schema summary. Schema validation results indicate which plugins were used for which modules. And the last information in the output is artifacts.

When you deploy a JSON schema, two artifacts are deployed as well: _json and _ds.

The _json artifact contains a zip file containing definition JSON messages for the schema. The artifact contains the main JSON file and validation results in JSON format. The optional file is description.md. This file contains documentation.

The _ds artifact contains a zip file containing a data schema (ds) bundle with the JSON messages. To download the _ds artifact, see the Download schema section.

Download schema

To download a _ds package with JSON messages, the OLP CLI provides the olp schema get command.

olp schema get hrn:here:schema::realm:com.here.test:test_schema_v1:1.0.0 --output ~/Desktop/ds_package

The OLP CLI should return the following output:

Schema package was successfully downloaded to ~/Desktop/ds_package/generated_schema_v1_ds-1.0.0.zip

The downloaded zip file contains the following structure:

generated_schema_v1_ds-1.0.0/
├── com
│   └── here
│       └── platform
│           └── tutorial
│               └── generate_json_schema
│                   └── v1
│                       └── generate_json_schema.json
├── META-INF
│   ├── layer.manifest.json
│   └── validation-results.json
├── renderers
│   └── ReadMe.txt
└── ResourcesReadMe.txt

From the project structure above, we can see that the _ds package contains the main generate_json_schema.json message, validation-results.json validation results, and a layer.manifest.json manifest that declares the top-level message.

Use Maven archetype

There is an additional option to generate, validate and upload schemas. Since a JSON schema project is a Maven project, you can use Maven commands to work with schemas. For this purpose, the SDK for Java & Scala provides the json_archetype that allows you to generate a JSON schema project using Maven.

To generate a JSON schema project with the json_archetype, use the following Maven command that does the same as the olp schema generate com.here.platform.tutorial generate_json_schema command in the Generate JSON schema section:

mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema \
-DarchetypeArtifactId=json_archetype \
-DarchetypeVersion=2.1.129 \
-DartifactId=generate_json_schema \
-Dversion=1.0.0 \
-DgroupId=com.here.platform.tutorial \
-DmajorVersion=1

mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema ^
-DarchetypeArtifactId=json_archetype ^
-DarchetypeVersion=2.1.129 ^
-DartifactId=generate_json_schema ^
-Dversion=1.0.0 ^
-DgroupId=com.here.platform.tutorial ^
-DmajorVersion=1

To validate and build generated schema, use the mvn install command from the root of the project which is similar to the olp schema package command.

To deploy the schema without using the olp schema put command, you can run the mvn deploy command from the root of the generated project.

Once you deployed the schema, you can inspect deployed schema using the OLP CLI commands mentioned in this tutorial.

Conclusion

In this tutorial, you have practiced generating, building, and publishing a JSON schema to the platform using the OLP CLI and Maven and got acquainted with the JSON schema structure and the OLP CLI commands that allow you to work with JSON schemas.

Further information

• For more information on how to work with schemas using the OLP CLI, see the OLP CLI Guide.