This guide walks you through the steps to migrate resources for a use case into a platform project, if those resources were not originally created in a platform project. Once your catalogs for a use case are in a project, you can still share them with all or specific projects in your organization so that they can be reused for other use cases.
This guide currently covers migration for a use case with batch pipeline(s). New features are coming soon that will allow you to migrate use cases that include streaming pipelines.
Home project: This typically means the project in which a resource was created. For this migration guide, it means the project to which you are adding or moving resources. This project now becomes the "home project" for those resources. A resource can only have one home project, but catalogs can be shared to other projects in your organization so that they can be reused in multiple use cases. Storage for a catalog is logged to the home project, and data i/o for using the catalog is logged to the project that's using it.
Adding vs. moving resources: This guide refers to adding catalogs to projects. This means that you are adding the catalog to a project, but not removing any permissions associated with the catalog outside the context of the project. You can choose to remove those permissions later and purely manage the resource in the project, but "adding" rather than completely "moving" the catalog into a project helps mitigate the risk of breaking anything as you migrate your use case to a project. Later, when we add capabilities to migrate pipelines, we will be moving them rather than adding them, that is, they will not be accessible outside of project context.
Link any HERE catalogs used in your use case to the project you created in step 1 using the platform portal or the CLI. To link a HERE catalog from the platform portal, go to the Projects Manager, click the project to which you want to add a catalog, and select the Resources tab. Click Add catalog and choose to Link a catalog. A list of available HERE catalogs will be displayed.
Add your input and output catalogs to the project you created in step 1 using the CLI "resource project add" command.
Permissions outside of the project will not be impacted, that is, any pipelines that use these catalogs will continue to run.
Access to schemas associated to a catalog will be available in the project scope.
What if you make a mistake? Resources created outside of a project can be removed from a project in order to help you "undo" a mistake in this step of the process. Just use the "resource project remove" CLI command. If the catalog is shared to other projects before it's removed from a project, this action revokes its availability to the other projects. If the catalog is shared and also linked to another project before it's removed from its home project, this action revokes those links.
Add members (users, groups, apps) to your project using the "project access grant" CLI command or the platform portal. IMPORTANT: Don't forget to add runtime credentials for pipeline versions as member(s) of the project.
To add a member to a project using the platform portal:
Full access to all resources in the project will be granted to these project members by default. You can optionally restrict access to resources by a project member by applying existing HERE or custom-created policies using the CLI. The CLI Projects Workflows explain how to set up granular access to the project resources through project policies. The Project Policy Commands allow you to create and manage custom policies.
Restrict access to catalogs by runtime credentials if desired (optional), but make sure they have the minimum required rights of reading from input catalog(s) and writing to output catalog(s).
Recreate batch pipeline(s) in the project (including pipeline template, pipeline, and first pipeline version) using the CLI or platform portal. Ensure that you specify project scope, regardless of the interface. If you use the platform portal and use the Pipelines tab instead of the Projects Manager, ensure that the project is selected in the drop-down menu in the top right-hand-side of the screen.
Once you have validated that your use case is working inside the project, you may choose to remove access to catalogs outside of the project context. This will ensure that access to those catalogs is now managed only within the context of the project. In order to remove permissions on the catalog outside of the project, use the catalog permission list on a catalog and revoke permissions CLI commands.
CLI commands to move a pipeline (batch or streaming) into a project. Once these are in place, you can opt to use them instead of steps 5-7 above.