Pipeline Quotas & Limits

When configuring a pipeline template, you need to choose the amount of resources to allocate to a Supervisor and Worker. Resources are allocated as units, each containing a pre-defined number of CPUs, RAM size, and disk space.

For each Supervisor or Worker (also referred to as supervisor units and worker units), the following size limits apply:

• Minimum Size: 1 unit
• Maximum Size: 15 units

A pipeline can contain:

• Number of Supervisors: 1
• Number of Workers: 199

The amount of resources per unit is defined by the Resource profile that you can select for each pipeline template.

Resource profiles

The HERE platform supports several resource profiles with different combinations of CPU number, RAM size, and disk space per unit to ensure the most effective resource consumption. There are memory-intensive, CPU-intensive, and default resource profiles. Depending on the data processing type and the tasks that the pipeline is designed to perform, you can choose a resource profile that is the most optimal for your project goals.

The following default resource profiles are available:

• Stream pipelines: 1 CPU, 7 GB RAM, and 8 GB disk space per unit
• Batch pipelines: 1 CPU, 7 GB RAM, and 32 GB disk space per unit

Memory-intensive and CPU-intensive resource profiles contain a higher or lower RAM amount, respectively. To find out which resource profiles are currently available on the HERE platform and the corresponding resource-per-unit ratios, make a request to one of the following endpoints:

/v2/resourceProfiles

or

/v3/resourceprofiles

The response will contain a list of the available resource profiles with the amount of resources per unit:

{
    "availableResourceProfiles": [
        {
            "id": "profileId",
            "label": "User-friendly label for display",
            "name": "Description of the profile",
            "cpuPerUnit": 1,
            "memoryInGiBPerUnit": 2,
            "diskInGiBPerUnit": 3
        }
    ]
}

To find the resource profiles available for a particular runtime environment, query the following endpoints:

/v2/runtimeenvironments

or

/v3/listRuntimeEnvironments

The response contains a list of runtime environments showing the available and default resource profiles:

{
    "availableRuntimes": [
        {
            "id": "batch-x.y.z",
            "availableResourceProfileIds": [
                [
                    "HS1"
                ]
            ],
            "defaultWorkerResourceProfileId": "HS1",
            "defaultSupervisorResourceProfileId": "HS1",
            "maxWorkerUnits": 15,
            "maxSupervisorUnits": 15,
            "description": "Spark x.y.z, Java x, Scala y",
            "deprecationPeriodAnnounced": "2007-12-03T10:15:30+01:00",
            "deprecationPeriodEnd": "2007-12-03T10:15:30+01:00",
            "displayName": "Batch x.y.z"
        }
    ]
}

For example:

For a Spark batch pipeline, choose the following units for resource allocation of the Supervisor (Spark Driver) and Worker (Spark Executor):

• Spark Driver: 2 units (equal to 2 CPU, 14 GB RAM, 64 GB Disk Space)
• Spark Executors:
  • Each Executor size: 3 units (equal to 3 CPU, 21 GB RAM, 96 GB Disk Space)
  • Number of Executors: 2
• Total Resource Requirement: 8 units (equal to 8 CPU, 56 GB RAM, 256 GB Disk Space)