Pipeline Quotas & Limits

For a pipeline, you must choose a certain number of units to allocate the resources for a Supervisor and Worker:

  • 1 unit is defined as 1 CPU, 7 GB RAM, and 8 GB Disk Space

For each Supervisor or Worker (aka supervisor-units and worker-units), the following size limits apply:

  • Minimum Size: 1 unit
  • Maximum Size: 15 units

A pipeline can contain:

  • Number of Supervisors: 1
  • Number of Workers: 199

Caution: Limited Resource Requirements

HERE platform limits are still applicable. A Pipeline can only consume a maximum of 200 CPU and 1.4 TB RAM. In other words, Size of Supervisor + (Size of Each Worker * Number of Workers) ≤ 200 CPU AND 1.4 TB RAM. If you experience any resource provisioning issues within these limits, you should contact HERE Support/Services.

For example:

For a Spark batch pipeline, choose the following units for resource allocation of the Supervisor (Spark Driver) and Worker (Spark Executor):

  • Spark Driver: 2 units (equal to 2 CPU, 14 GB RAM, 16 GB Disk Space)
  • Spark Executors:
    • Each Executor size: 3 units (equal to 3 CPU, 21 GB RAM, 24 GB Disk Space)
    • Number of Executors: 2
  • Total Resource Requirement: 8 units (equal to 8 CPU, 56 GB RAM, 64 GB Disk Space)

results matching ""

    No results matching ""