Write to Catalog Layers

The SDK provides separate write methods tailored to the type of layer being written to:

  • write_index_layer
  • write_stream_layers
  • write_versioned_layers
  • write_volatile_layers

In all cases except index layer, writing data simultaneously to multiple layers of the same type in the same catalog is supported. To allow this, and to maintain consistency for versioned layers, the write methods are surfaced as catalog operations, not layer operations.

The layers_write_info Structure

When writing data to stream, versioned, or volatile layers, you will need to construct a nested dict containing layer_ids, partition_ids, and files or data in bytes to be uploaded.

# layers_write_info for writing single partition to single layer
layer_id = 'my-test-layer'
partition_data = {'23300689': path_to_local_data_file_or_reference_to_in_memory_binary_data}
my_layer_info = {layer_id: partition_data}
# layers_write_info for writing multiple partitions to single layer
layer_id = 'my-test-layer'
partition_data = {'23300689': some_path_or_bytes, '23300347': some_other_path_or_bytes}
my_layer_info = {layer_id: partition_data}
# layers_write_info for writing to partition in two different layers
layer_id_1 = 'layer-1'
layer_id_2 = 'layer-2'
partition_data_1 = {'23300689': some_path_or_bytes}
partition_data_2 = {'23300689': some_other_path_or_bytes}
my_layer_info = {layer_id_1: partition_data_1, layer_id_2: partition_data_2}

Examples below show only required parameters for write methods. See the API documentation for details of optional parameters applicable to each method.

Write to Stream Layer

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_stream_layers(layers_write_info=my_layer_info)

Write to Versioned Layer

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_versioned_layers(layers_write_info=my_layer_info)

Write to Volatile Layer

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_volatile_layers(layers_write_info=my_layer_info)

Write to Index Layer

Writing to an index layer is different than other layer types, and there are additional required method arguments (fields). Below is a simple example of the syntax. Consult the API documentation for full details.

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
layer_id = "sample_index_layer"
file_path = "~/sample_data.parquet"
fields = {"ingestionTime": 1114, "partition_id": 343433}
catalog.write_index_layer(layer_id, path_or_data=file_path, fields=fields)

results matching ""

    No results matching ""