Skip to main content

write_partition_from_rows

Function write_partition_from_rows 

Source
pub async fn write_partition_from_rows(
    lake: Arc<DataLakeConnection>,
    view_metadata: ViewMetadata,
    file_schema: Arc<Schema>,
    insert_range: TimeRange,
    source_data_hash: Vec<u8>,
    sort_order: Option<Vec<String>>,
    retire_match: RetireMatch,
    same_run_ranges: Vec<TimeRange>,
    rb_stream: Receiver<Result<PartitionRowSet, Error>>,
    logger: Arc<dyn Logger>,
) -> Result<(), Error>
Expand description

Writes a partition to a Parquet file from a stream of PartitionRowSets.

sort_order is recorded on the resulting Partition as-is (see View::get_merged_partition_sort_order and MetadataPartitionSpec::sort_order).

retire_match is forwarded to retire_partitions (see RetireMatch’s docs): pass RetireMatch::Containment unless this call is one of the two BlockOrder::EventTime JIT views (thread_spans_view.rs / net_spans_view.rs), which must pass RetireMatch::Overlap.

same_run_ranges is forwarded to retire_partitions as-is (see its docs and RetireMatch::Overlap’s); pass an empty Vec unless retire_match is RetireMatch::Overlap, in which case it must list every insert range the calling jit_update run has already written (or found already up to date) earlier in its own loop. Taken by value (not &[TimeRange]): this whole call is normally wrapped in spawn_with_context, whose Future bound requires 'static, which a slice borrowed from the caller’s loop-local accumulator cannot satisfy.