Skip to main content

write_rows_and_track_times

Function write_rows_and_track_times 

Source
pub async fn write_rows_and_track_times(
    rb_stream: &mut Receiver<Result<PartitionRowSet, Error>>,
    arrow_writer: &mut AsyncArrowWriter<AsyncParquetWriter>,
    file_schema: &Schema,
    logger: &Arc<dyn Logger>,
    desc: &str,
) -> Result<RowSetTrackingResult, Error>
Expand description

Writes rows from the stream and tracks event time ranges.

Also folds a running max_sort_key_time across every received PartitionRowSet: the partition-level value is Some(max) only if every received row set carried Some – a single None poisons the whole partition to None. This must be a running max, never “last row set wins”: this function is shared by four multi-row-set senders, and BlockPartitionSpec in particular streams row sets out of order via buffer_unordered (block_partition_spec.rs), where “last wins” would be actively wrong. thread_spans_view.rs sends exactly one row set today, so the distinction is invisible there, but do not let that fact simplify this fold away.