Skip to main content

generate_stream_jit_partitions

Function generate_stream_jit_partitions 

Source
pub fn generate_stream_jit_partitions(
    config: &JitPartitionConfig,
    lakehouse: Arc<LakehouseContext>,
    blocks_view: &BlocksView,
    query_time_range: &TimeRange,
    stream: Arc<StreamMetadata>,
    process: Arc<ProcessMetadata>,
) -> impl Future<Output = Result<Vec<SourceDataBlocksInMemory>, Error>>
Expand description

generate_stream_jit_partitions lists the partitiions that are needed to cover a time span these partitions may not exist or they could be out of date Generates JIT partitions for a given time range.

Batches its block queries: batch_windows, derived from a per-bucket COUNT(*), picks how many insert-time buckets one query covers, and each batch’s rows are then split back into per-bucket runs (split_into_buckets) and grouped independently – byte-identical to running generate_stream_jit_partitions_segment once per bucket, just fewer, wider queries to get there. generate_stream_jit_partitions_segment itself is kept, not called from here anymore.