Skip to main content

generate_process_jit_partitions

Function generate_process_jit_partitions 

Source
pub fn generate_process_jit_partitions(
    config: &JitPartitionConfig,
    lakehouse: Arc<LakehouseContext>,
    blocks_view: &BlocksView,
    query_time_range: &TimeRange,
    process: Arc<ProcessMetadata>,
    stream_tag: &str,
) -> impl Future<Output = Result<Vec<SourceDataBlocksInMemory>, Error>>
Expand description

generate_process_jit_partitions lists the partitions that are needed to cover a time span for a specific process these partitions may not exist or they could be out of date Generates JIT partitions for a given time range filtered by process.

Batches its block queries the same way generate_stream_jit_partitions does, and additionally applies the lean projection: fetch_stream_metadata_map fetches every stream’s metadata once for the whole insert range, and each batch’s fetch_process_blocks call looks it up per block instead of projecting stream blobs onto every row and rebuilding StreamMetadata per row. The grouping call is inlined into the batch loop below.