Skip to main content

make_merge_session_context

Function make_merge_session_context 

Source
pub async fn make_merge_session_context(
    lakehouse: Arc<LakehouseContext>,
    part_provider: Arc<dyn QueryPartitionProvider>,
    query_range: Option<TimeRange>,
    view_factory: Arc<ViewFactory>,
    configurator: Arc<dyn SessionConfigurator>,
    caller: CallerContext,
) -> Result<SessionContext, Error>
Expand description

A merge is a bounded, streaming rewrite of a fixed set of files whose consumer is a single writer task (create_merged_partition -> write_partition_from_rows), not a query that benefits from scan parallelism. Left at DataFusion’s default (true), EnsureRequirements splits the source scan into target_partitions byte-range file groups and execute_stream coalesces them, so the reader working set is multiplied by the host’s core count for no throughput the writer can absorb. Forcing one sequential file group here fixes that for the Unordered and Concatenated shapes, which both start from a single file group spanning every input partition: left unsplit, one reader works through that group’s files one at a time instead of target_partitions concurrent byte-range readers. PerFile already builds one file group per input partition, and this setting is load-bearing there too: left at true, repartition_file_groups (via repartition_preserving_order) splits each of those k per-partition groups into target_partitions byte-range groups whenever k < target_partitions, so this setting is what keeps PerFile at one reader per input file for its k-way ordered merge. Downstream parallelism is untouched: a merge query with a GROUP BY still gets its round-robin fan-out above the scan.