pub async fn make_merge_session_context(
lakehouse: Arc<LakehouseContext>,
part_provider: Arc<dyn QueryPartitionProvider>,
query_range: Option<TimeRange>,
view_factory: Arc<ViewFactory>,
configurator: Arc<dyn SessionConfigurator>,
caller: CallerContext,
) -> Result<SessionContext, Error>Expand description
A merge is a bounded, streaming rewrite of a fixed set of files whose consumer is a single
writer task (create_merged_partition -> write_partition_from_rows), not a query that
benefits from scan parallelism. Left at DataFusion’s default (true), EnsureRequirements
splits the source scan into target_partitions byte-range file groups and execute_stream
coalesces them, so the reader working set is multiplied by the host’s core count for no
throughput the writer can absorb. Forcing one sequential file group here fixes that for the
Unordered and Concatenated shapes, which both start from a single file group spanning every
input partition: left unsplit, one reader works through that group’s files one at a time instead
of target_partitions concurrent byte-range readers. PerFile already builds one file group per
input partition, and this setting is load-bearing there too: left at true,
repartition_file_groups (via repartition_preserving_order) splits each of those k
per-partition groups into target_partitions byte-range groups whenever k < target_partitions,
so this setting is what keeps PerFile at one reader per input file for its k-way ordered merge.
Downstream parallelism is untouched: a merge query with a GROUP BY still gets its round-robin
fan-out above the scan.