Module merge 

Merge consecutive parquet partitions into a single file

Structs

MergeQueryResult

The outcome of running a merge query.

QueryMerger

A PartitionMerger that executes a SQL query to merge partitions.

Traits

PartitionMerger

A trait for merging partitions.

Functions

create_merged_partition

Creates a merged partition from a set of existing partitions.

make_merge_session_context

A merge is a bounded, streaming rewrite of a fixed set of files whose consumer is a single writer task (create_merged_partition -> write_partition_from_rows), not a query that benefits from scan parallelism. Left at DataFusion’s default (true), EnsureRequirements splits the source scan into target_partitions byte-range file groups and execute_stream coalesces them, so the reader working set is multiplied by the host’s core count for no throughput the writer can absorb. Forcing one sequential file group here fixes that for the Unordered and Concatenated shapes, which both start from a single file group spanning every input partition: left unsplit, one reader works through that group’s files one at a time instead of target_partitions concurrent byte-range readers. PerFile already builds one file group per input partition, and this setting is load-bearing there too: left at true, repartition_file_groups (via repartition_preserving_order) splits each of those k per-partition groups into target_partitions byte-range groups whenever k < target_partitions, so this setting is what keeps PerFile at one reader per input file for its k-way ordered merge. Downstream parallelism is untouched: a merge query with a GROUP BY still gets its round-robin fan-out above the scan.