Skip to main content

Module partitioned_execution_plan

Module partitioned_execution_plan 

Source
Expand description

ExecutionPlan based on a set of parquet files

Enums§

OrderingBounds
Which pair of bounds on Partition a declared ordering’s leading column is checked against.
ScanOrdering
How a partition scan’s declared output ordering is realized.

Functions§

assert_ordering_satisfied
Bails unless plan’s output ordering satisfies the declared columns, defensively re-verifying that a sort_order guarantee about to be recorded on a fresh partition is truthful. label is a short noun phrase used in the intermediate error-context messages (e.g. "per-file merge" or "extract-query"); subject names the query for the bail message; reason supplies the full, call-site-specific trailing text of the bail message (what was declared, and any guidance for diagnosing a mismatch). Shared by the two paths that record a sort_order guarantee: QueryMerger::execute_sorted_merge and sql_partition_spec::plan_sorted_extract. QueryMerger::execute_concatenated_merge does not call this – its ordering is a structural property of the sorted, non-overlapping file group rather than a query-plan sort DataFusion could get wrong.
assert_single_partition
Bails unless plan is a single-partition physical plan. subject names the query for the error message (e.g. format!("merge query {:?}", query) or format!("extract query for {view}")); reason supplies the full, call-site-specific trailing sentence(s) explaining what a non-single-partition plan means here – executing such a plan would coalesce partitions and silently destroy a declared ordering before it is safe to record or execute. Shared by the query-execution paths that must verify this before executing: QueryMerger::execute_concatenated_merge (only when its ordering is declared), QueryMerger::execute_sorted_merge, and sql_partition_spec::plan_sorted_extract.
make_lex_ordering
Builds the LexOrdering declaring the already-satisfied output ordering of the scan, matching DataFusion’s default ORDER BY semantics (ASC NULLS LAST unless descending).
make_partitioned_execution_plan
Creates a partitioned execution plan for scanning Parquet files.
sort_exprs
Builds the DataFrame::sort expressions for columns, pairing each with !descending for asc so the logical sort matches the nulls_first: c.descending physical ordering make_lex_ordering declares below – the two must stay in lockstep or assert_ordering_satisfied stops matching the sort that was actually applied.