Skip to main content

group_blocks_into_partitions

Function group_blocks_into_partitions 

Source
pub fn group_blocks_into_partitions(
    config: &JitPartitionConfig,
    blocks: Vec<Arc<PartitionSourceBlock>>,
) -> Vec<SourceDataBlocksInMemory>
Expand description

Groups a segment’s source blocks into JIT partitions.

Two invariants must both hold across the returned partitions:

  • Size – max_nb_objects is a soft, whole-block-granularity limit (a block is never split, so a single oversized block already exceeds it).
  • Insert-time non-overlap – the lakehouse_partitions_no_overlap exclusion constraint requires each partition’s [min_insert_time, max_insert_time] to be non-overlapping with (and non-decreasing relative to) every other partition’s. Adjacent partitions may touch (one’s max equals the next’s min): the constraint’s tstzrange is half-open [), so that is not an overlap. The one shape this does not cover is two or more partitions sharing an identical degenerate range (every block in both carrying the same insert_time), which requires more than max_nb_objects objects at a single microsecond and so does not occur in production; tstzrange(t, t) is empty, so the constraint stays quiet, but the BlockOrder::EventTime arm of is_jit_partition_up_to_date would see more than one row for that range and never report the partitions up to date.

Under BlockOrder::InsertTime both invariants hold for free: blocks is already sorted by insert_time, so every candidate cut point is insert-safe and this function cuts at exactly max_nb_objects – identical to a naive greedy cut except that an all-zero-object window emits nothing rather than a rows-free partition (see emit_partition; unreachable in practice).

Under BlockOrder::EventTime, blocks is stable-sorted by (begin_ticks, end_ticks) first (ties keep the incoming, insert-ordered position, so grouping stays deterministic); a cut point is then only taken where it is insert-safe: every block already in the partition being closed must have an insert_time no later than every remaining block’s, computed via a suffix-minimum over the event-time-sorted list. When the natural max_nb_objects cut point isn’t safe, the cut looks back to the most recent safe index; when no safe index exists in the window, the window grows past the soft limit and a debug! logs it.