pub fn group_blocks_into_partitions(
config: &JitPartitionConfig,
blocks: Vec<Arc<PartitionSourceBlock>>,
) -> Vec<SourceDataBlocksInMemory>Expand description
Groups a segment’s source blocks into JIT partitions.
Two invariants must both hold across the returned partitions:
- Size –
max_nb_objectsis a soft, whole-block-granularity limit (a block is never split, so a single oversized block already exceeds it). - Insert-time non-overlap – the
lakehouse_partitions_no_overlapexclusion constraint requires each partition’s[min_insert_time, max_insert_time]to be non-overlapping with (and non-decreasing relative to) every other partition’s. Adjacent partitions may touch (one’s max equals the next’s min): the constraint’ststzrangeis half-open[), so that is not an overlap. The one shape this does not cover is two or more partitions sharing an identical degenerate range (every block in both carrying the sameinsert_time), which requires more thanmax_nb_objectsobjects at a single microsecond and so does not occur in production;tstzrange(t, t)is empty, so the constraint stays quiet, but theBlockOrder::EventTimearm ofis_jit_partition_up_to_datewould see more than one row for that range and never report the partitions up to date.
Under BlockOrder::InsertTime both invariants hold for free: blocks is already sorted by
insert_time, so every candidate cut point is insert-safe and this function cuts at exactly
max_nb_objects – identical to a naive greedy cut except that an all-zero-object window emits
nothing rather than a rows-free partition (see emit_partition; unreachable in practice).
Under BlockOrder::EventTime, blocks is stable-sorted by (begin_ticks, end_ticks) first
(ties keep the incoming, insert-ordered position, so grouping stays deterministic); a cut point
is then only taken where it is insert-safe: every block already in the partition being closed
must have an insert_time no later than every remaining block’s, computed via a suffix-minimum
over the event-time-sorted list. When the natural max_nb_objects cut point isn’t safe, the cut
looks back to the most recent safe index; when no safe index exists in the window, the window
grows past the soft limit and a debug! logs it.