Module jit_partitions

Expand description

Management of process-specific partitions built on demand Just-in-time (JIT) partition generation groups a view’s source blocks (queried from blocks_view in ORDER BY insert_time, block_id order) into SourceDataBlocksInMemory partitions no larger than JitPartitionConfig::max_nb_objects.

JitPartitionConfig::block_order (see BlockOrder) picks between two orderings for the blocks a partition ends up holding: InsertTime (the SQL order, unchanged) for views that decode blocks independently, and EventTime for views that build cross-block trees or rely on event-time-sorted rows within a partition file. group_blocks_into_partitions is the single place both variants are cut, and it upholds one invariant regardless of ordering: every emitted partition’s [min_insert_time, max_insert_time] range is non-overlapping with, and non-decreasing relative to, every other emitted partition’s – the precondition the lakehouse_partitions_no_overlap exclusion constraint enforces at insert time. See group_blocks_into_partitions’s own docs for how that invariant is upheld under BlockOrder::EventTime, where the natural size-based cut point is not always safe, and for the one (production-unreachable) shape it does not cover.

Note that this is an insert-time invariant only. It says nothing about partitions’ event-time ranges, which for the block-derived views are computed from block begin_ticks/end_ticks. By design both micromegas_tracing and the Unreal producer now stamp a single shared timestamp per flush, used for both the outgoing block’s close and the replacement block’s begin, so consecutive blocks touch exactly and adjacent partitions’ event bounds do not overlap by construction. Data from producer versions predating that fix can still carry strictly-overlapping block ticks; ThreadSpansView’s recorded max_sort_key_time is what makes that harmless to the Concatenated scan check (partitioned_execution_plan::partition_bounds prefers it over the looser max_event_time) rather than requiring every source block to be re-ingested. See the ordering-invariant notes on View::get_scan_output_ordering and tasks/completed/thread_spans_segment_boundary_overlap_plan.md.

Structs§

JitPartitionConfig: Configuration for Just-In-Time (JIT) partition generation.
PartitionFreshnessRow: One lakehouse_partitions candidate row, as fetched by fetch_freshness_candidates: (begin_insert_time, end_insert_time, file_schema_hash, source_data_hash). Fields are pub so analytics/tests/jit_freshness_tests.rs can build rows directly, without a live database.

Enums§

BlockOrder: How source blocks are ordered before being cut into JIT partitions.

Functions§

batch_windows: Packs consecutive insert-time buckets of [insert_range.begin, insert_range.end) (stepping by slice, so every bucket edge is slice-aligned to insert_range.begin) into batch query windows, greedily closing a window just before it would exceed target_rows_per_query.
blocks_insert_time_range: Returns the real [min, max] insert_time range covered by blocks, independent of list order. Always use this instead of reading blocks[0]/blocks[last]: under BlockOrder::EventTime the list is sorted by event time, not insert time, so its endpoints are no longer the insert-time extremes.
fetch_process_blocks: Fetches and parses one batch window’s worth of process-scoped blocks (process_batch_sql), looking up each block’s stream metadata in stream_metadata (built once per call to generate_process_jit_partitions by fetch_stream_metadata_map) rather than rebuilding it per row – this is the lean projection’s fetch-and-parse half.
find_up_to_date_partitions: One candidates fetch over [specs.first().min, specs.last().max] (specs have ascending, non-overlapping insert ranges), then the matcher per spec, run to a fixpoint. An empty specs returns Ok(vec![]) without issuing any fetch – reachable at every call site, since generate_*_jit_partitions returns no specs when the range holds no blocks.
generate_process_jit_partitions: generate_process_jit_partitions lists the partitions that are needed to cover a time span for a specific process these partitions may not exist or they could be out of date Generates JIT partitions for a given time range filtered by process.
generate_stream_jit_partitions: generate_stream_jit_partitions lists the partitiions that are needed to cover a time span these partitions may not exist or they could be out of date Generates JIT partitions for a given time range.
generate_stream_jit_partitions_segment: Generates a segment of JIT partitions.
group_blocks_into_partitions: Groups a segment’s source blocks into JIT partitions.
group_contiguous_block_chains: Splits an event-time-ordered block list into maximal chains whose tick coverage is unbroken, so each chain can be decoded into one cross-block tree (a call tree for thread_spans, a net span tree for net_spans).
is_jit_partition_up_to_date: is_jit_partition_up_to_date compares a partition spec with the partitions that exist to know if it should be recreated Checks if a JIT partition is up to date.
process_batch_sql: The process-variant batch query: block columns plus stream_id only – no stream-level column (streams.dependencies_metadata/objects_metadata/tags/properties/format) may be added to the SELECT list. Re-adding one would reintroduce the per-row stream-blob-copy memory hazard the lean projection removes, with no test failing except the projection guard in analytics/tests/jit_batch_windows_tests.rs. array_has("streams.tags", ...) stays in the WHERE clause: filtering needs no projection.
resolve_up_to_date_fixpoint: The pure fixpoint at the heart of find_up_to_date_partitions (see that function’s docs for the derivation): runs spec_is_up_to_date once for every spec against candidates, then drains a worklist of specs that might need re-evaluation because a neighbor newly became stale. Because specs have ascending, non-overlapping insert ranges, a candidate row can be entirely contained in at most two specs’ ranges, and only when those two specs are adjacent in the sorted order (sharing a boundary point, for a degenerate begin == end row) – so the only specs whose verdict can ever change as a consequence of spec i becoming stale are i’s immediate neighbors, index i - 1 and i + 1. A spec is re-evaluated against a candidate set that drops rows entirely contained in some other, currently-stale spec’s insert range (never its own range – a row a spec would itself retire is not evidence about its own freshness); if that recompute still comes back stale, the spec is marked stale (it is never recomputed again – once stale, verdicts are monotone by construction, with no separate clamping step needed) and its own neighbors are pushed onto the worklist in turn. Takes an already-fetched candidates and does no I/O, so analytics/tests/jit_freshness_tests.rs can drive it directly without a live PgPool.
spec_is_up_to_date: Filters candidates down to the rows the per-spec SQL used to return (exact range equality for BlockOrder::EventTime, exact match for a degenerate BlockOrder::InsertTime range, inclusive overlap otherwise), then applies the same rows.len() == 1 / file_schema_hash / object-count checks is_jit_partition_up_to_date always has.
write_partition_from_blocks: Writes a partition from a set of blocks.