pub struct Partition {
pub view_metadata: ViewMetadata,
pub insert_time_range: TimeRange,
pub event_time_range: Option<TimeRange>,
pub updated: DateTime<Utc>,
pub file_path: Option<String>,
pub file_size: i64,
pub source_data_hash: Vec<u8>,
pub num_rows: i64,
pub sort_order: Option<Vec<String>>,
pub max_sort_key_time: Option<DateTime<Utc>>,
}Expand description
Partition metadata (without embedded file_metadata for performance) Use load_partition_metadata() to load metadata on-demand when needed
Fields§
§view_metadata: ViewMetadataMetadata about the view this partition belongs to.
insert_time_range: TimeRangeThe insert time range for this partition.
event_time_range: Option<TimeRange>The event time range for this partition. None for empty partitions.
updated: DateTime<Utc>The last time this partition was updated.
file_path: Option<String>The path to the Parquet file for this partition. None for empty partitions.
file_size: i64The size of the Parquet file in bytes. 0 for empty partitions.
source_data_hash: Vec<u8>A hash of the source data that generated this partition.
num_rows: i64The number of rows in this partition. 0 for empty partitions.
sort_order: Option<Vec<String>>The sort guarantee this partition’s rows carry, if any (e.g. Some(["insert_time"])).
None means no ordering guarantee is recorded – true for every partition written before
this field existed, and for any view/merge that doesn’t declare one. See
View::get_merged_partition_sort_order and the blocks_view_ordered_merges_plan.md plan.
max_sort_key_time: Option<DateTime<Utc>>The true maximum value of the view’s declared Concatenated leading sort column across
this partition’s rows (begin, for thread_spans) – as opposed to max_event_time,
which for thread_spans is the max span end and only a loose stand-in for it. None
means “not recorded”: every partition written before this field existed, and every
partition from a view that doesn’t declare a Concatenated event-time ordering (only
ThreadSpansView populates it today). See
tasks/completed/thread_spans_segment_boundary_overlap_plan.md, Part B.
Implementations§
Source§impl Partition
impl Partition
Sourcepub fn min_event_time(&self) -> Option<DateTime<Utc>>
pub fn min_event_time(&self) -> Option<DateTime<Utc>>
Returns the min event time, if this partition has data.
Sourcepub fn max_event_time(&self) -> Option<DateTime<Utc>>
pub fn max_event_time(&self) -> Option<DateTime<Utc>>
Returns the max event time, if this partition has data.
Sourcepub fn max_sort_key_time(&self) -> Option<DateTime<Utc>>
pub fn max_sort_key_time(&self) -> Option<DateTime<Utc>>
Returns the recorded true maximum leading-sort-column value, if any. See the field’s doc
for what “recorded” means and why it can be None even for a non-empty partition.
Sourcepub fn begin_insert_time(&self) -> DateTime<Utc>
pub fn begin_insert_time(&self) -> DateTime<Utc>
Returns the beginning of the insert time range.
Sourcepub fn end_insert_time(&self) -> DateTime<Utc>
pub fn end_insert_time(&self) -> DateTime<Utc>
Returns the end of the insert time range.
Sourcepub fn certifies_sort_order(&self, columns: &[ScanSortColumn]) -> bool
pub fn certifies_sort_order(&self, columns: &[ScanSortColumn]) -> bool
True when this partition’s recorded sort_order certifies columns as an ordering its
rows already satisfy: columns (ascending-only) must name a prefix of the recorded,
ascending-implied sort order. Empty partitions certify vacuously – there are no rows to
violate the ordering. An empty (non-empty-partition) columns never certifies: it names no
ordering to satisfy, so declaring it certified would let a ScanOrdering::PerFile { columns: vec![] } degrade to Unordered instead of silently planning and recording a vacuous
ordering (see ScanOrdering::PerFile).
Sourcepub fn validate(&self) -> Result<(), Error>
pub fn validate(&self) -> Result<(), Error>
Validates partition invariants. Returns error if partition is inconsistent.
Invariants:
- Non-empty partitions (num_rows > 0) MUST have both event_time_range and file_path
- Empty partitions (num_rows = 0) MUST NOT have event_time_range or file_path
- num_rows must not be negative
- A partition with no event_time_range MUST NOT carry a max_sort_key_time (an empty partition has no sort-key bound to record)
Deliberately not checked here: any ordering relationship between max_sort_key_time
and min_event_time/max_event_time (e.g. min_event_time <= max_sort_key_time <= max_event_time). That holds by construction for thread_spans today, but two properties
of this function rule out asserting it as an invariant:
validateruns only on the read path (its callers are all inpartition_cache.rs, propagating with?); nothing on the write path calls it. A violating partition would therefore be written and committed silently, then hard-fail reads – and since the fetch this feeds is not view-scoped, one bad row would fail materialization for every view, not just the offending one. The invariant must assert only what the writer guarantees by construction, never a hopeful expectation.- The bound would depend on data the writer doesn’t control:
min_event_time/max_event_timederive straight from client-supplied block ticks with no validation anywhere in ingestion or analytics, so a malformed client (e.g.begin_ticks > end_ticks) could make an ordering clause unsatisfiable – turning one bad row into a lakehouse-wide read outage. And the invariant would apply to every future view populatingmax_sort_key_time, not justthread_spans:BlocksView, for instance, has aConcatenatedsort key ofinsert_timewhile itsevent_time_rangeis[min begin_time, max insert_time], so a client clock skewed ahead of the server could put a block’sbegin_timeabove itsinsert_time.
The row-level property this plan actually relies on (the partition’s true max begin
really is bounded by its event_time_range) is asserted where it is cheap and safe
instead: at write time (ensure_begin_non_decreasing) and pinned by a no-DB test
(call_tree_tests.rs).
Trait Implementations§
Auto Trait Implementations§
impl Freeze for Partition
impl RefUnwindSafe for Partition
impl Send for Partition
impl Sync for Partition
impl Unpin for Partition
impl UnsafeUnpin for Partition
impl UnwindSafe for Partition
Blanket Implementations§
impl<T> Allocation for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
§fn with_current_context(self) -> WithContext<Self> ⓘ
fn with_current_context(self) -> WithContext<Self> ⓘ
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request§impl<L> LayerExt<L> for L
impl<L> LayerExt<L> for L
§fn named_layer<S>(&self, service: S) -> Layered<<L as Layer<S>>::Service, S>where
L: Layer<S>,
fn named_layer<S>(&self, service: S) -> Layered<<L as Layer<S>>::Service, S>where
L: Layer<S>,
Layered].