Skip to main content

Module ownership_rewrite

Module ownership_rewrite 

Source
Expand description

The row-level filter: injects an audience predicate into every MaterializedView scan based on the caller’s ReadScope. OwnershipRewrite – the row-level filter.

An AnalyzerRule that injects an audience-filtering predicate into every MaterializedView-backed TableScan a query plan touches, based on the caller’s ReadScope (read_scope.rs) and crate::lakehouse::read_scope::IsolationConfig’s MICROMEGAS_PUBLIC_VIEW_SETS.

§ReadScope::All is a true no-op

ReadScope::All is the internal/maintenance marker – never the output of a real ReadPolicy (see read_scope.rs). analyze() returns the plan unchanged when constructed with it, and query.rs::make_session_context does not even construct this rule for ReadScope::All callers, so their ViewFactory need not carry processes/streams either.

§The physical audience column

blocks, processes, streams, log_entries, measures, and log_stats each carry a physical audience column – declared non-nullable on blocks/log_entries/measures, and non-null by construction (though declared nullable in their inferred SqlBatchView schema) on processes/streams/log_stats – extracted once from Postgres’s own audience column at the blocks view’s materialization (its own row’s stamp) and propagated structurally into every downstream view. A DDL-defined materialized view carrying its own audience column (enforced at CREATE time, see view_definition.rs) joins this group too. Every view carrying an audience column is consequently filtered with a bare Filter on that column (the column-filter branch in the table below): no semi-join, no property_get, no per-process aggregate. async_events, thread_spans, net_spans, otel_spans, and images don’t carry the column. They keep the process_id/EXISTS machinery below, which still resolves through __processes__partitions, but only for a scan that is not a guarded view_instance(...) – see the skip rule in the Branch table below.

§One audience per row, not derived from the process (surviving branches only)

__processes__partitions (the raw, un-merged SqlBatchView partitions self.processes_source scans) can carry more than one historical row per process_id – the partition-level GROUP BY in processes_view.rs’s transform query only collapses rows within a partition, not across a long-lived process’s entire history. Filtering the views carrying an audience column, DDL-defined ones included, one row at a time (the column-filter branch) is sound without any aggregate: every row now carries its own stamp from the moment it was written, so there is nothing to reconcile across rows of the same process. The one exception is a legacy, pre-v8 row with no audience column at all – COALESCE resolves that to the deployment default at read time. The net_spans/ otel_spans/images semi-join branch and the async_events/thread_spans EXISTS branches still resolve through per_process_audience – Aggregate(GROUP BY process_id, MAX(audience) AS resolved_audience) – for a scan of those five that is not a guarded instance, since they have no audience column of their own to filter directly. A guarded instance skips the predicate entirely (see the Branch table below).

§Branch table

Keyed on MaterializedView::get_view().get_file_schema(): whether it has an audience field, failing that whether it has a process_id field – not on a hardcoded view-set list, so a future view set that gains either column falls into the matching branch automatically:

View setBranch
processes, streams, blocks, log_entries, measures, log_stats, and any DDL-defined view carrying an audience columncolumn filter: direct audience IN (...) filter on the view’s own column – no join
net_spans, otel_spans, images, async_events, thread_spans, scanned as a guarded, non-'global' view_instance(...)skip: no predicate at all – MaterializedView::instance_is_audience_guarded() reports AudienceGuard::authorize_view_instance already denies an unauthorized caller before any row is read, so the subquery predicates below would be redundant
net_spans, otel_spans, images, not a guarded instancesemi-join: process_id IN (subquery) against per_process_audience (outer process_id cast to Utf8: it is Dictionary(Int32, Utf8) in these, and nothing coerces an uncorrelated IN subquery’s join keys once DecorrelatePredicateSubquery turns it into a LeftSemi join – the analyzer’s own TypeCoercion has already run by the time this rule executes)
async_events, not a guarded instanceno process_id column either – async_events EXISTS: literal-valued EXISTS, keyed on get_view_instance_id() (the process_id string, canonicalized – see canonical_view_instance_id)
thread_spans, not a guarded instanceno process_id or stream_id column – thread_spans EXISTS: two-hop literal EXISTS, through streams (get_view_instance_id() is the stream_id string, canonicalized the same way)
anything in public_view_setspublic: no predicate at all – checked before any of the above
anything elseanalyze() returns Err (DataFusionError::Plan) rather than silently leaving the scan unfiltered – a future view set must add itself to this table, not fall through

Adding a new view set to a ViewFactory means adding it to this table too, and to the match arms in OwnershipRewrite::predicate_for below.

§No I/O in analyze()

in_subquery/exists build uncorrelated subquery Exprs; DecorrelatePredicateSubquery (an optimizer, not analyzer, rule) turns them into joins later. processes_source/streams_source come from MaterializedView::new(..., query_range: None), which is synchronous and does no I/O – the actual processes/streams scans happen during normal execution, exactly like TableScanRewrite’s injected time filter.

§The audience column is server-written and authenticated

processes, streams, and blocks are each stamped with their own audience column at write time, from the authenticated credential’s AuthContext.bound_audience (resolve_write_audience), and a client can neither assert nor forge that stamp (strip_reserved_properties drops any client-supplied micromegas.* property, and there is no property to re-assert in the first place any more – the stamp is a physical column, not a property). A credential carrying no bound audience resolves to the deployment default at the HTTP write edge and is stamped with it explicitly, the same as any other audience – only a legacy, pre-v8 row keeps a NULL column, resolved to the deployment’s MICROMEGAS_DEFAULT_AUDIENCE where the audience is read instead (see crate::audience’s module doc). Admin replication (replication.rs) now hard-fails on a missing audience column rather than ever writing a row with none. Registration (insert_process/register_otel_process) rejects a same-process_id, different-audience re-registration outright – needed because the OTLP process_id derivation formula is public, so without this guard a credential could pre-register (via the native path) the exact process_id a victim audience’s OTLP producer would later derive, silently exposing that audience’s data to the squatter; this is a confidentiality gap, not merely an integrity one. The stream-side mirror, check_stream_audience_conflict, closes the equivalent re-pointed-credential gap for streams.

What remains open, tracked separately:

  • The five instance-only view sets (net_spans, otel_spans, images, async_events, thread_spans) and the per-process JIT view_instance path still resolve their audience label through the owning process’s/stream’s row rather than a genuine per-row column of their own – via the call-level guard’s instance check (AudienceGuard::authorize_view_instance, IdKind::ProcessOrStream) for the guarded view_instance(...) scan every production caller takes, not by a predicate this rule injects (see the skip in the Branch table above). The cross-audience injection scenario (an attacker’s block naming a victim’s process_id/stream_id) is closed for both, as a side effect of the mismatch predicate on blocks_view’s own materialization – except against a victim whose processes/streams row is a legacy, pre-v8 NULL-audience row (next bullet).
  • The NULL-anchor window. A processes/streams row registered before its ingestion binary supported per-row stamping keeps audience = NULL for its entire remaining life (rows are immutable, no backfill). The mismatch predicate’s NULL-tolerant pass-through lets a mismatched block through unchecked against such a row for as long as it exists. This is an accepted, bounded limitation over already-public legacy data, bounded by the deploy-order requirement (every ingestion replica upgraded) and shrinking as legacy rows age out under retention.

There is no in-product enforcement knob left for either surface; the mitigation is operational – provision only audience-bound DB-backed ingestion credentials, and don’t run ingestion with OIDC or --disable-auth alongside them – see this stage’s CHANGELOG.md entry.

Structs§

OwnershipRewrite
Injects an audience predicate into every MaterializedView scan (see the module doc comment).