Expand description
The row-level filter: injects an audience predicate into every
MaterializedView scan based on the caller’s ReadScope.
OwnershipRewrite – the row-level filter.
An AnalyzerRule that injects an audience-filtering predicate into every
MaterializedView-backed TableScan a query plan touches, based on the caller’s ReadScope
(read_scope.rs) and crate::lakehouse::read_scope::IsolationConfig’s
MICROMEGAS_PUBLIC_VIEW_SETS.
§ReadScope::All is a true no-op
ReadScope::All is the internal/maintenance marker – never the output of a real ReadPolicy
(see read_scope.rs). analyze() returns the plan unchanged when constructed with it, and
query.rs::make_session_context does not even construct this rule for ReadScope::All
callers, so their ViewFactory need not carry processes/streams either.
§The physical audience column
blocks, processes, streams, log_entries, measures, and log_stats each carry a
physical audience column – declared non-nullable on blocks/log_entries/measures, and
non-null by construction (though declared nullable in their inferred SqlBatchView schema) on
processes/streams/log_stats – extracted once from Postgres’s own audience column at
the blocks view’s materialization (its own row’s stamp) and propagated structurally into
every downstream view. A DDL-defined materialized view carrying its own audience column
(enforced at CREATE time, see view_definition.rs) joins this group too. Every view
carrying an audience column is consequently filtered with a bare Filter on that column
(the column-filter branch in the table below): no semi-join, no property_get, no
per-process aggregate. async_events, thread_spans, net_spans, otel_spans, and images
don’t carry the column. They keep the process_id/EXISTS machinery below, which still
resolves through __processes__partitions, but only for a scan that is not a guarded
view_instance(...) – see the skip rule in the Branch table below.
§One audience per row, not derived from the process (surviving branches only)
__processes__partitions (the raw, un-merged SqlBatchView partitions self.processes_source
scans) can carry more than one historical row per process_id – the partition-level
GROUP BY in processes_view.rs’s transform query only collapses rows within a partition,
not across a long-lived process’s entire history. Filtering the views carrying an audience
column, DDL-defined ones included, one row at a time (the column-filter branch) is sound
without any aggregate: every row now carries
its own stamp from the moment it was written, so there is nothing to reconcile across rows of
the same process. The one exception is a legacy, pre-v8 row with no audience column at all –
COALESCE resolves that to the deployment default at read time. The net_spans/
otel_spans/images semi-join branch and the async_events/thread_spans EXISTS branches
still resolve through per_process_audience – Aggregate(GROUP BY process_id, MAX(audience) AS resolved_audience) – for a scan of those five that is not a guarded instance, since they
have no audience column of their own to filter directly. A guarded instance skips the
predicate entirely (see the Branch table below).
§Branch table
Keyed on MaterializedView::get_view().get_file_schema(): whether it has an audience field,
failing that whether it has a process_id field – not on a hardcoded view-set list, so a
future view set that gains either column falls into the matching branch automatically:
| View set | Branch |
|---|---|
processes, streams, blocks, log_entries, measures, log_stats, and any DDL-defined view carrying an audience column | column filter: direct audience IN (...) filter on the view’s own column – no join |
net_spans, otel_spans, images, async_events, thread_spans, scanned as a guarded, non-'global' view_instance(...) | skip: no predicate at all – MaterializedView::instance_is_audience_guarded() reports AudienceGuard::authorize_view_instance already denies an unauthorized caller before any row is read, so the subquery predicates below would be redundant |
net_spans, otel_spans, images, not a guarded instance | semi-join: process_id IN (subquery) against per_process_audience (outer process_id cast to Utf8: it is Dictionary(Int32, Utf8) in these, and nothing coerces an uncorrelated IN subquery’s join keys once DecorrelatePredicateSubquery turns it into a LeftSemi join – the analyzer’s own TypeCoercion has already run by the time this rule executes) |
async_events, not a guarded instance | no process_id column either – async_events EXISTS: literal-valued EXISTS, keyed on get_view_instance_id() (the process_id string, canonicalized – see canonical_view_instance_id) |
thread_spans, not a guarded instance | no process_id or stream_id column – thread_spans EXISTS: two-hop literal EXISTS, through streams (get_view_instance_id() is the stream_id string, canonicalized the same way) |
anything in public_view_sets | public: no predicate at all – checked before any of the above |
| anything else | analyze() returns Err (DataFusionError::Plan) rather than silently leaving the scan unfiltered – a future view set must add itself to this table, not fall through |
Adding a new view set to a ViewFactory means adding it to this table too, and to the match
arms in OwnershipRewrite::predicate_for below.
§No I/O in analyze()
in_subquery/exists build uncorrelated subquery Exprs; DecorrelatePredicateSubquery (an
optimizer, not analyzer, rule) turns them into joins later. processes_source/streams_source
come from MaterializedView::new(..., query_range: None), which is synchronous and does no I/O
– the actual processes/streams scans happen during normal execution, exactly like
TableScanRewrite’s injected time filter.
§The audience column is server-written and authenticated
processes, streams, and blocks are each stamped with their own audience column at
write time, from the authenticated credential’s AuthContext.bound_audience
(resolve_write_audience), and a client can neither assert nor forge that stamp
(strip_reserved_properties drops any client-supplied micromegas.* property, and there is
no property to re-assert in the first place any more – the stamp is a physical column, not a
property). A credential carrying no bound audience resolves to the deployment default at the
HTTP write edge and is stamped with it explicitly, the same as any other audience – only a
legacy, pre-v8 row keeps a NULL column, resolved to the deployment’s
MICROMEGAS_DEFAULT_AUDIENCE where the audience is read instead (see crate::audience’s
module doc). Admin replication (replication.rs) now hard-fails on a missing audience
column rather than ever writing a row with none. Registration
(insert_process/register_otel_process) rejects a same-process_id, different-audience
re-registration outright – needed because the OTLP process_id derivation formula is public,
so without this guard a credential could pre-register (via the native path) the exact
process_id a victim audience’s OTLP producer would later derive, silently exposing that
audience’s data to the squatter; this is a confidentiality gap, not merely an integrity one.
The stream-side mirror, check_stream_audience_conflict, closes the equivalent
re-pointed-credential gap for streams.
What remains open, tracked separately:
- The five instance-only view sets (
net_spans,otel_spans,images,async_events,thread_spans) and the per-process JITview_instancepath still resolve their audience label through the owning process’s/stream’s row rather than a genuine per-row column of their own – via the call-level guard’s instance check (AudienceGuard::authorize_view_instance,IdKind::ProcessOrStream) for the guardedview_instance(...)scan every production caller takes, not by a predicate this rule injects (see the skip in the Branch table above). The cross-audience injection scenario (an attacker’s block naming a victim’sprocess_id/stream_id) is closed for both, as a side effect of the mismatch predicate onblocks_view’s own materialization – except against a victim whoseprocesses/streamsrow is a legacy, pre-v8 NULL-audience row (next bullet). - The NULL-anchor window. A
processes/streamsrow registered before its ingestion binary supported per-row stamping keepsaudience = NULLfor its entire remaining life (rows are immutable, no backfill). The mismatch predicate’s NULL-tolerant pass-through lets a mismatched block through unchecked against such a row for as long as it exists. This is an accepted, bounded limitation over already-public legacy data, bounded by the deploy-order requirement (every ingestion replica upgraded) and shrinking as legacy rows age out under retention.
There is no in-product enforcement knob left for either surface; the mitigation is
operational – provision only audience-bound DB-backed ingestion credentials, and don’t run
ingestion with OIDC or --disable-auth alongside them – see this stage’s CHANGELOG.md
entry.
Structs§
- Ownership
Rewrite - Injects an audience predicate into every
MaterializedViewscan (see the module doc comment).