Module lakehouse 

Module dedicated to the maintenance and query of materialized views

Unlike the telemetry data lake where it’s fast & cheap to write but costly to read, the lakehouse partitions are costly to write but allow for cheap & fast queries using datafusion.

Views based on a low frequency of events (< 1k events per second per process) are kept updated regularly. Views based on a high frequency of events (up to 100k events per second per process) are metrialized on demand.

Modules

answer

Record batches + schema

async_events_block_processor

Implementation of BlockProcessor for async events

async_events_view

Materializable view of async span events accessible through datafusion

async_parquet_writer

Write parquet in object store

audience_guard

The call-level guard: per-id audience resolution (AudienceIndex) and the fail-closed guard (AudienceGuard) for the span/metadata UDTFs and get_payload UDF OwnershipRewrite (the row-level filter) structurally cannot reach, plus a scan-time guard on view_instance(...). For the six view sets carrying a physical audience column, the row-level filter can and does reach a view_instance scan too, row-filtering it the same as the named-table form – there view_instance joined the call-level guard to close a cost/availability residual (stopping JIT materialization for a foreign-audience instance), not because the row-level filter can’t filter it. For the other five view sets, reachable only through a guarded view_instance(...), the row-level filter injects no predicate at all, so this guard is their sole enforcement. The call-level guard – arg-addressed guards for five entry points: the span/metadata UDTFs and the get_payload UDF that crate::lakehouse::ownership_rewrite::OwnershipRewrite (the row-level filter) structurally cannot reach (they bake their target id into a provider at plan time, return schemas with no process_id column to filter on, and some build their own inner session under ReadScope::All), plus view_instance(...). For the six view sets carrying a physical audience column, the row-level filter can and does reach a view_instance scan too – it row-filters it the same as the named-table form – so this guard’s job there is only to stop a caller from triggering JIT materialization (real compute and object-storage writes) for an instance outside their audiences before the row-level filter ever gets to run, a cost/availability concern rather than a confidentiality one. For the other five view sets (net_spans, otel_spans, images, async_events, thread_spans – reachable only through a guarded view_instance(...), never as a named table), the row-level filter injects no per-row predicate at all once this guard is in place – see crate::lakehouse::materialized_view::MaterializedView::instance_is_audience_guarded and OwnershipRewrite’s module doc for the other side of that coupling – so this guard is their sole confidentiality enforcement, not a redundant belt-and-braces check.

batch_update

Materialize views on a schedule based on the time data was received from the ingestion service

block_object_decoder

Format → decoder registry for parse_block, mirroring BlockProcessorMap Format → decoder registry for parse_block: turns a block payload of any registered streams.format into generic (type_name, jsonb) objects, one per row, mirroring the existing BlockProcessorMap pattern (block_partition_spec.rs).

block_partition_spec

Specification for a view partition backed by a set of telemetry blocks which can be processed out of order

blocks_view

Replicated view of the blocks table of the postgresql metadata database.

catalog

Catalog utilities for discovering and managing view schemas Catalog utilities for discovering and managing view schemas.

dataframe_time_bounds

deny_queries_table_function

Admin UDTF that validates and inserts a query-deny-list rule deny_queries(match_expr, reason) – admin UDTF that validates and inserts a new query-deny- list rule. call_with_args runs every validation synchronously and fail-loud with plan_err!: expression compilation (compile_match_expr, against the bare SessionContext QueryDenyList holds for exactly this, not the caller’s own session – rule compilation must produce the same result on every replica regardless of whose session triggered it), the empty-reason check, the caller-identity check (CallerContext::identity must be Some), and the rule-count check against the current snapshot. Only the DB write and the local snapshot refresh happen in the async body, behind LogStreamTableProvider/TaskLogExecPlan – QueryDenyList::insert takes the already-compiled expression and cannot itself fail on a bad expression.

export_log_view

Export mechanism that doubles as audit trail

get_payload_function

Fetch payload from the object store using SQL

image_block_processor

Implementation of BlockProcessor for images

images_view

Materializable view of images accessible through datafusion

jit_partitions

Management of process-specific partitions built on demand Just-in-time (JIT) partition generation groups a view’s source blocks (queried from blocks_view in ORDER BY insert_time, block_id order) into SourceDataBlocksInMemory partitions no larger than JitPartitionConfig::max_nb_objects.

lakehouse_context

Bundles runtime resources for lakehouse query execution

list_audience_grants_table_function

Caller-scoped UDTF listing rows of the audience_grants table list_audience_grants() – caller-scoped UDTF listing rows of the audience_grants table. Modeled on list_query_denials_table_function.rs’s shape (a TableFunctionImpl whose sync call_with_args returns a lazy TableProvider, an async scan that runs the DB query and builds arrays), but registered outside the admin gate in register_lakehouse_functions, next to list_view_sets() – every authenticated caller (and, per is_admin(metadata)’s documented --disable-auth convention, every caller when auth is disabled) can call it, scoped by crate::lakehouse::list_audience_grants_table_function::GrantVisibility.

list_partitions_table_function

Read access to the list of lakehouse partitions

list_query_denials_table_function

Admin UDTF listing every query-deny-list rule currently in force list_query_denials() – admin UDTF listing every query-deny-list rule currently in force. Registered inside the same admin-gated block as list_partitions()’s mutating siblings.

list_view_set_definitions_table_function

Admin UDTF listing every row of lakehouse_view_set_definitions, straight from Postgres list_view_set_definitions() – admin UDTF listing every row of lakehouse_view_set_definitions, straight from Postgres rather than from ViewRegistry::current() – so an admin can see a definition that exists on disk but failed to load (present here, absent from list_view_sets()). Registered inside the same admin-gated block as list_query_denials()’s siblings.

list_view_sets_table_function

Read access to view sets with their schema information

log_block_processor

Implementation of BlockProcessor for log entries

log_stats_view

SQL-based view for log statistics aggregated by process, minute, level, and target

log_view

Materializable view of log entries accessible through datafusion

materialize_partitions_table_function

Exposes materialize_partitions as a table function

materialized_view

TableProvider implementation for the lakehouse

merge

Merge consecutive parquet partitions into a single file

metadata_cache

Global LRU cache for partition metadata

metadata_partition_spec

Specification for a view partition backed by a table in the postgresql metadata database.

metrics_block_processor

Implementation of BlockProcessor for measures

metrics_view

Materializable view of measures accessible through datafusion

migration

Maintenance of the postgresql tables and indices use to track the parquet files used to implement the views

net_spans_view

Jit view of pre-paired network bandwidth spans (Connection / Object / Property / RPC)

otel

OTLP reader path: attribute helpers, block processors, and the otel_spans view.

ownership_rewrite

The row-level filter: injects an audience predicate into every MaterializedView scan based on the caller’s ReadScope. OwnershipRewrite – the row-level filter.

parse_block_table_function

Table function to parse all transit objects in a block and return them as JSONB

partition

Write & delete sections of views

partition_cache

In-memory copy of a subnet of the list of the partitions in the db

partition_metadata

Reads and parses partition Parquet footers directly from object storage

partition_source_data

Describes the event blocks backing a partition

partitioned_execution_plan

ExecutionPlan based on a set of parquet files

partitioned_table_provider

TableProvider based on a set of parquet files

perfetto_trace_execution_plan

ExecutionPlan for generating Perfetto trace chunks

perfetto_trace_table_function

Table function for generating Perfetto trace chunks

process_spans_table_function

Table function returning thread and/or async spans from all CPU streams of a process

process_streams

Shared utilities for discovering CPU streams of a process

processes_view

Replicated view of the processes table of the postgresql metadata database.

query

property_get function support from SQL Datafusion integration

query_deny_list

Admin-managed query deny list: fingerprinting, the match context, and the Postgres-backed, TTL-refreshed cache execute_query checks every query against. Admin-managed query deny list: a small set of admin-authored rules, stored in Postgres, cached in every flight-sql-srv replica, and evaluated at the front of execute_query – before the session context is built and before planning – so a matching query is rejected for a few microseconds of work instead of a memory-pool reservation and a wave of object-store reads.

read_scope

Authorization seam input: ReadScope and the CallerContext bundle threaded into make_session_context. Consumed by the row-level filter (ownership_rewrite::OwnershipRewrite) and the call-level guard (audience_guard::AudienceGuard). The analytics-side half of the authorization seam.

reader_factory

Wrapper around ParquetObjectreader to provide ParquetMetaData, reading footers from the ObjectStore through a cache-backed reader

regenerate_partitions_table_function

Exposes regenerate_partitions as a table function that force-regenerates partitions directly from source data

remove_query_denial_udf

Admin scalar UDF that removes a single query-deny-list rule by id remove_query_denial(rule_id) – admin scalar UDF that deletes a query-deny-list rule. The audit log is the durable record of what was denied and what it rejected, so the row itself does not need to survive its removal (hard delete, no removed_at/removed_by trail).

retire_partition_by_file_udf

Scalar UDF to retire a single partition by file path

retire_partition_by_metadata_udf

Scalar UDF to retire a single partition by metadata

retire_partitions_table_function

Exposes retire_partitions as a table function

runtime

Runtime resources

scoped_memory_pool

Per-query MemoryPool wrapper that adds accounting on top of a shared pool Per-query [datafusion::execution::memory_pool::MemoryPool] wrapper that adds accounting on top of a shared pool.

session_configurator

SessionConfigurator trait for custom session context configuration

sql_batch_view

Sql-defined view updated in batch

sql_partition_spec

Specification for a view partition backed by a SQL query on the lakehouse.

static_tables_configurator

Auto-discovery configurator for static JSON/CSV tables

streams_view

Replicated view of the streams table of the postgresql metadata database.

table_scan_rewrite

Rewrite table scans to take the query range into account

temp

Tracking of expired partitions

thread_spans_view

Jit view of the call tree built from the thread events of a single stream

view

Basic interface for a set of rows queryable and materializable

view_definition

The normalized shape of a DDL-defined (or seeded) materialized view, and the validation shared by the DDL executor and the registry loader. The normalized shape of a DDL-defined (or seeded) materialized view – the three queries plus every option – and the one validate_view_definition function shared by the DDL executor (rust/public/src/servers/view_ddl.rs) and the registry loader (view_registry.rs), so a definition the loader would skip can never be accepted at CREATE time in the first place.

view_definition_store

Postgres-backed persistence for DDL-defined view definitions (lakehouse_view_set_definitions). Postgres-backed persistence for lakehouse_view_set_definitions.

view_factory

default_view_factory makes the default ViewFactory, giving users access to view instances, grouped in sets.

view_instance_table_function

Table function to query process-specific views

view_registry

Shared Arc<ViewFactory> rebuilt from Postgres-backed view definitions on a timer, giving telemetry-maintenance-srv and flight-sql-srv a common, reloadable view of DDL-defined view sets without a restart. ViewRegistry: a shared Arc<ViewFactory> (the code-driven base plus every DDL-defined global view) rebuilt from lakehouse_view_set_definitions on a timer, so telemetry-maintenance-srv and flight-sql-srv both start reading/materializing a new or changed view set without a restart. Shaped after QueryDenyList: a Postgres-backed store, an immutable compiled snapshot behind a lock, a refresh/reload that logs and meters its own failures, and spawn_refresh_task.

write_partition

Add or remove view partitions