Maintenance Daemon¶

telemetry-maintenance-srv is the maintenance daemon that keeps the data lake healthy. It is a long-running service you deploy alongside ingestion and FlightSQL: it materializes views on a schedule and runs retention cleanup.

Ad-hoc administration — inspecting partitions, retiring incompatible ones — is done through SQL and the Python API, not by driving this binary. See Admin SQL Functions.

It reads the lake from the environment:

Variable	Required	Description
`MICROMEGAS_SQL_CONNECTION_STRING`	Yes	PostgreSQL connection for lake metadata
`MICROMEGAS_OBJECT_STORE_URI`	Yes	Object store holding the partitions
`MICROMEGAS_DEFAULT_AUDIENCE`	No	The audience a credential with no bound ingestion audience is stamped with at write time (default `public`). Set it identically on every role that builds a lakehouse — FlightSQL, this one, the monolith, and ingestion — since this role bakes the value into partitions and the ingestion role stamps new rows with it; setting it on only some of them materializes and stamps under mismatched defaults. See Audience stamping
`MICROMEGAS_DATAFUSION_MEMORY_BUDGET_MB`	No	Query engine memory budget in MB; unset means an unbounded pool (the local-development default). This is set in real deployments. Unlike `flight-sql-srv`, the daemon's merges and materialization run on the shared, unscoped pool — there is no per-query audit record here, so this budget is a process-wide ceiling rather than something attributable to one query. Merge scans open one reader per source file group -- one reader total for the concatenating path, or one per input partition for the ordered sort-merge path -- so merge memory does not scale with host core count
`MICROMEGAS_DATAFUSION_MAX_TEMP_DIRECTORY_MB`	No	Cap on total spill-file bytes across all concurrent work, in MB; default 100 GB (DataFusion's own default), far larger than a typical Fargate container's local disk. Applies the same way as on `flight-sql-srv`: exceeding the cap fails whichever task's spill write pushes past it, process-wide and shared across all concurrent work, not necessarily the task that consumed most of the budget
`MICROMEGAS_STATIC_TABLES_URL`	No	Auto-discovery source for static JSON/CSV tables (see FlightSQL). Set it identically on every role that builds a lakehouse — this daemon resolves it too, so a materialized view whose extract query reads a static table builds the same way here as it does on `flight-sql-srv`
`MICROMEGAS_VIEW_DEFINITION_REFRESH_SECONDS`	No	How often the daemon reloads DDL-defined materialized views from Postgres (default `60`). The daemon now picks up a view set created, replaced, or dropped by `CREATE`/`DROP MATERIALIZED VIEW` without a restart, bounded by this interval

Lakehouse schema migrations run automatically at service startup. The v7 migration runs CREATE EXTENSION IF NOT EXISTS btree_gist (required by the partition-overlap exclusion constraint). On PostgreSQL 13+ — including Amazon Aurora and RDS — btree_gist is a trusted extension, so the service's database role only needs the CREATE privilege on the database it already owns. On PostgreSQL 12 or older, or if the role lacks that privilege, have a superuser pre-create the extension once; the migration then proceeds unchanged.

Running the daemon¶

# from the rust/ directory
cargo run --release --bin telemetry-maintenance-srv

The Docker image (maintenance.Dockerfile) runs telemetry-maintenance-srv as its entrypoint; no arguments are required.

Flag	Default	Description
`--shutdown-grace-period-seconds`	`25`	Seconds to let in-flight tasks finish on `SIGTERM`
`--retention-days`	`90`	Delete lake data older than this many days (retention horizon)

On SIGTERM the daemon stops scheduling new tasks and drains those already running, up to the grace period. See Service Lifecycle & Shutdown. Interrupted materialization is safe to redo — a task that doesn't finish simply leaves its partition unwritten, and the next scheduled run redoes it.

A view that fails to materialize (e.g. a bad SqlBatchView query) does not starve the views ordered after it: each view's materialization is attempted and its failure isolated independently, so every other view still gets materialized in the same pass. A failing view is logged and counted (see materialize_view_failure below); if any view failed, the pass as a whole is still reported as failed so the failure surfaces in the daemon's logs, but that failure is attributed per view rather than to whichever view happened to fail first.

`materialize_view_failure`¶

Each of the four materialization tasks (every second/minute/hour/day) emits materialize_view_failure (count, tags {view_set_name, view_instance_id}) as one event with value = 1 per failed view per materialization pass. To count failures over a window, use count(*) or sum(value).

Run a single telemetry-maintenance-srv instance per lake. The scheduled tasks are not partitioned across instances, so multiple daemons would redundantly materialize the same partitions. Materialization is idempotent, so this is wasteful rather than corrupting — but there is no benefit to more than one.

What it does¶

The daemon keeps materialized views current by running several scheduled tasks. The four materialization tasks each work a trailing window at their own granularity, so recent data lands in fine-grained partitions quickly while older data is consolidated into coarser ones:

Task	Period	Work
Every second	1 s	Materialize the newest 1-second partitions. Skipped when the daemon is more than 10 s behind — the minute task backfills the gap.
Every minute	1 min	Materialize 1-minute partitions.
pg_stats	1 min	Sample the metadata Postgres's `pg_stat_*` views (see below).
Every hour	1 h	Retention cleanup (see below), the `blocks`/`streams`/`audience` integrity counts (see below), then materialize 1-hour partitions.
Every day	1 day	Materialize 1-day partitions.

Retention¶

The hourly task performs cleanup automatically:

Deletes lake data older than the retention horizon — blocks, streams, and processes past the horizon are removed.
Deletes expired temporary files left behind by query execution.

Integrity counts¶

Alongside retention cleanup, the hourly task also counts two blocks-row anomalies over the last hour, directly against Postgres. Both cost nothing on the ingestion hot path — that is precisely why they run here instead of at write time.

`block_stream_process_id_mismatch`¶

count, no tags: the number of blocks rows in the last hour whose process_id disagrees with their stream's own process_id. No longer security-critical under per-row audience stamping — a block's own audience column governs its label regardless of what process_id it claims — so this is a plain data-integrity signal. Healthy baseline: a flat zero. Every non-zero reading is a bug or an attack; a hard-reject at write time is a deferred follow-up pending data from this counter.

`block_audience_mismatch_rows`¶

count, no tags: the number of blocks rows in the last hour whose own audience disagrees with their stream's or process's audience — built from the same NULL-tolerant comparison the blocks_view.rs materialization-time exclusion predicate uses, so the two can never drift apart. A non-zero reading is not necessarily a bug — it may reflect a re-pointed ingestion credential — but it always means telemetry was silently dropped from blocks (and so from log_entries/measures/log_stats and every other view built from it) by that exclusion. A deployment should watch this metric read a flat zero for a representative period before trusting that the exclusion is only ever discarding attacker-injected blocks and not legitimate telemetry; a nonzero, non-attack reading is a sign to fix the underlying cause (e.g. stop re-pointing a credential's audience mid-stream) rather than to treat the drop as expected.

Kept separate from, and never summed with, block_audience_mismatch_excluded below — both run in this same maintenance-role process, so an identically-named counter from both sites would land in the same process's measures stream with no tag to tell them apart, and any query summing them would be summing two incompatible quantities: this one is a live Postgres row count over the last hour, the other is a per-partition exclusion count at materialization time.

`block_audience_mismatch_excluded`¶

count, no tags: emitted from MetadataPartitionSpec::write, not from the hourly task — only when a blocks partition is actually written (never on a scheduled pass that decides nothing needs writing), naming the count of rows the audience-mismatch predicate excluded from that partition. It still double-counts across a materialization retry, or a fresh CreateFromSource re-write of a range whose source rows changed, since each such write re-runs the comparison — but not across a re-merge, which writes through the view's PartitionMerger and never calls MetadataPartitionSpec::write. So it is a "some write saw a mismatch" signal rather than a running total of distinct excluded blocks — trust block_audience_mismatch_rows above for sizing the drop.

The retention horizon defaults to 90 days and is configurable via --retention-days or the MICROMEGAS_RETENTION_DAYS environment variable:

telemetry-maintenance-srv --retention-days 30
# or
export MICROMEGAS_RETENTION_DAYS=30

The flag takes precedence over the environment variable, which in turn takes precedence over the default.

Metadata Postgres self-observability¶

Once a minute, the daemon samples the metadata Postgres's standard pg_stat_* catalog views (plus index/table sizes) and emits the readings as micromegas metrics through its own tracing sink, so they land in the lake's measures view like any other telemetry — no extra wiring or credentials required. This turns questions that could previously only be answered by connecting to the DB directly (e.g. which indexes are dead weight) into evidence queryable via FlightSQL.

All counters are emitted raw and cumulative, exactly as Postgres reports them — the collector never calls pg_stat_reset* and holds no state between ticks. Deltas and rates are a query-time concern.

Metric family	Tags	Source
`pg_index_scans`, `pg_index_tuples_read`, `pg_index_tuples_fetched`, `pg_index_size_bytes`	`{table, index}`	`pg_stat_user_indexes` + `pg_relation_size`
`pg_table_seq_scans`, `pg_table_idx_scans`, `pg_table_live_tuples`, `pg_table_dead_tuples`, `pg_table_tuples_inserted`, `pg_table_tuples_updated`, `pg_table_tuples_deleted`, `pg_table_seconds_since_autovacuum`	`{table}`	`pg_stat_user_tables`
`pg_db_blocks_hit`, `pg_db_blocks_read`, `pg_db_xact_commit`, `pg_db_xact_rollback`, `pg_db_deadlocks`, `pg_db_temp_bytes`, `pg_db_stats_reset_timestamp`	—	`pg_stat_database`
`pg_activity_connections`	`{state}`	`pg_stat_activity`, grouped
`pg_activity_oldest_xact_age_seconds`	—	`pg_stat_activity`
`pg_pool_size`, `pg_pool_idle`	—	the daemon's own `sqlx::PgPool` (client-side, no query)

Tags are read with the property_get SQL function, e.g. property_get(properties, 'index').

pg_db_stats_reset_timestamp marks counter-reset boundaries (a clean restart on PG15+, a crash, or an Aurora failover/patch/instance replacement) — segment on it rather than assuming counters only ever increase.

Sample queries¶

Indexes with zero scans over the observed window (candidates for removal):

SELECT property_get(properties, 'table')  AS table,
       property_get(properties, 'index')  AS index,
       max(value)                          AS idx_scans
FROM measures
WHERE name = 'pg_index_scans'
GROUP BY 1, 2
HAVING max(value) = 0
ORDER BY 1, 2;

Cache-hit ratio over a window (as a delta, since the counters are cumulative):

WITH bounds AS (
    SELECT min(value) FILTER (WHERE name = 'pg_db_blocks_hit')  AS hit_start,
           max(value) FILTER (WHERE name = 'pg_db_blocks_hit')  AS hit_end,
           min(value) FILTER (WHERE name = 'pg_db_blocks_read') AS read_start,
           max(value) FILTER (WHERE name = 'pg_db_blocks_read') AS read_end
    FROM measures
    WHERE name IN ('pg_db_blocks_hit', 'pg_db_blocks_read')
)
SELECT (hit_end - hit_start)::float
       / nullif((hit_end - hit_start) + (read_end - read_start), 0) AS cache_hit_ratio
FROM bounds;

Out of scope (follow-ups)¶

pg_stat_statements — needs shared_preload_libraries and CREATE EXTENSION on the Aurora cluster parameter group.
Aurora/CloudWatch-only signals (ACUUtilization, Performance Insights db.load.avg) — need the AWS SDK and IAM credentials, unlike the in-DB views above which require none.

Ad-hoc administration¶

Manual maintenance — backfilling a time range, retiring stale or schema-incompatible partitions — runs through the FlightSQL server, not this binary:

SQL functions such as materialize_partitions() (backfill a time range), regenerate_partitions() (force-rebuild existing partitions directly from source data, bypassing the freshness check materialize_partitions() stops at), retire_partitions(), and retire_partition_by_metadata(). All four require an authenticated admin — see Authentication.
Python helpers such as micromegas.admin.list_incompatible_partitions() and micromegas.admin.retire_incompatible_partitions() (the latter internally calls retire_partition_by_metadata() and so also requires admin).

Both are documented in Admin SQL Functions.