API Keys¶

Micromegas keys live in two Postgres tables — ingestion_api_keys and analytics_api_keys. The tables hold only a SHA-256 hash of each key plus a created_at/created_by/last_used_at/revoked_at/revoked_by audit trail.

analytics-web-srv is the sole HTTP surface for both tables — its /api/ingestion-api-keys* and /api/analytics-api-keys* routes let an operator mint, list, and revoke either table without a redeploy, writing directly to Postgres. Ingestion exposes no key-management HTTP surface: it only validates incoming keys against ingestion_api_keys. Both tables have an admin page in the web app (Admin → Ingestion API Keys / Analytics API Keys) that calls analytics-web-srv's routes directly (see Web app admin pages).

Minting an ingestion key is not purely an admin operation: a caller — admin or not — with a matching mint grant, or naming a brand-new audience explicitly (which lazily claims it), can mint an ingestion_api_keys row directly; a non-admin additionally needs an operator to have turned on MICROMEGAS_SELF_SERVICE_MINT (off by default), while an admin is exempt from that knob. See Self-service mint for the full mechanism. List and revoke stay admin-only and unconditional, unaffected by any of this — see HTTP routes below. The analytics-key table's routes are admin-only throughout.

The env-var keyring (MICROMEGAS_API_KEYS and its per-role forms) is no longer read by ingestion or flight-sql as of v0.31.0 — the DB-backed key store is the only source, and migrating off the keyring is required, not optional. See Migrating from the env keyring.

TLS is a prerequisite for minting

Every mint route returns the cleartext key exactly once, over whatever transport the request arrives on. Neither the ingestion service nor analytics-web-srv binds TLS itself. Put a TLS-terminating ingress in front of both services before calling any of these routes in anything but a fully trusted local network.

Why two tables¶

The security model is asymmetric: a stolen write (ingestion) key is an integrity problem; a stolen read (analytics) credential is a confidentiality one. Keeping the tables separate means a key valid on both ingestion and flight-sql must be two distinct keys — see Migrating from the env keyring. The split is enforced in code, not just schema — see Security.

Schema¶

CREATE TABLE ingestion_api_keys (
  key_id       UUID PRIMARY KEY,
  key_hash     BYTEA NOT NULL,          -- sha256 of the full key string, 32 bytes
  name         VARCHAR(255) NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL,
  created_by   VARCHAR(255) NOT NULL,   -- OIDC email/subject of the minting caller
  last_used_at TIMESTAMPTZ,
  revoked_at   TIMESTAMPTZ,
  revoked_by   VARCHAR(255),
  audience     VARCHAR(255) NOT NULL    -- immutable write audience
    CONSTRAINT ingestion_api_keys_audience_name CHECK (audience ~ '^[A-Za-z0-9_-]+$')
);
CREATE UNIQUE INDEX ingestion_api_keys_key_hash ON ingestion_api_keys(key_hash);

-- analytics_api_keys: identical shape apart from `audience` -- its read-side
-- mirror is a per-key `read_audiences` grant, not a column on this table
CREATE TABLE analytics_api_keys (
  key_id       UUID PRIMARY KEY,
  key_hash     BYTEA NOT NULL,
  name         VARCHAR(255) NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL,
  created_by   VARCHAR(255) NOT NULL,
  last_used_at TIMESTAMPTZ,
  revoked_at   TIMESTAMPTZ,
  revoked_by   VARCHAR(255)
);
CREATE UNIQUE INDEX analytics_api_keys_key_hash ON analytics_api_keys(key_hash);

key_id is a UUID handle, distinct from key_hash: DELETE {base_path}/api/ingestion-api-keys/<id> keys on it, and GET never returns key_hash. name carries no uniqueness constraint — rotating a key under a stable name means two live rows can share a name while the old one is retired. Every revoke path keys on key_id, never name.

There is no cleartext column. SHA-256 with no KDF is safe only because these are high-entropy random keys, not passwords.

HTTP routes (key management)¶

All key-management routes for both tables live on analytics-web-srv. Every route except ingestion's own mint is gated by the same admin check every other analytics-web-srv admin route uses (ValidatedUser.is_admin, resolved from membership in the reserved admins local group; see Groups). POST {base_path}/api/ingestion-api-keys (mint) runs through a MintGate/AuthenticatedUser extractor instead, so a non-admin caller with a matching grant (or a lazy claim) can reach it once MICROMEGAS_SELF_SERVICE_MINT is on. Ingestion itself exposes no key-management HTTP surface — consolidating both tables' admin surface onto one service keeps a single admin list (see Security).

Route	Body / result
`POST {base_path}/api/ingestion-api-keys`	`{"name","audience"?}` → 201 `{"key_id","name","created_at","key","audience","claimed"}`
`GET {base_path}/api/ingestion-api-keys?limit=&offset=&include_revoked=`	200 `[{"key_id","name","created_at","created_by","last_used_at","revoked_at","revoked_by","audience"}]`
`DELETE {base_path}/api/ingestion-api-keys/{key_id}`	200 `{"revoked_at"}` or 404
`POST {base_path}/api/analytics-api-keys`	`{"name"}` → 201 `{"key_id","name","created_at","key"}`
`GET {base_path}/api/analytics-api-keys?limit=&offset=&include_revoked=`	200 `[{"key_id","name","created_at","created_by","last_used_at","revoked_at","revoked_by"}]`
`DELETE {base_path}/api/analytics-api-keys/{key_id}`	200 `{"revoked_at"}` or 404

Both route groups share request/response shapes and validation for name/list/revoke; audience is an ingestion_api_keys-only field — see What audience does a key carry.

Ingestion's mint route has more error shapes than the rest of this table: 400 BAD_REQUEST, 503 NOT_CONFIGURED (no DB pool), 403 FORBIDDEN (self-service off, no matching grant, or a per-caller bound reached), 503 UNAVAILABLE (the audience-grant query itself failed), 401 UNAUTHENTICATED (no AuthContext, normally unreachable), and 409 CLAIM_CONTENDED (two concurrent lazy claims raced for the same audience name; retry). Mint remains the one route here a non-admin caller can reach at all.

Mint (POST .../{table}-api-keys) — {"name"} (plus, for ingestion, an optional "audience") → 201 {"key_id","name","created_at","key"} (plus "audience" for ingestion). key is the cleartext key, returned exactly once — never logged, never retrievable afterwards. mmk_ marks the key as a Micromegas secret for scanners; validation hashes the whole string. 400 if name is empty or exceeds 255 bytes; for ingestion, also 400 if an explicit audience is invalid — an omitted one resolves to the deployment default (see What audience does a key carry).

List (GET .../{table}-api-keys?limit=&offset=&include_revoked=) — 200, newest first. limit defaults to 100, clamps at 500, and is 400 if <= 0. offset defaults to 0. include_revoked defaults to true. Never returns key_hash or the key. Unchanged by this document's mint narrowing: still every row, every audience, AdminUser-gated and unconditional.

Revoke (DELETE .../{table}-api-keys/{key_id}) — 200 {"revoked_at"}, idempotent (a second call returns the same value). 404 for an unknown key_id. The revocation latency is bounded by whichever ingestion/flight-sql process's cache TTL is validating the key — see Cache and audit env vars. Unchanged by this document's mint narrowing: still any key, any audience, AdminUser-gated and unconditional.

Precondition: the telemetry DB must already have run the migration that creates ingestion_api_keys.audience (schema v6), which only ingestion or a lakehouse-role monolith runs. Run one of those against the target DB at least once before relying on these routes, or every call fails with an opaque 500.

Deploy ordering matters in the other direction too, since audience is NOT NULL with no default: once the schema is at v6, a not-yet-upgraded analytics-web-srv whose INSERTs omit audience starts failing with a NOT NULL violation (500). Upgrade analytics-web-srv to a version that writes audience in the same deploy that runs the v6 migration.

One env var backs both route groups:

Variable	Description
`MICROMEGAS_SQL_CONNECTION_STRING`	Telemetry-DB connection string `analytics-web-srv` opens its own small (`max_connections(2)`) pool from, backing both route groups. Required whenever auth is enabled — `analytics-web-srv` bails at startup if it's unset. Under `--disable-auth`, both route groups instead return a fixed 503 (`AUTH_DISABLED`).

What audience does a key carry¶

Every ingestion_api_keys row carries a single, immutable write audience — the value every process, stream, and block that key ingests is stamped with. analytics_api_keys has no such column: its read-side equivalent is a per-key read_audiences grant, in the opposite direction (which audiences a caller may read, not which one it writes).

An audience is an opaque label, not a principal encoding — public, team-alpha, payments-svc, alice-laptop. Who may read or mint into it is separate, editable configuration: rows in the audience_grants table (POST/GET/DELETE {base_path}/api/audience-grants, or the micromegas-grants CLI). See Audiences and Grants for the full model. A fresh deployment ships with a seeded ('public', 'read', '*') row, which makes every authenticated principal able to read public with no further grant; delete that row to change it.

The binding is immutable by design. Once a key is minted with an audience, that audience never changes for that key. Re-sharing already-ingested data with a wider audience is a grants edit (add a read selector for that audience), never a restamp.

A request that names no audience gets the deployment default (MICROMEGAS_DEFAULT_AUDIENCE, public when unset) — the same value a credential with no bound audience is stamped with at ingestion write time (see Audience stamping). Name the audience explicitly when minting a key for anything that isn't deployment-wide-public data, or set MICROMEGAS_DEFAULT_AUDIENCE to a label no principal is granted so an omission fails visibly at read time instead of publishing. An explicitly supplied but malformed audience is still a 400. Minting for the resolved audience still requires a matching mint grant (or a lazy claim).

Any caller naming a brand-new audience explicitly claims it — a genuinely fresh, never-before-granted name is minted and granted in the same request, writing user:<email> mint+read rows in the same transaction as the key insert. For a non-admin this additionally requires MICROMEGAS_SELF_SERVICE_MINT to be on; an admin is exempt from that knob but takes the identical claim path otherwise — is_admin confers no bypass of its own. micromegas-setup-telemetry exposes this via --user-audience SUFFIX: the prefix is composed server-side from the caller's own email (e.g. --user-audience ci-runner resolves to alice-ci-runner for alice@example.com), so the identical command works for an admin and a non-admin caller alike — there is no separate admin recipe. --audience NAME mints under a name verbatim instead, for an org/team/service audience that isn't namespaced under any one caller; it also lazily claims a brand-new name. See Self-service mint for the full mechanism.

MintResponse.claimed is true only when this call actually created the audience's first grant rows. Losing the claim race is never silently swallowed: losing the advisory lock itself is a 409 CLAIM_CONTENDED (retry), and losing the in-lock existence recheck — another caller's grant or key row landing first — is a 403. A caller with no email is unaffected by any of this — no user: row can be formed, so nothing is ever claimed for them; minting still requires a pre-existing grant. Minting into an existing audience the caller holds no grant on is a plain 403 for every caller, admin included — an admin can no longer mint silently into an audience they were never granted.

A hand-edited row takes effect within the key's cache TTL, not instantly (MICROMEGAS_AUTH_CACHE_TTL_SECONDS, default 60s; see Cache and audit env vars) — since the audience is immutable, that caching is free.

Minting an analytics key over HTTP¶

curl -X POST https://analytics.example.com/api/analytics-api-keys \
  -H 'Content-Type: application/json' -H "Cookie: id_token=$TOKEN" \
  -d '{"name": "grafana-datasource"}'

Or use the Admin → Analytics API Keys page in the web app, which shows the minted key exactly once in a dismissable banner with a copy-to-clipboard button — the browser never receives it a second time, and it's never persisted client-side.

A freshly minted analytics key is consumed from Python via StaticTokenAuthProvider — see Static Analytics API Keys and the api_key_file profile setting in the Python API Reference. The Grafana plugin consumes the same key the same way — see Grafana Authentication.

Minting an ingestion key uses the same shape against /api/ingestion-api-keys (or the Admin → Ingestion API Keys page), but the body must supply an audience:

curl -X POST https://analytics.example.com/api/ingestion-api-keys \
  -H 'Content-Type: application/json' -H "Cookie: id_token=$TOKEN" \
  -d '{"name": "grafana-datasource", "audience": "team-alpha"}'

Omitting audience mints for MICROMEGAS_DEFAULT_AUDIENCE (public when unset).

Revoke — DELETE {base_path}/api/{ingestion,analytics}-api-keys/{key_id}, keyed only on key_id. GET {base_path}/api/{ingestion,analytics}-api-keys is the way to discover a key_id to revoke.

Web app admin pages¶

Two admin pages, both reachable from Admin (/admin) in the sidebar:

Analytics API Keys (/admin/analytics-keys) — calls analytics-web-srv's /api/analytics-api-keys* routes directly. Fully admin-gated (AuthGuard requireAdmin) — analytics keys have no self-service story.
Ingestion API Keys (/admin/ingestion-keys) — calls analytics-web-srv's /api/ingestion-api-keys* routes directly. No proxy, no forwarding to ingestion, no service credential.

/admin and /admin/ingestion-keys are viewable by every authenticated user, with role-filtered content. AuthGuard on both routes carries no requireAdmin; each branches on the caller's role:

On /admin/ingestion-keys, an admin sees the full list/mint/revoke table (ApiKeysAdminPage); a non-admin sees a mint-only panel — the same self-service mint dialog Audience Access uses, no table, no revoke UI. list_keys/revoke_key stay admin-only server-side. Mint only appears once MICROMEGAS_SELF_SERVICE_MINT is on and the caller holds a matching mint grant (or names a fresh audience); with the knob off, the panel explains why and points to Audience Access.
On /admin, see Admin hub for the role-filtered card grid.

Every other admin page under /admin — Data Sources, Export Screens, Import Screens, Maps, Analytics API Keys, Query Deny List — stays fully gated by AuthGuard requireAdmin.

Both ingestion/analytics pages stay visible even when the backing pool isn't configured; the page surfaces the 503 the route returns, same as MapsPage does for an unconfigured maps store. There is no way to bring an externally-chosen key string into either table: every key is generated server-side by its mint route and returned exactly once, so a key never transits a browser form on its way in.

A third page, open to every authenticated user, not just admins: Audience Access (/audiences) is the self-service counterpart of the ingestion-key mint flow — it drives the mint route's shared claim-and-mint path (every caller, admin included) from a browser dialog, plus the audience-grant read/write routes covered in Authorization → the grant store. See web-app.md for the full page reference.

created_by/revoked_by always reflect the acting admin's own OIDC identity, for both key tables. Every mint/revoke handler resolves the caller's identity (user.email or user.subject) and writes that directly — there is no service-credential hop, so no attribution gap to document.

Single admin group, for administration. List/revoke for both tables gate on the same analytics-web-srv admin check (membership in the reserved admins local group — see Groups). Ingestion's own mint route is authorized by a mint grant for every caller, admin included; admins membership only waives the MICROMEGAS_SELF_SERVICE_MINT knob that otherwise blocks a non-admin caller from minting at all.

Under --disable-auth on analytics-web-srv, all three key/grant route groups are unavailable — not just gated. With auth disabled, every request would otherwise be treated as an admin, which would let an unauthenticated caller mint/revoke real keys or grants; instead all three path prefixes (/api/ingestion-api-keys, /api/analytics-api-keys, /api/audience-grants) return a fixed 503 ({"code": "AUTH_DISABLED", ...}), including any sub-path.

Cache and audit env vars¶

Variable	Default	Description
`MICROMEGAS_API_KEY_CACHE_SIZE`	`10000`	Max distinct live keys cached per process
`MICROMEGAS_AUTH_CACHE_TTL_SECONDS`	`60`	Shared, flat positive-cache TTL for the API-key, audience-grant, and group stores — also the API-key revocation-latency bound (see Groups for the group-store latency this same knob governs)
`MICROMEGAS_API_KEY_UNKNOWN_CACHE_TTL_SECONDS`	`10`	Negative-cache TTL (shorter, so a freshly minted key isn't masked by an earlier probe)
`MICROMEGAS_API_KEY_UNKNOWN_CACHE_SIZE`	`10000`	Max distinct unknown tokens cached per process

MICROMEGAS_API_KEY_CACHE_SIZE/_UNKNOWN_CACHE_TTL_SECONDS/_UNKNOWN_CACHE_SIZE each accept a role prefix on the monolith — MICROMEGAS_INGESTION_API_KEY_CACHE_SIZE / MICROMEGAS_ANALYTICS_API_KEY_CACHE_SIZE, and so on — falling back to the unprefixed name, the same convention MICROMEGAS_OIDC_CONFIG uses. MICROMEGAS_AUTH_CACHE_TTL_SECONDS is the one exception: it is a single flat, unprefixed knob with no role-scoped variant, since it governs the API-key, audience-grant, and group stores together as one process-wide value.

Revocation takes effect within cache_ttl_secs (default 60s), not instantly — raising the TTL trades revocation latency for DB load.

Monitoring a key-store outage¶

A Postgres outage on the key-lookup path is surfaced as 503 (or Status::unavailable on gRPC/FlightSQL) — never 401 — so clients retry rather than treat it as a rejected credential. Alert on the db_api_key_error_count metric (tagged {table}), emitted on every DB error the key-lookup path hits, independent of the accompanying error! log line, which is rate-limited to at most once per cache_ttl_secs (floored at 60s) window per table.

Grant recipe (separated DB roles)¶

By default every service shares one DB role via MICROMEGAS_SQL_CONNECTION_STRING. The table split is still enforced in code: analytics-web-srv's mint/list/revoke routes each hardcode which table they target, and the ingestion/analytics key-validation providers are each constructed bound to their own table. Operators who run separate DB roles per service can additionally enforce the split at the grant level:

-- ingestion role: read + touch only (key *validation*, not administration --
-- analytics-web-srv is the only role that mints/revokes)
GRANT SELECT ON ingestion_api_keys TO micromegas_ingestion;
GRANT UPDATE (last_used_at) ON ingestion_api_keys TO micromegas_ingestion;
-- and no grant of any kind on analytics_api_keys

-- analytics role: read + touch only (flight-sql's own key-validation identity)
GRANT SELECT ON analytics_api_keys TO micromegas_analytics;
GRANT UPDATE (last_used_at) ON analytics_api_keys TO micromegas_analytics;
-- and no grant of any kind on ingestion_api_keys

-- analytics-web-srv's own role: write + touch only, on BOTH key tables --
-- it is the sole admin surface for both
GRANT SELECT, INSERT ON ingestion_api_keys TO micromegas_web;
GRANT UPDATE (revoked_at, revoked_by) ON ingestion_api_keys TO micromegas_web;
GRANT SELECT, INSERT ON analytics_api_keys TO micromegas_web;
GRANT UPDATE (revoked_at, revoked_by) ON analytics_api_keys TO micromegas_web;

-- analytics role: read-only on the audience grant store -- DbAudienceGrantsSource
-- re-queries this table on every snapshot refresh
GRANT SELECT ON audience_grants TO micromegas_analytics;

-- analytics-web-srv's own role: the sole admin surface for audience_grants too
GRANT SELECT, INSERT, DELETE ON audience_grants TO micromegas_web;

-- analytics role: read-only on the group store -- DbGroupsSource re-queries
-- groups/group_members on every snapshot refresh
GRANT SELECT ON groups TO micromegas_analytics;
GRANT SELECT ON group_members TO micromegas_analytics;

-- analytics-web-srv's own role: the sole admin surface for groups too
GRANT SELECT, INSERT, DELETE ON groups TO micromegas_web;
GRANT SELECT, INSERT, DELETE ON group_members TO micromegas_web;

The last grant is DELETE, not UPDATE: audience_grants/group_members rows are hard-deleted on revocation (groups rows too, on delete). micromegas_web (analytics-web-srv's role) is the only role granted INSERT on any of these tables in a fully separated-role deployment; micromegas_ingestion is read + last_used_at touch only on its own table; micromegas_analytics is read + last_used_at touch on analytics_api_keys, plus read-only on audience_grants/groups/group_members. Neither service role has any grant on the other service's table. analytics-web-srv's role does gain write access to ingestion_api_keys under this design, since every ingestion-key mint/revoke goes through it.

Migrating from the env keyring¶

This is a v0.31.0 upgrade requirement, not an option. As of v0.31.0, ingestion and flight-sql no longer read MICROMEGAS_API_KEYS (or its per-role/prefixed forms) at all — only ingestion_api_keys / analytics_api_keys and OIDC authenticate a caller. A still-set variable logs a warn! naming it, but does not stop startup or restore the old behavior.

The consequence of skipping this migration depends on what else is configured: with OIDC also configured, the service starts normally and every keyring token silently stops authenticating — the warn! line is the only signal. Without OIDC and with an empty ingestion_api_keys / analytics_api_keys table, the service refuses to start at all (the pre-existing "no auth providers configured" bail), since an env-only keyring no longer counts as configured auth.

A legacy key's own string cannot be carried forward. Every key in the DB store is generated server-side by its mint route; there is no route, CLI, or admin page that accepts an externally-chosen key string. Migrating therefore means minting replacement keys and redistributing them to the clients still presenting the old ones.

Deploy the new binaries. The migration creates the tables (schema v5). In a split deployment, start ingestion (or the monolith) before flight-sql — flight-sql never runs the migration. Violating this ordering surfaces as a warn! log line naming the table (flight-sql still starts if OIDC is configured; otherwise it fails to start, per the consequence above). For these key-management routes specifically, analytics-web-srv never runs this migration itself — the target telemetry DB must already have had ingestion or a lakehouse-role monolith run against it at least once.
Mint a replacement for every key still in the keyring, one per keyring entry, from the web app's Admin → Ingestion API Keys / Analytics API Keys pages (or the mint routes directly). Each mint returns its cleartext key exactly once — capture it then, and configure the client that used the corresponding legacy key with the new string. micromegas-setup-telemetry covers the common case of an individual minting their own ingestion key and printing the matching OTLP exporter env vars.

Three cases need routing decisions: - An object-cache-srv client key stays env-only forever — no replacement needed, see Object Cache. - A key that is also a service's own ingestion self-telemetry credential (MICROMEGAS_INGESTION_API_KEY) gets its replacement from ingestion_api_keys. - A key valid on both ingestion and flight-sql today must become two distinct keys, one per table — mint one from each page and configure each client with whichever half it actually needs. 3. Unset MICROMEGAS_API_KEYS (and prefixed variants) on ingestion and flight-sql once the tables are populated; leaving it set logs a warning and changes nothing, since neither service reads it anymore. A non-empty key store counts as "auth configured" on its own, so both services keep serving without OIDC — including a key-only flight-sql deployment (see Grafana Authentication). object-cache-srv keeps MICROMEGAS_API_KEYS permanently — see Object Cache.

Security¶

API keys can never manage keys: is_admin is hardcoded false on every API-key auth context, and analytics-web-srv's AdminUser extractor rejects any caller whose is_admin isn't true before a list/revoke handler runs. There is no bearer-key authenticator on analytics-web-srv's /api/* routes at all, so an ingestion API key has no code path to reach the mint route's extractors, admin or not. Self-service mint only ever widens which browser-authenticated OIDC caller may mint — never what kind of credential can.
No cleartext at rest. Only a SHA-256 hash is stored; the cleartext key exists only in the one-time POST response and in the client's own storage.
No timing side channel on the DB path — the token is hashed and the hash used as an index key, so lookup time doesn't depend on how many leading bytes of a guess are correct.
Revocation latency is bounded, not zero — see Cache and audit env vars.
A key-store outage is never client-visible as a rejected credential — see Monitoring a key-store outage.