OTLP Ingestion¶

Micromegas accepts native OpenTelemetry Protocol (OTLP) traffic over HTTP alongside its custom transit/CBOR wire format. Any OTel-instrumented program — Claude Code, Goose, generic OTel SDKs (Python, Go, JS, .NET, Java) — can point OTEL_EXPORTER_OTLP_ENDPOINT at the ingestion service and have logs, metrics, and spans land in the lakehouse.

Overview¶

The ingestion service exposes the following HTTP ingestion routes. The first three mirror the OpenTelemetry specification directly; the rest accept non-OTLP payloads (Kinesis Firehose deliveries) and translate them internally:

Route	Payload	Lands in
`POST /ingestion/otlp/v1/logs`	`ExportLogsServiceRequest`	`log_entries`
`POST /ingestion/otlp/v1/metrics`	`ExportMetricsServiceRequest`	`measures`
`POST /ingestion/otlp/v1/traces`	`ExportTraceServiceRequest`	`otel_spans` (per-process JIT view)
`POST /ingestion/otlp/v1/metrics/firehose`	`ExportMetricsServiceRequest` per Firehose record	`measures` (see CloudWatch Metric Streams)
`POST /ingestion/cloudwatch/v1/logs/firehose`	CloudWatch Logs subscription-filter record per Firehose record (not OTLP-framed — see CloudWatch Logs)	`log_entries`

Routes share the existing listener (default 127.0.0.1:9000) and authentication chain. OTLP payloads are stored as-is in object storage; decoding into parquet rows happens lazily at the analytics layer.

Wire format: OTLP/HTTP with Content-Type: application/x-protobuf or Content-Type: application/json. Optional Content-Encoding: gzip is supported. gRPC OTLP is not supported in the current release.

Quick Start¶

Point an OTel SDK at the ingestion service:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:9000/ingestion/otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

The SDK appends /v1/{logs,metrics,traces} to the base URL per the OTLP spec, so a request lands on http://127.0.0.1:9000/ingestion/otlp/v1/logs. If your operator has set per-signal endpoints (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT), those are full URLs and need to include the /v1/<signal> suffix themselves.

For a production deployment with auth, see Authentication below.

Authentication¶

The OTLP routes share the same auth chain as the rest of the ingestion service: API-key bearer tokens (configured via MICROMEGAS_API_KEYS) and OIDC.

OTel SDKs read OTEL_EXPORTER_OTLP_HEADERS and attach the parsed headers to every export request:

# Server side — same JSON keyring telemetry-ingestion-srv already uses
export MICROMEGAS_API_KEYS='[{"name":"team-platform","key":"mm_abc123def..."}]'

# Client side
export OTEL_EXPORTER_OTLP_ENDPOINT="https://micromegas.example.com/ingestion/otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer mm_abc123def..."

If different signals need different keys, use the per-signal headers variants:

export OTEL_EXPORTER_OTLP_LOGS_HEADERS="Authorization=Bearer key-for-logs"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer key-for-traces"

Per-signal headers override the catch-all.

TLS in production

Bearer tokens over plaintext leak in transit. Run the listener behind an HTTPS-terminating load balancer (or terminate TLS in-process via axum_server::tls_rustls). Plaintext is fine for localhost development only.

Variable expansion

OTel SDKs do not expand ${VAR} inside OTEL_EXPORTER_OTLP_HEADERS. Your shell expands those at export time. Config-file deployments that read headers from a JSON/YAML file need pre-substituted values or a wrapper script.

Process identity¶

OTLP has no "process" concept; it has a Resource (key/value attributes) attached to each batch. Micromegas synthesizes a stable process_id by hashing the OS-honest identifying tuple together with the OTel service identity:

process_id = uuid_v5(NS_OTEL_PROCESS_V1,
    host.id · host.name ·
    process.pid · process.creation.time ·
    service.namespace · service.name · service.instance.id · process.owner ·
    os.type · os.version · os.name · os.description · os.build_id ·
    host.arch · host.type ·
    host.image.id · host.image.name · host.image.version ·
    host.cpu.model.id · host.cpu.model.name · host.cpu.family ·
    host.cpu.vendor.id · host.cpu.stepping · host.cpu.cache.l2.size ·
    service.version ·
    telemetry.sdk.name · telemetry.sdk.language · telemetry.sdk.version ·
    process.runtime.name · process.runtime.version · process.runtime.description)

· denotes \x1F (ASCII unit separator). All fields pass through lower-case + trim except process.pid and process.creation.time which are used verbatim. Missing fields are treated as empty strings.

The formula was extended in-place under the same NS_OTEL_PROCESS_V1 namespace UUID — re-deriving existing process_ids is always acceptable, so no namespace bump is needed. In-flight processes receive a new process_id on their next batch; existing rows are unaffected and decay under the normal retention policy.

The first time a process_id is observed, a row is inserted into processes with these mappings:

OTel attribute	Process column
`service.name` (or `service.namespace + "/" + service.name`)	`exe`
`host.name`	`computer`
`user.name`	`username` / `realname`
`os.description`	`distro`
`host.cpu.model.name`	`cpu_brand`
`process.creation.time` (or first event time)	`start_time`
Everything else	`process.properties.otel.resource.*`

tsc_frequency is set to 1_000_000_000 so ticks ≡ Unix nanoseconds — OTel timestamps pass through the existing tick-to-time conversion as identity.

Stream identity¶

One stream per signal per process (max 3 streams per process):

stream_id = uuid_v5(NS_OTEL_STREAM_V1, process_id + "\x1F" + signal)

Stream tags reuse the existing micromegas vocabulary:

Signal	Stream tag	Stream format
logs	`"log"`	`otlp/v1/logs`
metrics	`"metrics"`	`otlp/v1/metrics`
traces	`"trace"`	`otlp/v1/traces`

The streams.format column (added in data-lake schema v4) tells the analytics layer which decoder to use per block; tags carry signal/purpose. log_entries and measures materialize blocks from both native and OTel streams uniformly.

Schema mapping¶

Logs → `log_entries`¶

OTel field	parquet column
`time_unix_nano` (or `observed_time_unix_nano` if zero)	`time`
`severity_number` 1–24	`level` (collapsed to the Micromegas `Level` enum: TRACE 1–4 → `6`, DEBUG 5–8 → `5`, INFO 9–12 → `4`, WARN 13–16 → `3`, ERROR 17–20 → `2`, FATAL 21–24 → `1`)
`body.string_value`	`msg`
`body.kvlist_value` / `array_value`	JSON-stringified into `msg`
`attributes.*`	`properties`
`instrumentation_scope.name`	`target`
`trace_id`, `span_id`	`properties.otel.trace_id` / `otel.span_id`
`severity_text`	`properties.otel.severity_text`

Scope identity (name, version, schema_url) and scope attributes land on per-row properties under the otel.scope.* prefix.

Metrics → `measures`¶

Sum and Gauge data points are materialized directly. Histogram, ExponentialHistogram, and Summary are skipped with a debug log in the current release — they will land in a follow-up that defines a histogram-aware schema.

OTel field	parquet column
`name`	`name`
`unit`	`unit`
`value` (int widened to f64)	`value`
`time_unix_nano`	`time`
`aggregation_temporality`, `is_monotonic`, `otel.metric.kind`	`properties`

Traces → `otel_spans`¶

otel_spans is a per-process JIT view — query it as view_instance('otel_spans', '<process_id>'). There is no global instance in the current release; cross-process trace traversal requires UNION-ing across the participating processes.

See Schema Reference: otel_spans for the full column list.

Attribute encoding¶

OTel KeyValue.value is an AnyValue oneof. JSONB encoding:

OTel `AnyValue` variant	JSONB representation
`string_value`	JSON string
`bool_value`	JSON bool
`int_value` (i64)	JSON number
`double_value`	JSON number (f64)
`bytes_value`	base64-encoded JSON string
`array_value`	JSON array, recursively encoded
`kvlist_value`	JSON object, recursively encoded

Nested structures are preserved. Query-time access uses the existing jsonb_* UDFs:

SELECT jsonb_as_string(jsonb_get(properties, 'otel.scope.name'))
FROM log_entries
WHERE process_id = '...';

HTTP semantics¶

Concern	Behavior
Body limit	20 MiB compressed (matches the OTel Collector's default `confighttp.max_request_body_size`)
Compression	`Content-Encoding: gzip` supported; other codecs return `415`
Content-Type	`application/x-protobuf` or `application/json` (parameters like `; charset=utf-8` accepted); other types return `415`
Empty top-level request	`200 OK` with empty `Export*ServiceResponse` body, no rows written (per spec)
Success	`200 OK`, response `Content-Type` mirrors the request encoding; body is an empty `Export*ServiceResponse`
Parse error	`400 Bad Request`, body is a `google.rpc.Status` proto with `code = INVALID_ARGUMENT (3)`
Auth failure	`401 Unauthorized`, body is `google.rpc.Status`
Body too large	`413 Payload Too Large`, body is `google.rpc.Status`
Unsupported media type	`415 Unsupported Media Type`, body is `google.rpc.Status`
Backend transient failure	`503 Service Unavailable` with `Retry-After: 30` header, body is `google.rpc.Status` (retryable per spec)

Per the OTLP spec, error responses always carry a google.rpc.Status proto, not an Export*ServiceResponse.

Idempotency¶

Block IDs are content-addressed: block_id = uuid_v5(NS_OTEL_BLOCK_V1, payload_bytes). Retried POSTs collide on ON CONFLICT (block_id) DO NOTHING and add no rows. This makes the OTLP endpoints safe to retry on transient errors without double-counting.

Client recipes¶

Claude Code¶

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_EXPORTER_OTLP_ENDPOINT="https://micromegas.example.com/ingestion/otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer mm_abc123def..."

# Optional — distributed tracing (Claude Code beta)
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp

# Optional — multi-team rollups via resource attributes
export OTEL_RESOURCE_ATTRIBUTES="team.id=platform,deployment.environment=prod"

claude

After Claude runs once, verify on the server:

SELECT process_id, exe, computer,
       jsonb_as_string(jsonb_get(properties, 'otel.resource.service.instance.id')) AS instance
FROM processes
WHERE jsonb_as_string(jsonb_get(properties, 'otel.resource.service.name')) = 'claude-code'
ORDER BY start_time DESC LIMIT 5;

SELECT count(*) FROM log_entries
WHERE process_id IN (
    SELECT process_id FROM processes
    WHERE jsonb_as_string(jsonb_get(properties, 'otel.resource.service.name')) = 'claude-code'
);

Python OTel SDK¶

import os

os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://127.0.0.1:9000/ingestion/otlp"
os.environ["OTEL_EXPORTER_OTLP_PROTOCOL"] = "http/protobuf"

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource.create({"service.name": "my-service", "service.instance.id": "i-1"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("hello"):
    pass

Go OTel SDK¶

export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:9000/ingestion/otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_SERVICE_NAME="my-service"

Then use otlptracehttp.New(ctx) (or the equivalent for logs/metrics) — it picks up the env vars.

OTLP/JSON & EventBridge API Destinations¶

AWS EventBridge API Destinations send Content-Type: application/json; charset=utf-8 by default, which is accepted by the ingestion server. Use an input transformer to produce the full ExportLogsServiceRequest envelope:

{
  "resourceLogs": [{
    "resource": { "attributes": [{"key": "service.name", "value": {"stringValue": "<$.source>"}}] },
    "scopeLogs": [{
      "scope": {"name": "eventbridge"},
      "logRecords": [{
        "timeUnixNano": "<$.time_ns>",
        "severityNumber": 9,
        "body": {"stringValue": "<$.detail.message>"}
      }]
    }]
  }]
}

timeUnixNano must be a quoted string in the template (e.g. "<$.time_ns>"). EventBridge input transformers substitute variables as strings inside quotes, satisfying the OTLP/JSON spec requirement. No Lambda translation layer is needed.

Webhook ingestion¶

POST /ingestion/webhook accepts a raw webhook delivery from any header-capable producer (GitLab, GitHub, a generic SaaS) with no per-source configuration on the server. It synthesizes an OTLP Resource from three request headers and stores the request body as a single log record's body, reusing the OTLP logs identity/block/write path end-to-end — the same auth, body-limit, and idempotency rules described above apply unchanged. Since log_entries.msg is Utf8-typed, a valid-UTF8 body (the common case: JSON payloads from GitLab/GitHub/etc.) is stored verbatim. There is no header to describe an alternate codec, so a non-UTF8 body is stored via lossy UTF-8 conversion (invalid byte sequences become U+FFFD) rather than rejected or stored as opaque binary.

Header	Maps to	Result
`X-Micromegas-Service-Name`	resource `service.name`	`processes.exe` / `log_entries.exe`
`X-Micromegas-Service-Namespace`	resource `service.namespace`	folded into `exe` as `namespace/name`, and into `process_id`
`X-Micromegas-Target`	instrumentation scope name	`log_entries.target`

All three headers are optional — a missing header behaves like an OTLP resource that omits the attribute. The body is never parsed or validated server-side; an empty body returns 400 Bad Request (nothing to store). Content-Type is not negotiated — send whatever the producer sends (typically application/json).

Because no per-record timestamp is known, time is the server's ingestion wall-clock time. Retried deliveries dedup via the same content-addressed block_id scheme described in Idempotency, with two webhook-specific wrinkles:

block_id is hashed from the full incoming header set, not just the 3 recognized ones. Only X-Micromegas-Service-Name/-Service-Namespace/-Target become resource attrs, but a producer-specific header this endpoint doesn't otherwise interpret (a GitLab delivery UUID, a GitHub event-type header, a signature) still changes block_id if it differs — otherwise two unrelated deliveries with byte-identical bodies but different unrecognized headers would collide and dedup as if they were retries of each other. The flip side: a genuine retry that picks up a new value for some header along the way (e.g. a proxy stamping a fresh Date or request-id on each hop) is no longer deduped, since that header now participates in the hash too.
The hash is computed before the server backfills the record's timestamp, so the wall-clock time written on a retry doesn't affect block_id — otherwise identical deliveries would never dedup, since the backfilled timestamp is different every time.

GitLab example¶

Configure a GitLab group or project webhook to point at the endpoint, with the three custom headers set once in the webhook configuration:

URL:     https://micromegas.example.com/ingestion/webhook
Headers: X-Micromegas-Service-Name: gitlab
         X-Micromegas-Service-Namespace: my-group
         X-Micromegas-Target: gitlab.push
         Authorization: Bearer mm_abc123def...

Every push/merge-request/pipeline event GitLab sends lands as one log_entries row with target = 'gitlab.push', exe = 'my-group/gitlab', and msg equal to the raw JSON payload GitLab sent.

Querying the stored body¶

The body is opaque JSON text in msg; parse it at query time with the jsonb_* UDFs (jsonb_parse, jsonb_get, jsonb_as_i64, jsonb_array_length, jsonb_path_query_first for nested/dotted access — there is no dotted-path variant of jsonb_get):

SELECT
  jsonb_as_string(jsonb_get(jsonb_parse(msg), 'object_kind')) AS kind,
  jsonb_as_i64(jsonb_path_query_first(jsonb_parse(msg), '$.object_attributes.iid')) AS iid,
  jsonb_array_length(jsonb_get(jsonb_parse(msg), 'commits')) AS nb_commits
FROM log_entries
WHERE target = 'gitlab.push'
ORDER BY time DESC
LIMIT 10;

CloudWatch Metric Streams (Kinesis Firehose)¶

POST /ingestion/otlp/v1/metrics/firehose speaks the Amazon Kinesis Data Firehose HTTP Endpoint Delivery protocol, so a CloudWatch Metric Stream can push metrics straight into micromegas: Metric Stream → Firehose → micromegas, with no Lambda, no Kinesis Data Stream, and no collector process in between. Firehose is just a dumb managed pipe: it wraps each record in a small JSON envelope and expects a fixed ack shape back.

This works because a Metric Stream configured with OpenTelemetry 1.0.0 output format delivers each record as an OTLP ExportMetricsServiceRequest protobuf — the exact message the native /ingestion/otlp/v1/metrics route already decodes. The Firehose route only unwraps the envelope (gzip-aware, base64 records) and hands each record's bytes to the same decode/split/write path; records land in measures, same as native OTLP metrics.

Requirement: OpenTelemetry 1.0.0 output format¶

The Metric Stream must be configured with OutputFormat: opentelemetry1.0 (or the equivalent console option). Other output formats (JSON, Parquet) are not OTLP and are not supported by this endpoint.

AWS delivery-stream setup¶

Configure a Kinesis Firehose delivery stream with an HTTP endpoint destination:

HTTP endpoint URL: https://micromegas.example.com/ingestion/otlp/v1/metrics/firehose
Access key: a micromegas API key — the value from MICROMEGAS_API_KEYS — sent by Firehose as X-Amz-Firehose-Access-Key on every request (Firehose cannot send Authorization: Bearer, so this route authenticates via that header instead, reusing the same keyring check as every other ingestion route).
Content encoding: gzip (recommended — reduces wire bytes; the route decompresses transparently, same as the other OTLP routes).
Buffering hints: tune buffer size/interval for your metric volume; every buffered batch arrives as one HTTP POST carrying one JSON record per underlying Metric Stream record.
S3 backup: configure "backup all records" or "backup failed data only" — Firehose retries non-200 responses and eventually spills to the configured S3 bucket, so no data is silently lost even during an extended micromegas outage.

Then point a CloudWatch Metric Stream at the delivery stream, with output format set to OpenTelemetry 1.0.0.

Ack contract¶

Success is 200 OK with Content-Type: application/json and body:

{"requestId": "<echoed from X-Amz-Firehose-Request-Id>", "timestamp": 1700000000000}

Any non-200 status triggers a Firehose retry, and body:

{"requestId": "<echoed>", "timestamp": 1700000000000, "errorMessage": "..."}

requestId always echoes the X-Amz-Firehose-Request-Id header — this is required by the Firehose HTTP Endpoint Delivery contract.

TLS in production

Same as the Bearer OTLP routes: X-Amz-Firehose-Access-Key over plaintext leaks in transit. Terminate TLS in front of the listener for any production delivery stream.

Idempotency¶

Same content-addressed block_id scheme as the rest of OTLP ingestion (see Idempotency): a Firehose retry of a previously-succeeded batch re-computes identical block_ids and dedups on write. On a partial batch failure, Firehose retries the whole batch — already-written records dedup, the failed one is retried. CloudWatch Metric Streams stamp distinct timestamps per scrape, so genuinely distinct data never collides.

CloudWatch Logs (Kinesis Firehose)¶

POST /ingestion/cloudwatch/v1/logs/firehose speaks the same Amazon Kinesis Data Firehose HTTP Endpoint Delivery protocol as the metrics route above, but for CloudWatch Logs subscription filters: CloudWatch Logs → subscription filter → Firehose → micromegas, with no Lambda, no Kinesis Data Stream, and no collector process in between.

Unlike the metrics route, this one is not OTLP-framed on the wire — CloudWatch Logs subscription-filter delivery has exactly one proprietary record format, gzip-compressed regardless of the delivery stream's own Content-Encoding setting. Once decoded, micromegas synthesizes an OTLP ExportLogsServiceRequest internally (one Resource, one LogRecord per logEvent) and feeds it through the same logs split/write path as native OTLP logs — so log_entries sees these rows exactly like any other log producer.

Payload format¶

Each Firehose record's data, after base64-decode, is gzip-compressed. Decompressed, it is CloudWatch's subscription-filter JSON:

{
  "messageType": "DATA_MESSAGE",
  "owner": "123456789012",
  "logGroup": "/ecs/my-service",
  "logStream": "ecs/my-service/abcd1234",
  "subscriptionFilters": ["my-filter"],
  "logEvents": [
    { "id": "...", "timestamp": 1510109208016, "message": "raw log line" }
  ]
}

CONTROL_MESSAGE records — which CloudWatch sends periodically to verify reachability — are recognized and dropped silently (not an error, no row written, no process registered).

AWS delivery-stream setup¶

Configure a Kinesis Firehose delivery stream with an HTTP endpoint destination, subscribed from a CloudWatch Logs log group via a subscription filter:

HTTP endpoint URL: https://micromegas.example.com/ingestion/cloudwatch/v1/logs/firehose
Access key: a micromegas API key — the value from MICROMEGAS_API_KEYS — sent by Firehose as X-Amz-Firehose-Access-Key on every request, same as the metrics route.
Content encoding: CloudWatch always gzips each record's payload at the source; this is independent of (and unaffected by) any additional Content-Encoding: gzip Firehose itself may apply to the whole HTTP body — both layers are handled transparently.
S3 backup: same recommendation as the metrics route — Firehose retries non-200 responses and eventually spills to S3, so no data is silently lost during an extended micromegas outage.

How `logGroup`/`logStream`/`owner` surface¶

service.name = logGroup, service.instance.id = logStream — feeds the same process_id_from_resource identity formula every other OTLP/OTel producer uses, so distinct log streams (distinct ECS tasks, Lambda instances, RDS instances) resolve to distinct process_ids with no CloudWatch-specific identity logic.
logGroup, logStream, and owner (AWS account id) are all set as resource attributes (aws.log.group.name, aws.log.stream.name, cloud.account.id), so they surface per-row via process_properties.otel.resource.* — the same discovery path as any other OTel resource attribute (see Process identity above).
The per-event CloudWatch id is attached as a record-level attribute (aws.log.event.id), queryable via properties, letting you correlate a log_entries row back to the exact CloudWatch event.

Cross-account collisions

cloud.account.id (owner) is not part of the process_id identity hash, so two different AWS accounts with the same logGroup+logStream names collapse onto the same process_id. Rows remain unambiguous (owner is still queryable per-row), only the process grouping is coarser than ideal across accounts — most relevant for RDS Postgres logs, where stream names are user-chosen DB-instance identifiers that can repeat across environments.

Ack contract¶

Same as CloudWatch Metric Streams: 200 OK with {"requestId": "<echoed>", "timestamp": ...} on success; any non-200 status (with errorMessage) triggers a Firehose retry.

Idempotency¶

Same content-addressed block_id scheme as the rest of OTLP ingestion (see Idempotency): a Firehose retry of a previously-succeeded batch dedups on write. CloudWatch Logs events carry real per-event timestamps (no backfill), so genuinely distinct log lines never collide.

TLS in production

Same as the metrics Firehose route: X-Amz-Firehose-Access-Key over plaintext leaks in transit. Terminate TLS in front of the listener for any production delivery stream.

Limitations¶

OTLP/HTTP only. gRPC OTLP is not implemented; SDKs that default to gRPC need OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.
OTLP/JSON: string-encoded 64-bit fields required. The OTLP/JSON spec mandates "timeUnixNano" and similar 64-bit integer fields as quoted strings (e.g. "1700000000000000000"). Bare JSON numbers are rejected. Conformant OTel SDKs and EventBridge input transformers produce the string form automatically.
No mTLS / client certs. Only bearer-token and OIDC auth.
Histograms not yet materialized. Sum and Gauge land in measures; Histogram, ExponentialHistogram, and Summary are skipped with a debug log.
otel_spans is JIT-only and per-process. Cross-process trace queries (WHERE trace_id = X across all services) need to UNION across each participating process.
parse_block does not decode OTel payloads. It returns a clean error on format != "micromegas-transit".
No per-tenant rate limiting. Add at the load balancer if needed.

Troubleshooting¶

415 Unsupported Media Type — the SDK is sending an unsupported Content-Type or omitting it entirely. Accepted types are application/x-protobuf and application/json. Other compression codecs (deflate, zstd) also return 415; only gzip is accepted.

401 Unauthorized — verify the bearer token matches an entry in MICROMEGAS_API_KEYS on the server. Check that the SDK is actually attaching the header (OTEL_EXPORTER_OTLP_HEADERS is processed at export time, not at SDK init — typos are silently ignored).

413 Payload Too Large — the compressed body exceeds 20 MiB. Lower the SDK's batch size (OTEL_BSP_MAX_EXPORT_BATCH_SIZE, OTEL_BLRP_MAX_EXPORT_BATCH_SIZE) or split into more frequent exports.

Process collapses across runs — the formula expects service.instance.id to vary per OS process. If your SDK omits it (some FaaS configurations), every invocation hashes to the same process_id. Set it explicitly via OTEL_RESOURCE_ATTRIBUTES=service.instance.id=$(uuidgen) or have the SDK generate one.

process_id looks identical across very different services — host.id, host.name, process.pid, and service.instance.id all came back empty. Check the resource detector configuration on the SDK side; the server logs a degenerate-resource warning when this happens.

Logs without an explicit severity appear with level = 4 (Info) — severity_number = 0 (UNSPECIFIED) maps to Info so unspecified records pass the default WHERE level <= 4 filter (lower number = more severe in micromegas; level <= 4 keeps Info-and-more-severe). Set severity_number explicitly on the SDK side if you want a different mapping.

Trace queries return nothing — otel_spans is a JIT view and only materializes when queried with a specific process_id. Use view_instance('otel_spans', '<process_id>'), not FROM otel_spans. Find the right process_id via the processes view first.

OTLP Ingestion¶

Overview¶

Quick Start¶

Authentication¶

Process identity¶

Stream identity¶

Schema mapping¶

Logs → log_entries¶

Metrics → measures¶

Traces → otel_spans¶

Attribute encoding¶

HTTP semantics¶

Idempotency¶

Client recipes¶

Claude Code¶

Python OTel SDK¶

Go OTel SDK¶

OTLP/JSON & EventBridge API Destinations¶

Webhook ingestion¶

GitLab example¶

Querying the stored body¶

CloudWatch Metric Streams (Kinesis Firehose)¶

Requirement: OpenTelemetry 1.0.0 output format¶

AWS delivery-stream setup¶

Ack contract¶

Idempotency¶

CloudWatch Logs (Kinesis Firehose)¶

Payload format¶

AWS delivery-stream setup¶

How logGroup/logStream/owner surface¶

Ack contract¶

Idempotency¶

Limitations¶

Troubleshooting¶

References¶

Logs → `log_entries`¶

Metrics → `measures`¶

Traces → `otel_spans`¶

How `logGroup`/`logStream`/`owner` surface¶