Crate micromegas

Expand description

Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.

§Very high level architecture

┌─────────────────┐
│ rust application│──────▶
└─────────────────┘       ┌─────────┐     ┌───────┐     ┌─────────┐     ┌──────────┐
                          │ingestion│────▶│pg & S3│◀────│analytics│◀────│python API│
┌─────────────────┐       └─────────┘     └───────┘     └─────────┘     └──────────┘
│ unreal engine   │──────▶
└─────────────────┘

§Cargo features

default: instrumentation only (micromegas::tracing and the telemetry sink).
client: the FlightSQL query client in micromegas::client::flightsql_client and its factory, without the service stack. Re-exports arrow, arrow_flight and tonic; TimeRange is at micromegas::telemetry::time_range::TimeRange.
server: everything (ingestion, analytics, auth, servers). Implies client.

§Rust Instrumentation

For rust applications, use micromegas::tracing for minimal overhead. Interoperability with tokio tracing’s logs is also enabled by default.

§Unreal instrumentation

MicromegasTracing should be added to Unreal’s Core module and MicromegasTelemetrySink can be added to a game or to a high level plugin. See https://github.com/madesroches/micromegas/tree/main/unreal for implementation. It has been tested in editor, client and server builds on multiple platforms.

§Local developer configuration

For testing purposes, you can run the entire stack on your local workstation.

§Environment variables

MICROMEGAS_DB_USERNAME and MICROMEGAS_DB_PASSWD: used by the database configuration script
export MICROMEGAS_TELEMETRY_URL=http://localhost:9000
export MICROMEGAS_SQL_CONNECTION_STRING=postgres://{uname}:{passwd}@localhost:5432
export MICROMEGAS_OBJECT_STORE_URI=file:///some/local/path

Clone the github repository

> git clone https://github.com/madesroches/micromegas.git

Start a local instance of postgresql (requires docker and python)

> cd micromegas/local_test_env/db
> ./run.py

In a new shell, start the ingestion server

> cd micromegas/rust
> cargo run -p telemetry-ingestion-srv -- --ingestion-listen-addr 127.0.0.1:9000

In a new shell, start the flightsql server

> cd micromegas/rust
> cargo run -p flight-sql-srv -- --disable-auth

In a new shell, start the daemon

> cd micromegas/rust
> cargo run -p telemetry-maintenance-srv

In a python interpreter, query the analytics service

# local connection test
import datetime
import micromegas
client = micromegas.connect() #connects to localhost by default
now = datetime.datetime.now(datetime.timezone.utc)
begin = now - datetime.timedelta(days=1)
end = now
sql = """
SELECT *
FROM log_entries
ORDER BY time DESC
LIMIT 10
;"""
df = client.query(sql, begin, end)
df #dataframe containing the result of the query

Re-exports§

pub use chrono;
pub use uuid;
pub use arrow;
pub use arrow_flight;
pub use tonic;
pub use axum;
pub use datafusion;
pub use object_store;
pub use prost;
pub use sqlx;

Modules§

analytics: makes the telemetry data lake accessible and useful
auth: authentication providers
client
config: CLI args shared by every long-running server binary.
ingestion: records telemetry in data lake
object_cache: shared object range cache
perfetto: perfetto protobufs
servers
telemetry: telemetry protocol
telemetry_sink: publication of the recorded events using http
tracing: low level tracing - has minimal dependencies
transit: binary heterogeneous queue serialization, used by the tracing event blocks

Macros§

declare_jemalloc_conf: re-export of declare_jemalloc_conf!, kept behind jemalloc-metrics (rather than always available) so an app depending on micromegas only for TelemetryGuardBuilder doesn’t find a server-tuned allocator conf in its API surface. See mkdocs/docs/admin/memory-allocator.md. Declares the _rjem_malloc_conf symbol jemalloc’s conf.c resolves at precedence 1 (weaker than _RJEM_MALLOC_CONF, stronger than the compile-time --with-malloc-conf default), pointing it at CONF.

Attribute Macros§

micromegas_main: micromegas_main: Creates a tokio runtime with proper micromegas tracing callbacks and telemetry setup