Skip to content

Build Guide

This guide covers building Micromegas from source and setting up a development environment.

Prerequisites

  • Rust - Toolchain version pinned in rust/rust-toolchain.toml; rustup will install it automatically on first build
  • Python 3.8+
  • Docker - For running PostgreSQL
  • Git
  • Build tools - C/C++ compiler and linker (required for Rust compilation)
  • Linux: sudo apt-get install build-essential clang mold
  • macOS: xcode-select --install
  • Windows: Install Visual Studio Build Tools

mold linker requirement

On Linux, the project requires the mold linker as configured in .cargo/config.toml. This provides faster linking for large projects.

Additional CI Tools

For running the full CI pipeline locally, you'll need:

# Install cargo-machete for unused dependency checking
cargo install cargo-machete

Rust Development

Clone and Build

git clone https://github.com/madesroches/micromegas.git
cd micromegas/rust

# Build all components
cargo build

# Build with optimizations
cargo build --release

# Build specific component
cargo build -p telemetry-ingestion-srv

Testing

# Run all tests
cargo test

# Run tests with output
cargo test -- --nocapture

# Run specific test
cargo test -p micromegas-tracing

Format and Lint

# Format code (required before commits)
cargo fmt

# Run linter
cargo clippy --workspace -- -D warnings

# Run full CI pipeline
python3 ../build/rust_ci.py

Advanced Builds

# Clean build
cargo clean && cargo build

# Cross-compile for Windows (from Linux)
rustup target add x86_64-pc-windows-gnu
cargo build --target x86_64-pc-windows-gnu

ARM64 Cross-Compilation

Production Docker images support ARM64 (aarch64) via a Docker-based cross-compilation environment. The toolchain lives in local_test_env/arm64/.

# Build the ARM64 cross-compilation Docker image
cd local_test_env/arm64
python3 build.py

# Run the image to cross-compile (mounts the repo)
python3 run.py

The Dockerfile installs g++-aarch64-linux-gnu, adds the aarch64-unknown-linux-gnu Rust target, and cross-compiles OpenSSL statically for ARM64. The build_docker_images.py script at the repo root also accepts --arm64 to build production images for ARM64.

python3 build/build_docker_images.py --arm64

JavaScript / TypeScript Development

The repository has two independent JS/TS workspaces. Both use Yarn 4 (Berry) — run corepack enable once per machine to activate the pinned version.

Analytics Web App (analytics-web-app/)

Vite + React 19 frontend for the analytics UI.

cd analytics-web-app

corepack enable     # Once per machine
yarn install        # Install dependencies

yarn dev            # Vite dev server on port 3000
yarn build          # Production build to dist/
yarn lint           # ESLint
yarn type-check     # TypeScript check (no emit)
yarn test           # Vitest unit tests

Grafana Plugin (grafana/)

Grafana datasource plugin (React frontend + Go backend).

Additional prerequisites: Go 1.23+ and mage (go install github.com/magefile/mage@latest).

cd grafana

corepack enable     # Once per machine
yarn install        # Install Node dependencies

mage -v build       # Build Go backend binaries
yarn build          # Production bundle
yarn dev            # Dev mode with hot reload
yarn test:ci        # Tests
yarn lint:fix       # Lint + autofix
yarn server         # Start Grafana via docker compose at http://localhost:3000

Python Development

cd python/micromegas

# Install dependencies
poetry install

# Run tests
pytest

# Format code (required before commits)
black .

Running a Development Instance

Once you've built the components above, you can run a full local stack backed by your own build instead of the published Docker image (see Getting Started for the Docker quickstart).

Environment Variables

# Database credentials (used by setup scripts)
export MICROMEGAS_DB_USERNAME=your_username
export MICROMEGAS_DB_PASSWD=your_password

# Service endpoints
export MICROMEGAS_TELEMETRY_URL=http://localhost:9000
export MICROMEGAS_SQL_CONNECTION_STRING=postgres://your_username:your_password@localhost:5432

# Object storage (replace with your local path)
export MICROMEGAS_OBJECT_STORE_URI=file:///path/to/local/storage

Object Storage Path

Choose a local directory for object storage, e.g., /tmp/micromegas-storage or C:\temp\micromegas-storage on Windows.

Advanced: transport tuning

The Rust telemetry sink exposes environment variables for queue size limits, concurrency, and timeouts if the ingestion service falls behind or becomes unreachable. See Telemetry Sink Transport Tuning.

The simplest way to start everything is the monolith script, which builds and launches a single micromegas-monolith process running all roles (ingestion, analytics, web, maintenance):

python3 local_test_env/ai_scripts/start_services.py --monolith

This will automatically:

  • Build the monolith binary and the analytics web app (including DataFusion WASM)
  • Start PostgreSQL if not already running
  • Launch micromegas-monolith --roles all on port 9000 (HTTP/ingestion), port 50051 (FlightSQL), and port 3000 (web app)
  • Write all PIDs to /tmp/micromegas_pids.txt
# Stop all services
python3 local_test_env/ai_scripts/stop_services.py

Option B: Split Services

To run the four services separately (closer to a production topology):

python3 local_test_env/ai_scripts/start_services.py

Option C: Manual Startup

If you prefer full control, start each service in a separate terminal:

Terminal 1: PostgreSQL Database 

cd local_test_env/db
python run.py

Terminal 2: Ingestion Server 

cd rust
cargo run -p telemetry-ingestion-srv -- --listen-endpoint-http 127.0.0.1:9000

Terminal 3: FlightSQL Server 

cd rust
cargo run -p flight-sql-srv -- --disable-auth

Terminal 4: Maintenance Service 

cd rust
cargo run -p telemetry-maintenance-srv

Service Roles

  • PostgreSQL: Stores metadata and service configuration
  • Ingestion Server: Receives telemetry data from applications (port 9000)
  • FlightSQL Server: Provides SQL query interface for analytics (port 50051)
  • Maintenance Service: Handles background processing and global view materialization

Documentation

# Install dependencies
pip install -r mkdocs/docs-requirements.txt

# Start development server
cd mkdocs
mkdocs serve

# Build static site
mkdocs build

Self-Hosted CI Runner

Developer workstations can contribute to CI builds using a Docker-based self-hosted GitHub Actions runner. Builds from the repo owner route to the dev worker when it's online, falling back to GitHub-hosted runners when it's not.

Prerequisites

  • Docker
  • A fine-grained GitHub PAT with Administration: Read and write scoped to madesroches/micromegas

Setup

Store the PAT locally (choose one):

# Option 1: environment variable
export MICROMEGAS_RUNNER_PAT=ghp_xxx

# Option 2: file (recommended for persistent use)
mkdir -p ~/.config/micromegas
echo "ghp_xxx" > ~/.config/micromegas/runner-pat
chmod 600 ~/.config/micromegas/runner-pat

The same PAT must be stored as the repository secret RUNNER_PAT:

gh secret set RUNNER_PAT

Usage

# Start the worker (runs until Ctrl+C)
python3 build/dev_worker.py

# With resource limits
python3 build/dev_worker.py --cpus 8 --memory 16g

# Build the container image without starting the worker
python3 build/dev_worker.py --build-image

# Remove offline dev-worker runners from GitHub and exit
python3 build/dev_worker.py --cleanup

How It Works

Each workflow has a check-runner job that runs on ubuntu-latest and decides where the real jobs run:

  1. If the build author is the repo owner and a dev worker is online, jobs route to dev-worker
  2. Otherwise, jobs run on ubuntu-latest (existing behavior)

The runner container is ephemeral: each container registers with GitHub, picks up one job, executes it, and exits. The worker loop then starts a fresh container for the next job. Each container gets a unique name so successive runs cannot collide while Docker drains the previous --rm cleanup.

The build caches live in a named Docker volume (micromegas-runner-cache) mounted at /cache, so they persist across the ephemeral containers. The volume holds:

  • Cargo registry and target directories (CARGO_HOME, CARGO_TARGET_DIR)
  • Yarn package downloads (YARN_CACHE_FOLDER)
  • Go module and build cache (GOMODCACHE, GOCACHE)
  • Playwright browser downloads (PLAYWRIGHT_BROWSERS_PATH)

This assumes a single worker per workstation — two concurrent workers sharing the volume would corrupt cargo's locks. To wipe the cache, stop the worker and run docker volume rm micromegas-runner-cache.

See tasks/container_based_dev_worker_plan.md for the full design.

Next Steps