Python API Reference¶

Reference for the Micromegas Python client: connection options, client methods, and the bundled CLI tools.

Installation¶

Install the Micromegas Python client from PyPI:

pip install micromegas

Basic Usage¶

Connection¶

import micromegas

# Connect to local Micromegas instance
client = micromegas.connect()

# Connect with dictionary encoding preservation (for memory efficiency)
client = micromegas.connect(preserve_dictionary=True)

The connect() function connects to the analytics service at a plain URI. It never reads ~/.micromegas/config.json and never authenticates the connection it returns.

The installed package version is available as micromegas.__version__.

Parameters: - preserve_dictionary (bool, optional): Enable dictionary encoding preservation for memory-efficient data transfer. Default: False

Connecting via a named profile¶

For a remote or authenticated deployment, use connect_with_profile() instead. It reads ~/.micromegas/config.json (see Config file (~/.micromegas/config.json) below) and picks the auth mechanism the resolved profile names -- a static API key, OIDC, or none:

import micromegas

# Uses the "prod" profile from ~/.micromegas/config.json
client = micromegas.connect_with_profile("prod")

Simple Queries¶

import datetime

# Set up time range
now = datetime.datetime.now(datetime.timezone.utc)
begin = now - datetime.timedelta(hours=1)
end = now

# Execute query with time range
sql = "SELECT * FROM log_entries LIMIT 10;"
df = client.query(sql, begin, end)
print(df)

Client Methods¶

`query(sql, begin=None, end=None)`¶

Execute a SQL query and return results as a pandas DataFrame.

Parameters:

sql (str): SQL query string
begin (datetime or str, optional): ⚡ Recommended - Start time for partition elimination. Can be a datetime object or RFC3339 string (e.g., "2024-01-01T00:00:00Z")
end (datetime or str, optional): ⚡ Recommended - End time for partition elimination. Can be a datetime object or RFC3339 string (e.g., "2024-01-01T23:59:59Z")

Returns:

pandas.DataFrame: Query results

Performance Note: Using begin and end parameters instead of SQL time filters allows the analytics server to eliminate entire partitions before query execution, providing significant performance improvements.

Example:

# ✅ EFFICIENT: API time range enables partition elimination
df = client.query("""
    SELECT time, process_id, level, msg
    FROM log_entries
    WHERE level <= 3
    ORDER BY time DESC
    LIMIT 100;
""", begin, end)  # ⭐ Time range in API parameters

# ❌ INEFFICIENT: SQL time filter scans all partitions
df = client.query("""
    SELECT time, process_id, level, msg
    FROM log_entries
    WHERE time >= NOW() - INTERVAL '1 hour'  -- Server scans ALL partitions
      AND level <= 3
    ORDER BY time DESC
    LIMIT 100;
""")  # Missing API time parameters!

# ✅ Using RFC3339 strings for time ranges
df = client.query("""
    SELECT time, process_id, level, msg
    FROM log_entries
    WHERE level <= 3
    ORDER BY time DESC
    LIMIT 100;
""", "2024-01-01T00:00:00Z", "2024-01-01T23:59:59Z")  # ⭐ RFC3339 strings

# ✅ OK: Query without time range (for metadata queries)
processes = client.query("SELECT process_id, exe FROM processes LIMIT 10;")

`query_stream(sql, begin=None, end=None)`¶

Execute a SQL query and return results as a stream of Apache Arrow RecordBatch objects. Use this for large datasets to avoid memory issues.

Parameters:

sql (str): SQL query string
begin (datetime or str, optional): ⚡ Recommended - Start time for partition elimination. Can be a datetime object or RFC3339 string (e.g., "2024-01-01T00:00:00Z")
end (datetime or str, optional): ⚡ Recommended - End time for partition elimination. Can be a datetime object or RFC3339 string (e.g., "2024-01-01T23:59:59Z")

Returns:

Iterator of pyarrow.RecordBatch: Stream of result batches

Example:

import pyarrow as pa

# Stream large dataset
sql = """
    SELECT time, process_id, level, target, msg
    FROM log_entries
    WHERE time >= NOW() - INTERVAL '7 days'
    ORDER BY time DESC;
"""

for record_batch in client.query_stream(sql, begin, end):
    # record_batch is a pyarrow.RecordBatch
    print(f"Batch shape: {record_batch.num_rows} x {record_batch.num_columns}")
    print(f"Schema: {record_batch.schema}")

    # Convert to pandas for analysis
    df = record_batch.to_pandas()

    # Process this batch
    error_logs = df[df['level'] <= 3]
    if not error_logs.empty:
        print(f"Found {len(error_logs)} errors in this batch")
        # Process errors...

    # Memory is automatically freed after each batch

`query_arrow(sql, begin=None, end=None)`¶

Execute a SQL query and return results as an Apache Arrow Table. This method preserves dictionary encoding when preserve_dictionary=True is set during connection.

Parameters:

sql (str): SQL query string
begin (datetime or str, optional): Start time for partition elimination
end (datetime or str, optional): End time for partition elimination

Returns:

pyarrow.Table: Query results as Arrow Table

Example:

# Connect with dictionary preservation
dict_client = micromegas.connect(preserve_dictionary=True)

# Get Arrow table with preserved dictionary encoding
table = dict_client.query_arrow("""
    SELECT properties_to_dict(properties) as dict_props
    FROM measures
""", begin, end)

# Check if column uses dictionary encoding
print(f"Dictionary encoded: {pa.types.is_dictionary(table.schema.field('dict_props').type)}")
print(f"Memory usage: {table.nbytes:,} bytes")

Dictionary Encoding for Memory Efficiency¶

For large datasets with repeated values (like properties), dictionary encoding cuts memory usage by 50-80%.

Using Dictionary-Encoded Properties¶

# Connect with dictionary preservation enabled
client = micromegas.connect(preserve_dictionary=True)

# Use properties_to_dict UDF for dictionary encoding
sql = """
SELECT 
    time,
    process_id,
    properties_to_dict(properties) as dict_props,
    properties_length(properties_to_dict(properties)) as prop_count
FROM measures
WHERE time >= NOW() - INTERVAL '1 hour'
"""

# Option 1: Get as pandas DataFrame (automatic conversion)
df = client.query(sql, begin, end)
print(f"DataFrame shape: {df.shape}")
print(f"Memory usage: {df.memory_usage(deep=True).sum():,} bytes")

# Option 2: Get as Arrow Table (preserves dictionary encoding)
table = client.query_arrow(sql, begin, end)
print(f"Arrow table memory: {table.nbytes:,} bytes")

# Dictionary encoding typically uses 50-80% less memory

Compatibility with Standard Functions¶

Dictionary-encoded data works seamlessly with Micromegas UDFs:

sql = """
SELECT 
    -- Direct property access
    property_get(properties, 'source') as source,

    -- Length calculation (works with both formats)
    properties_length(properties) as regular_count,
    properties_length(properties_to_dict(properties)) as dict_count,

    -- Convert back to array when needed
    array_length(properties_to_array(properties_to_dict(properties))) as array_count
FROM measures
"""

df = client.query(sql, begin, end)

Working with Results¶

pandas DataFrames¶

All query() results are pandas DataFrames, giving you access to the full pandas ecosystem:

# Basic DataFrame operations
result = client.query("SELECT process_id, exe, start_time FROM processes;")

# Inspect the data
print(f"Shape: {result.shape}")
print(f"Columns: {result.columns.tolist()}")
print(f"Data types:\n{result.dtypes}")

# Filter and analyze
recent = result[result['start_time'] > datetime.datetime.now() - datetime.timedelta(days=1)]
print(f"Recent processes: {len(recent)}")

# Group and aggregate
by_exe = result.groupby('exe').size().sort_values(ascending=False)
print("Processes by executable:")
print(by_exe.head())

pyarrow RecordBatch¶

Streaming queries return Apache Arrow RecordBatch objects:

for batch in client.query_stream(sql, begin, end):
    # RecordBatch properties
    print(f"Rows: {batch.num_rows}")
    print(f"Columns: {batch.num_columns}")
    print(f"Schema: {batch.schema}")

    # Access individual columns
    time_column = batch.column('time')
    level_column = batch.column('level')

    # Convert to pandas (zero-copy operation)
    df = batch.to_pandas()

    # Convert to other formats
    table = batch.to_pylist()  # List of dictionaries
    numpy_dict = batch.to_pydict()  # Dictionary of numpy arrays

Connection Configuration¶

`FlightSQLClient(uri, headers=None, preserve_dictionary=False, auth_provider=None, client_entrypoint=None)`¶

For advanced connection scenarios, use the FlightSQLClient class directly:

from micromegas.flightsql.client import FlightSQLClient

# Recommended: Connect with OIDC authentication (automatic token refresh)
from micromegas.auth import OidcAuthProvider

auth = OidcAuthProvider.from_file("~/.micromegas/tokens.json")
client = FlightSQLClient(
    "grpc+tls://remote-server:50051",
    auth_provider=auth
)

# Connect with a static analytics API key
from micromegas.auth import StaticTokenAuthProvider

auth = StaticTokenAuthProvider.from_file("~/.micromegas/local.key")
client = FlightSQLClient(
    "grpc+tls://remote-server:50051",
    auth_provider=auth
)

# Connect with dictionary preservation for memory efficiency
client = FlightSQLClient(
    "grpc://localhost:50051",
    preserve_dictionary=True
)

# Connect to local server (equivalent to micromegas.connect())
client = FlightSQLClient("grpc://localhost:50051")

Parameters: - uri (str): FlightSQL server URI. Use grpc:// for unencrypted or grpc+tls:// for TLS connections - headers (dict, optional): Deprecated. Use auth_provider instead. Static headers for authentication. This parameter is deprecated because it doesn't support automatic token refresh - preserve_dictionary (bool, optional): When True, preserve dictionary encoding in Arrow arrays for memory efficiency. Useful when using dictionary-encoded UDFs. Defaults to False - auth_provider (optional): Recommended. Authentication provider that implements get_token() method. When provided, tokens are automatically refreshed before each request. Example: OidcAuthProvider (browser/PKCE with refresh) or StaticTokenAuthProvider (a pre-minted analytics API key, sent verbatim). This is the recommended way to handle authentication - client_entrypoint (str, optional): Explicit label for how this client was invoked (e.g. "cli-query", the label micromegas-query passes). When omitted, the entrypoint is auto-detected. Raises ValueError if the value isn't a safe gRPC header value (printable ASCII, 64 chars or fewer). See Client Attribution below.

See the Authentication Guide for setting up OIDC authentication.

Client Attribution¶

Every query sends three additional headers, resolved once per FlightSQLClient instance and recorded in the server's query audit log:

x-client-agent — who is driving the client. Auto-detected from known agent-harness marker environment variables (currently just Claude Code's CLAUDECODE, reported as claude-code); "none" when no known harness is detected. Override with MICROMEGAS_CLIENT_AGENT.
x-client-entrypoint — how the client was invoked: "script", "jupyter", or "repl", auto-detected from the interpreter's own state (sys.modules, sys.argv, sys.flags). The CLI passes the explicit label "cli-query" instead of relying on auto-detection. Override with the client_entrypoint constructor parameter (raises ValueError on an unsafe value) or MICROMEGAS_CLIENT_ENTRYPOINT (silently falls back to the detected value on an unsafe value).
x-client-session — an opaque id correlating every query issued through one client instance. A fresh UUID per FlightSQLClient instance, unless a known agent harness's session environment variable is present (currently just Claude Code's CLAUDE_CODE_SESSION_ID), in which case that value is reused verbatim so multiple FlightSQLClient instances within the same agent session correlate (this is what lets micromegas-query, which builds a fresh client per invocation, still correlate queries from the same agent session).

These are analytics-only signals — never used for authentication, quota, or rate limiting — and, like the existing x-client-type header, trivially spoofable or omittable by the caller. MICROMEGAS_CLIENT_AGENT/MICROMEGAS_CLIENT_ENTRYPOINT overrides must be printable ASCII, 64 characters or fewer; an invalid override is silently ignored in favor of the auto-detected value (unlike the client_entrypoint constructor parameter, which raises ValueError instead).

Of the two top-level connect functions, plain micromegas.connect() does not expose a client_entrypoint parameter — auto-detection always runs for it. micromegas.connect_with_profile() does take it, forwarded on every auth branch. For an explicit label, use connect_with_profile(), construct FlightSQLClient directly, or use oidc_connection.connect()/the now-aliased cli.connection.connect(), all of which take client_entrypoint.

Static Analytics API Keys¶

A static analytics API key is minted server-side — via POST /api/analytics-api-keys or the Admin page — and travels verbatim as the bearer token on every request; there is no exchange or wrapping. Store it in a file readable only by you (chmod 0600) and load it with StaticTokenAuthProvider.from_file(...):

from micromegas.auth import StaticTokenAuthProvider
from micromegas.flightsql.client import FlightSQLClient

auth = StaticTokenAuthProvider.from_file("~/.micromegas/local.key")
client = FlightSQLClient("grpc+tls://remote-server:50051", auth_provider=auth)

See Minting an analytics key over HTTP for how to mint a key.

Schema Discovery¶

`prepare_statement(sql)`¶

Get query schema information without executing the query:

# Prepare statement to discover schema
stmt = client.prepare_statement(
    "SELECT time, level, msg FROM log_entries WHERE level <= 3"
)

# Inspect the schema
print("Query result schema:")
for field in stmt.dataset_schema:
    print(f"  {field.name}: {field.type}")

# Output:
#   time: timestamp[ns]
#   level: int32  
#   msg: string

# The query is also available
print(f"Query: {stmt.query}")

`prepared_statement_stream(statement)`¶

Execute a prepared statement (mainly useful after schema inspection):

# Execute the prepared statement
for batch in client.prepared_statement_stream(stmt):
    df = batch.to_pandas()
    print(f"Received {len(df)} rows")

Note: Prepared statements are primarily for schema discovery. Execution offers no performance benefit over query_stream().

Process and Stream Discovery¶

`find_process(process_id)`¶

Find detailed information about a specific process:

# Find process by ID
process_info = client.find_process('550e8400-e29b-41d4-a716-446655440000')

if not process_info.empty:
    print(f"Process: {process_info['exe'].iloc[0]}")
    print(f"Started: {process_info['start_time'].iloc[0]}")
    print(f"Computer: {process_info['computer'].iloc[0]}")
else:
    print("Process not found")

`query_streams(begin, end, limit, process_id=None, tag_filter=None)`¶

Query event streams with filtering:

# Query all streams from the last hour
end = datetime.datetime.now(datetime.timezone.utc)
begin = end - datetime.timedelta(hours=1)
streams = client.query_streams(begin, end, limit=100)

# Filter by process
process_streams = client.query_streams(
    begin, end, 
    limit=50,
    process_id='550e8400-e29b-41d4-a716-446655440000'
)

# Filter by stream tag
log_streams = client.query_streams(
    begin, end,
    limit=20, 
    tag_filter='log'
)

print(f"Found {len(streams)} total streams")
print(f"Streams per process:\n{streams['process_id'].value_counts()}")

`query_blocks(begin, end, limit, stream_id)`¶

Query data blocks within a stream (for low-level inspection):

# First find a stream
streams = client.query_streams(begin, end, limit=1)
if not streams.empty:
    stream_id = streams['stream_id'].iloc[0]

    # Query blocks in that stream
    blocks = client.query_blocks(begin, end, 100, stream_id)
    print(f"Found {len(blocks)} blocks")
    print(f"Total objects: {blocks['nb_objects'].sum()}")
    print(f"Total size: {blocks['payload_size'].sum()} bytes")

`query_spans(begin, end, limit, stream_id)`¶

Query execution spans for performance analysis:

# Query spans for detailed performance analysis
spans = client.query_spans(begin, end, 1000, stream_id)

# Find slowest operations
slow_spans = spans.nlargest(10, 'duration')
print("Slowest operations:")
for _, span in slow_spans.iterrows():
    duration_ms = span['duration'] / 1000000  # Convert nanoseconds to milliseconds
    print(f"  {span['name']}: {duration_ms:.2f}ms")

# Analyze span hierarchy (root spans have depth 0)
root_spans = spans[spans['depth'] == 0]
print(f"Found {len(root_spans)} root operations")

Data Management¶

`bulk_ingest(table_name, table)`¶

Requires admin

Only callable by a caller whose resolved local-group membership includes the reserved admins group (or a --disable-auth deployment) — see Groups and Admin SQL Functions. An API-key credential (both ingestion_api_keys and analytics_api_keys) normally carries no email and so can never match a user:/group: member of admins — except while admins still holds its seeded wildcard (*) member, in which case every authenticated caller, API keys included, is admin until an operator adds a user: member and removes * (see Groups's upgrade path).

Bulk ingest metadata for replication or administrative tasks. table is a pyarrow.Table whose schema matches the target table exactly; complex columns (struct, list, binary) are passed through natively.

import pyarrow as pa

# Example: Replicate process metadata
processes = pa.table({
    'process_id': ['550e8400-e29b-41d4-a716-446655440000'],
    'exe': ['/usr/bin/myapp'],
    'username': ['user'],
    'realname': ['User Name'],
    'computer': ['hostname'],
    'distro': ['Ubuntu 22.04'],
    'cpu_brand': ['Intel Core i7'],
    'tsc_frequency': [2400000000],
    'start_time': [datetime.datetime.now(datetime.timezone.utc)],
    'start_ticks': [1234567890],
    'insert_time': [datetime.datetime.now(datetime.timezone.utc)],
    'parent_process_id': [''],
    'properties': [[]],
    'audience': ['public'],
})

# Ingest process metadata
result = client.bulk_ingest('processes', processes)
if result:
    print(f"Ingested {result.record_count} process records")

streams and blocks need the same column, alongside their other required fields:

streams = pa.table({
    'stream_id': ['...'],
    'process_id': ['550e8400-e29b-41d4-a716-446655440000'],
    'dependencies_metadata': [b''],
    'objects_metadata': [b''],
    'tags': [[]],
    'properties': [[]],
    'insert_time': [datetime.datetime.now(datetime.timezone.utc)],
    'format': ['micromegas-transit'],
    'audience': ['public'],
})

blocks = pa.table({
    'block_id': ['...'],
    'stream_id': ['...'],
    'process_id': ['550e8400-e29b-41d4-a716-446655440000'],
    'begin_time': [datetime.datetime.now(datetime.timezone.utc)],
    'begin_ticks': [0],
    'end_time': [datetime.datetime.now(datetime.timezone.utc)],
    'end_ticks': [0],
    'nb_objects': [0],
    'object_offset': [0],
    'payload_size': [0],
    'insert_time': [datetime.datetime.now(datetime.timezone.utc)],
    'audience': ['public'],
})

Supported tables: processes, streams, blocks, payloads

Note: This method is for metadata replication and administrative tasks. Use the telemetry ingestion service HTTP API for normal data ingestion.

`materialize_partitions(view_set_name, begin, end, partition_delta_seconds)`¶

Requires admin

Only callable by an authenticated admin; not audience-filtered — an admin acts across every audience. See Admin SQL Functions.

Create materialized partitions for performance optimization:

# Materialize hourly partitions for the last 24 hours
end = datetime.datetime.now(datetime.timezone.utc)
begin = end - datetime.timedelta(days=1)

client.materialize_partitions(
    'log_entries',
    begin,
    end,
    3600  # 1-hour partitions
)
# Prints progress messages for each materialized partition

`regenerate_partitions(view_set_name, begin, end, partition_delta_seconds)`¶

Requires admin

Only callable by an authenticated admin; not audience-filtered — an admin acts across every audience. See Admin SQL Functions.

Force-regenerate existing partition(s) directly from source data, bypassing the "already up to date" freshness check materialize_partitions() stops at. Useful for rebuilding a partition whose content is unchanged but whose internal row order needs to be refreshed (e.g. an existing merged blocks partition materialized before ordered merges were introduced, or a SqlBatchView such as log_stats whose live partitions predate it declaring a merge sort order):

# Regenerate yesterday's daily blocks partition
end = datetime.datetime.now(datetime.timezone.utc).replace(
    hour=0, minute=0, second=0, microsecond=0
)
begin = end - datetime.timedelta(days=1)

client.regenerate_partitions(
    'blocks',
    begin,
    end,
    86400  # must exactly match the existing daily partition's boundaries
)
# Prints progress messages for the regenerated partition

Warning: (begin, end, partition_delta_seconds) must exactly cover an existing partition's boundaries, or the call fails loudly instead of silently creating a duplicate partition. This is an admin/rollout tool -- run calls serially, never with overlapping ranges in flight concurrently.

For a SqlBatchView, the bucket size is dictated by the existing partition's boundaries, not freely choosable: an already-large, merged partition can only be regenerated as one equally large bucket, and its extract query's applied sort order (for a view declaring with_merge_sort_order) then sorts that whole bucket's aggregated output in a single blocking pass. There is no smaller partition_delta_seconds that avoids this once a partition has already grown large -- retire it (retire_partitions) and re-materialize at a smaller delta instead.

`retire_partitions(view_set_name, view_instance_id, begin, end)`¶

Requires admin

Only callable by an authenticated admin; not audience-filtered — an admin acts across every audience. See Admin SQL Functions.

Remove materialized partitions to free up storage:

# Retire old partitions
client.retire_partitions(
    'log_entries',
    'process-123-456', 
    begin,
    end
)
# Prints status messages as partitions are retired

Warning: This operation cannot be undone. Retired partitions must be re-materialized if needed.

Administrative Functions¶

The micromegas.admin module provides administrative functions for schema evolution and partition lifecycle management. These functions are intended for system administrators and should be used with caution.

`list_incompatible_partitions(client, view_set_name=None)`¶

Lists partitions with schemas incompatible with current view set schemas. Returns one row per partition.

import micromegas
import micromegas.admin

client = micromegas.connect()

incompatible = micromegas.admin.list_incompatible_partitions(client)
print(f"Found {len(incompatible)} incompatible partitions")
print(f"Total size: {incompatible['file_size'].sum() / (1024**3):.2f} GB")

Returns: view_set_name, view_instance_id, begin_insert_time, end_insert_time, incompatible_schema_hash, current_schema_hash, file_path, file_size

`retire_incompatible_partitions(client, view_set_name=None)`¶

Requires admin

Internally calls retire_partition_by_metadata(), which requires an authenticated admin; not audience-filtered. See Admin SQL Functions.

Retires partitions with incompatible schemas using metadata-based retirement (works for empty and non-empty partitions).

# Preview first
preview = micromegas.admin.list_incompatible_partitions(client, 'log_entries')
print(f"Would retire {len(preview)} partitions")

# Retire
result = micromegas.admin.retire_incompatible_partitions(client, 'log_entries')
print(f"Retired {result['partitions_retired'].sum()} partitions")

Returns: view_set_name, view_instance_id, partitions_retired, partitions_failed, storage_freed_bytes, retirement_messages

⚠️ DESTRUCTIVE OPERATION: Irreversible. Always preview first.

`WebClient` — self-service mint¶

micromegas.web_client.WebClient is the HTTP client micromegas-setup-telemetry (and every other analytics-web-srv-facing CLI — -grants, -groups, -screens) is built on, talking to analytics-web-srv's REST API over Bearer auth. Two methods back the setup script:

mint_ingestion_api_key(name, audience=None) — POST {base_path}/api/ingestion-api-keys. Mints a fresh key, generated server-side. On a 409 with {"code": "CLAIM_CONTENDED"} — transient advisory-lock contention with another concurrent claim of the same brand-new audience, not a denial — retries the same request exactly once before raising; any other non-OK status (including a second CLAIM_CONTENDED) raises RuntimeError the same way every other WebClient method does. Returns the mint response dict, including the one-time cleartext key.
my_audiences() — GET {base_path}/api/audience-grants/my-audiences. Caller-scoped, no admin access required. Returns {"is_admin", "audiences", "mint_prefix", "email", "held_pairs"} — the audiences whose mint selector matches the caller today, the caller's own admin flag, a caller-derived namespace prefix a name is minted under via --user-audience (identical composition for an admin and a non-admin caller — --audience never applies it, minting under the name it is given verbatim), the caller's own email, and held_pairs: the "{audience}:{axis}" pairs the caller holds via an identity selector ("*" excluded), used to tell an audience the caller personally holds a grant on from one they can merely see via a wildcard grant.

from micromegas.web_client import WebClient

client = WebClient("https://analytics.example.com", auth_provider=auth_provider)
info = client.my_audiences()
result = client.mint_ingestion_api_key("my-laptop", audience=info["audiences"][0])
print(result["key"])  # cleartext key, returned exactly once

Command-Line Interface¶

The Micromegas Python client includes CLI tools for quick queries and administrative tasks.

micromegas-query - Run SQL Queries¶

Execute arbitrary SQL queries against the analytics service from the command line.

Usage:

pip install micromegas
micromegas-query "SELECT * FROM list_partitions() LIMIT 5"

Arguments: - sql (positional, optional): SQL query to execute (omit when using --file)

Options: - --file: Read SQL from a file path, or use - to read from stdin - --begin: Begin timestamp (RFC 3339 like 2024-01-01T00:00:00Z, or relative like 1h, 30m, 7d). Required unless --all is used - --end: End timestamp (RFC 3339 like 2024-01-01T00:00:00Z, or relative like 1h, 30m, 7d). Default: now - --all: Query the entire time range (mutually exclusive with --begin/--end) - --format: Output format - table (default), csv, or json - --max-colwidth: Maximum column width for table format (default: 50, use 0 for unlimited) - --profile: Named connection profile from ~/.micromegas/config.json (see Named profiles below) - --version: Print the installed micromegas package version and interpreter, then exit

Examples:

# Query with relative time range (last hour)
micromegas-query "SELECT * FROM processes LIMIT 10" --begin 1h

# Query with relative time range (last 24 hours)
micromegas-query "SELECT * FROM log_entries LIMIT 100" --begin 24h

# Query with specific timestamps
micromegas-query "SELECT * FROM measures LIMIT 50" \
    --begin 2024-01-01T00:00:00Z --end 2024-01-02T00:00:00Z

# Read SQL from a file (avoids shell quoting issues with JSONPath)
micromegas-query --file query.sql --begin 1h

# Read SQL from stdin
echo "SELECT 1" | micromegas-query --file - --all

# Output as CSV for piping to other tools
micromegas-query "SELECT * FROM list_partitions()" --all --format csv

# Output as JSON
micromegas-query "SELECT view_set_name, num_rows FROM list_partitions()" --all --format json

Configuration:

The CLI resolves connection settings from four sources, in this order:

Profile selection, if the config file has a profiles map (--profile > MICROMEGAS_PROFILE > default_profile) — see Named profiles below
Environment variables (highest priority for each individual setting, applied on top of the selected profile — or the flat config file if there's no profiles map)
Config file: ~/.micromegas/config.json (the flat file, or the selected profile's entry)
Built-in defaults (e.g., grpc://localhost:50051)

Each setting is resolved independently once a profile (if any) is selected, so you can put stable values in the config file and override individual settings via env vars (e.g., for CI). On a machine with no profiles map, this is the whole story and per-field env vars are all you need to switch environments. Once a profiles map exists, though, profile selection happens first: an env-var-only invocation (e.g. MICROMEGAS_ANALYTICS_URI=... micromegas-query ... in CI) fails with a usage error unless --profile, MICROMEGAS_PROFILE, or default_profile also picks a profile — per-field env vars only override settings within the selected profile, they don't substitute for selecting one.

Environment Variables:

Variable	Description	Default
`MICROMEGAS_ANALYTICS_URI`	FlightSQL server URI	`grpc://localhost:50051`
`MICROMEGAS_OIDC_ISSUER`	OIDC issuer URL (enables OIDC auth when set with `_CLIENT_ID`)	—
`MICROMEGAS_OIDC_CLIENT_ID`	OAuth client ID	—
`MICROMEGAS_OIDC_CLIENT_SECRET`	OAuth client secret (optional; required by some IdPs)	—
`MICROMEGAS_OIDC_AUDIENCE`	API audience/identifier (e.g., for Auth0, Azure)	—
`MICROMEGAS_OIDC_SCOPE`	Custom OAuth scopes	`openid email profile offline_access`
`MICROMEGAS_PROFILE`	Named profile to select from the config file's `profiles` map (see Named profiles below)	—
`MICROMEGAS_CLIENT_AGENT`	Override for the auto-detected `x-client-agent` value (see Client Attribution above)	auto-detected
`MICROMEGAS_CLIENT_ENTRYPOINT`	Override for the auto-detected `x-client-entrypoint` value (see Client Attribution above); the CLI always passes `"cli-query"` regardless of this var	auto-detected

Config file (~/.micromegas/config.json)¶

A small JSON file you can drop in your home directory to avoid setting env vars. Every key is optional, and any env var with the same role overrides it.

{
  "uri": "grpc+tls://analytics.example.com:50051",
  "client_id": "your-app-client-id",
  "issuers": [
    {
      "issuer": "https://accounts.example.com",
      "audience": "your-api-audience"
    }
  ]
}

Key	Maps to	Notes
`uri`	`MICROMEGAS_ANALYTICS_URI`	FlightSQL server URI
`client_id`	`MICROMEGAS_OIDC_CLIENT_ID`	OAuth client ID
`issuers[0].issuer`	`MICROMEGAS_OIDC_ISSUER`	First issuer entry is used
`issuers[0].audience`	`MICROMEGAS_OIDC_AUDIENCE`	First issuer entry is used
`api_key_file`	— (no env var; see Named profiles below)	Path to a static analytics API key file

There are three auth mechanisms, checked in this order: a static API key, when api_key_file resolves; OIDC, when both an issuer and a client ID are resolved (from any source); otherwise the CLI connects without auth. A profile (or flat config) that resolves both api_key_file and a complete OIDC pair is a configuration error — see Named profiles below.

api_key_file is a FlightSQL credential: only micromegas-query, micromegas-views, and connect_with_profile() honor it, because the analytics web API validates OIDC tokens only, not a static key. Every WebClient-based CLI (micromegas-screens, micromegas-grants, -groups, -setup-telemetry) resolves auth through the shared web_auth.resolve_web_auth() helper, which only ever branches on the OIDC fields. Of those, only micromegas-screens reports a profile that resolves api_key_file and nothing else as an error; micromegas-grants, -groups, and -setup-telemetry still silently connect unauthenticated on that path.

Named profiles:

The config file can hold more than one named connection — prod/dev/local, for instance — under an optional profiles map, selected by --profile or MICROMEGAS_PROFILE:

{
  "default_profile": "prod",
  "profiles": {
    "prod": {
      "uri": "grpc://analytics.example.com:50051",
      "client_id": "...",
      "issuers": [{ "issuer": "https://issuer.example.com/v2.0", "audience": "..." }]
    },
    "dev": {
      "uri": "grpc://analytics-dev.example.com:50051",
      "client_id": "...",
      "issuers": [{ "issuer": "https://issuer.example.com/v2.0", "audience": "..." }]
    },
    "ci": {
      "uri": "grpc+tls://analytics.example.com:50051",
      "api_key_file": "~/.micromegas/ci.key"
    },
    "local": { "uri": "grpc://localhost:50051" }
  }
}

Each entry under profiles has exactly the shape of the flat config above (uri, client_id, issuers, api_key_file). The flat shape shown earlier is still fully supported — omit profiles entirely and the whole file is used as a single connection, exactly as before.

A profile (or the flat config) must use exactly one auth mechanism. Naming api_key_file alongside a complete OIDC pair (an issuers/client_id pair, from the profile or from MICROMEGAS_OIDC_ISSUER/MICROMEGAS_OIDC_CLIENT_ID) is a configuration error, not a precedence rule — connect_with_profile() and the CLI both raise before attempting either auth mechanism, naming the real source of each side, e.g.:

profile 'prod' resolves two auth mechanisms: a static API key (profile key 'api_key_file')
and OIDC (issuer from MICROMEGAS_OIDC_ISSUER, client_id from profile key 'client_id').
A profile must use exactly one -- remove 'api_key_file', or unset the OIDC settings.

Selection precedence, once a profiles map is present: --profile (flag) > MICROMEGAS_PROFILE (env var) > default_profile (config key). There is no implicit selection, even with a single profile defined — a profiles map always requires a selected profile, so set default_profile even for a single-profile config. If none of --profile/MICROMEGAS_PROFILE/default_profile resolves to a profile, or the resolved name isn't one of the keys under profiles, the CLI exits with a usage error listing the available profile names rather than guessing.

Don't mix flat keys with a profiles map

Once a profiles map is present, it takes over completely: any top-level uri/client_id/issuers/api_key_file left in the same file are ignored in favor of the selected profile's values. If you're migrating from a flat config to profiles, move those values into a profile entry rather than leaving them at the top level — they'd otherwise become dead config with no error or warning. Similarly, default_profile has no effect at all unless a profiles map is also present.

Adding a profiles map also moves the OIDC token cache from the single ~/.micromegas/tokens.json to a per-profile ~/.micromegas/tokens-<profile>.json, so switching profiles never reuses another profile's cached token. This means turning profiles on forces one fresh login even for an otherwise-unchanged connection — rename your existing tokens.json to the new profile's tokens-<profile>.json path beforehand to avoid it.

micromegas-logout¶

Clears cached OIDC authentication tokens. A bare invocation clears every cached token file — the plain ~/.micromegas/tokens.json plus every tokens-<profile>.json — so it always means "log out of everything":

micromegas-logout

Pass --profile to narrow this to just one profile's cached token, leaving every other token file (including the plain tokens.json) untouched:

micromegas-logout --profile prod

micromegas-logout doesn't read config.json and doesn't look at MICROMEGAS_PROFILE — --profile is its only narrowing mechanism.

A static-key profile (api_key_file) caches no token, so micromegas-logout is a no-op for it — there's no tokens-<profile>.json to clear. Revoking such a key is a server-side operation: DELETE /api/analytics-api-keys/{key_id}.

Pass --version to print the installed package and interpreter version and exit.

micromegas-grants¶

Creates and deletes DB-backed audience grants (audience_grants table) via analytics-web-srv's /api/audience-grants routes — never direct Postgres access. These two write routes are not admin-only: a non-admin caller with a matching hold on the pair can share it too, once MICROMEGAS_SELF_SERVICE_MINT is on (see Authorization → the grant store).

micromegas-grants --url https://analytics.example.com create team-alpha read group:eng
micromegas-grants --url https://analytics.example.com delete team-alpha read group:eng

--url always points at analytics-web-srv's base URL. Two subcommands:

create <audience> <axis> <selector> — creates (or reports the pre-existing) grant row. <axis> is read or mint; <selector> is *, user:<id>, or group:<id>.
delete <audience> <axis> <selector> — deletes one grant row, keyed by its natural triple.

There is no list subcommand. Listing goes through the caller-scoped list_audience_grants() SQL function instead — a non-admin gets their own scoped view (every grant on a pair they hold), an admin gets every row, filterable and orderable like any other table:

micromegas-query --all "SELECT * FROM list_audience_grants()" --profile analytics

Auth follows the same OIDC setup as micromegas-query/-screens (MICROMEGAS_OIDC_* for a non-interactive run, or --profile for an interactive/cached login).

Pass --version to print the installed package and interpreter version and exit.

micromegas-groups¶

Manages local group membership (groups/group_members tables) via analytics-web-srv's /api/groups routes — never direct Postgres access. Every subcommand is admin-only — see Groups for the full model, including the reserved admins group.

micromegas-groups --url https://analytics.example.com list
micromegas-groups --url https://analytics.example.com create eng --description "Engineering"
micromegas-groups --url https://analytics.example.com members admins
micromegas-groups --url https://analytics.example.com add admins user:alice@example.com
micromegas-groups --url https://analytics.example.com remove admins '*'
micromegas-groups --url https://analytics.example.com delete eng

--url always points at analytics-web-srv's base URL. Six subcommands:

list — every group with its member count.
create <name> [--description TEXT] — creates a new, empty group.
delete <name> — deletes a group (fails, 409, on admins or while still referenced by a nested membership or an audience grant).
members <name> — a group's direct members.
add <name> <member> — adds *, user:<id>, or group:<id> to a group.
remove <name> <member> — removes a member from a group.

There is no bootstrap convenience command. Taking over admin access from a wildcard-seeded admins group (the state a fresh install or an unmigrated upgrade seeds — see Groups → Upgrade path) is the two-command sequence:

micromegas-groups --url https://analytics.example.com add admins user:<you>
micromegas-groups --url https://analytics.example.com remove admins '*'

Auth follows the same OIDC setup as micromegas-query/-screens/-grants (MICROMEGAS_OIDC_* for a non-interactive run, or --profile for an interactive/cached login).

Pass --version to print the installed package and interpreter version and exit.

micromegas-setup-telemetry¶

Mints a personal ingestion_api_keys key for the caller and prints the OTLP exporter env vars needed to send that caller's own telemetry to the deployment. Named for what it does from the user's point of view ("set up telemetry"), not the server-side term ("ingestion"). Requires self-service mint to be enabled on the target deployment (MICROMEGAS_SELF_SERVICE_MINT; see Self-service mint) unless the caller is an admin.

# Resolve the audience automatically via GET .../audience-grants/my-audiences, from the
# caller's personally held mint grants only (exactly one match is used silently; more
# than one asks you to pick with --audience).
eval "$(micromegas-setup-telemetry --url https://analytics.example.com --name my-laptop)"

# An audience you already have a grant for -- e.g. the deployment's shared `public`
# audience, mintable by every authenticated caller once the operator has granted it:
micromegas-setup-telemetry --url https://analytics.example.com --name my-laptop \
    --audience public

# A fresh audience of your own: --user-audience composes the name under a prefix
# derived server-side from your email -- the identical command works whether the
# caller is an admin or not.
micromegas-setup-telemetry --url https://analytics.example.com --name ci-runner \
    --user-audience ci-runner --env-file ~/.micromegas/telemetry.env

# From a PowerShell prompt (Windows or pwsh on Linux/macOS) -- --format is never
# inferred from the OS, so pass it explicitly:
micromegas-setup-telemetry --url https://analytics.example.com --name my-laptop `
    --format powershell | Invoke-Expression

# For a container/CI env-file loader (a compose service's env_file:, docker run
# --env-file, python-dotenv, ...):
micromegas-setup-telemetry --url https://analytics.example.com --name ci-runner \
    --format dotenv --env-file .env

--url (required) is analytics-web-srv's base URL. --name (required) names the minted key (e.g. a hostname). --user-audience and --audience are mutually exclusive, and mean two different things:

--user-audience SUFFIX (recommended): mints under f"{mint_prefix}{SUFFIX}", where mint_prefix is derived server-side from the caller's own email and is composed identically for an admin and a non-admin caller (e.g. alice@example.com → alice-, so --user-audience ci-runner resolves to alice-ci-runner). Lazily claims the audience if it's genuinely fresh, writing the caller's own read/mint grant in the same request; if it already exists and the caller holds no grant for it (someone else's namespace), the route's ordinary 403 applies, admin included. Note that mint_prefix is derived from the email's local part only and isn't guaranteed unique, so two callers with the same local part on different domains can share a prefix and genuinely hit this 403 under what looks like "your own" prefix. Requires a caller whose email yields a mint_prefix; errors locally otherwise, with distinct messages for "no email at all" (ask an admin for a grant) vs. "email sanitizes to empty" (use --audience <name> instead).
--audience NAME: mints under NAME verbatim, unconditionally — no client-side check of the caller's mintable set. A genuinely fresh name is lazily claimed by the mint route itself, writing the caller's own read/mint grant in the same request (the printed mint line adds claimed audience <name> when it did); a name someone else already holds is refused with the route's ordinary 403, admin included, which the CLI enriches with the caller's mintable audiences, a --user-audience suggestion, and the exact micromegas-grants commands an admin would run to grant this one (concretely, with the audience and the caller's email already substituted). Use this for an org/team/service audience that isn't namespaced under any one caller.
Omitted entirely: resolved via GET .../audience-grants/my-audiences, filtered to audiences the caller personally holds a mint grant on (the response's held_pairs) — a deployment-wide wildcard grant that puts an audience in every caller's audiences list (e.g. a seeded public mint row) is not enough on its own to be silently auto-selected here. Exactly one personally-held match is used silently; more than one prints the choices and asks for --audience; none prints the audiences the caller can see but does not personally hold (if any), plus a hint to --user-audience a fresh name or ask an admin. Admin and non-admin alike: is_admin grants no audience of its own, so an admin with no held mint audience gets the same "none" error as anyone else, and one with exactly one held mint audience resolves it silently just like a non-admin would.

--otlp-endpoint defaults to f"{MICROMEGAS_TELEMETRY_URL}/ingestion/otlp" when that env var is set (the repo's established ingestion-endpoint convention — see OTLP); it is a required flag only when that env var is unset. --env-file PATH writes the exports to a 0o600 file instead of stdout (parent directory created 0o700 if needed) — useful for sourcing from a shell profile instead of eval-ing directly. On Windows, those POSIX mode bits aren't enforced; the file lands at its parent directory's inherited ACL instead. --profile selects a named connection profile, but (like -grants) only its OIDC fields are honored — an api_key_file-only profile yields no auth here.

--format {posix,powershell,cmd,dotenv} (default posix) picks the rendered syntax and applies to both the stdout and --env-file output paths — it is never inferred from the OS (a Git Bash prompt on Windows wants posix; pwsh runs on Linux/macOS too), so pass it explicitly:

`--format`	rendered shape	consume with
`posix` (default)	`export NAME=value`, `shlex.quote`d	`eval "$(micromegas-setup-telemetry ...)"`
`powershell`	`$env:NAME = 'value'`	`micromegas-setup-telemetry ... \\| Invoke-Expression`
`cmd`	`@set "NAME=value"`	redirect to a `.cmd` file, then `call` it
`dotenv`	`NAME=value`, unquoted	a compose service's `env_file:` or `docker run --env-file` to inject into the container; `python-dotenv` to inject into a process

Each dialect quotes a value the way that syntax represents a literal credential safely: posix via shlex.quote (bare unless the value needs quoting), powershell as a single-quoted literal (a literal ' doubled), cmd as a quoted @set "NAME=value" (the only form that keeps quote characters out of the value), and dotenv unquoted (the value is everything after the first =). Because cmd and dotenv cannot represent every character, --otlp-endpoint is validated against the chosen format before the key is minted, and the command errors out naming the offending character rather than minting a key it then can't render (dotenv also rejects a leading/trailing-whitespace endpoint, which a loader would silently trim). Passing --env-file does not change --format's own default to dotenv — the documented use of --env-file is sourcing it from a shell profile, and dotenv output is not a shell script, so posix stays the default for both output paths.

Auth follows the same OIDC setup as micromegas-query/-screens/-grants.

Pass --version to print the installed package and interpreter version and exit.

Time Utilities¶

`format_datetime(value)`, `parse_datetime(value)`, and `parse_time_delta(user_string)`¶

Utility functions for time handling:

from micromegas.time import format_datetime, parse_datetime, parse_time_delta

# Format datetime for queries
dt = datetime.datetime.now(datetime.timezone.utc)
formatted = format_datetime(dt)
print(formatted)  # "2024-01-01T12:00:00+00:00"

# Parse an RFC 3339 timestamp string (accepts both 'Z' and 'z')
parsed = parse_datetime('2024-01-01T12:00:00Z')
print(parsed)  # 2024-01-01 12:00:00+00:00

# Parse human-readable time deltas
one_hour = parse_time_delta('1h')
thirty_minutes = parse_time_delta('30m') 
seven_days = parse_time_delta('7d')

# Use in calculations
recent_time = datetime.datetime.now(datetime.timezone.utc) - parse_time_delta('2h')

Supported units: m (minutes), h (hours), d (days)

Advanced Features¶

Query Streaming Benefits¶

Use query_stream() for large datasets: it processes data in chunks instead of loading everything into memory, starts returning results before the query completes, and can handle result sets larger than available RAM.

# Example: Process week of data in batches
total_errors = 0
total_rows = 0

for batch in client.query_stream("""
    SELECT level, msg FROM log_entries 
    WHERE time >= NOW() - INTERVAL '7 days'
""", begin, end):
    df = batch.to_pandas()
    errors_in_batch = len(df[df['level'] <= 2])

    total_errors += errors_in_batch
    total_rows += len(df)

    print(f"Batch: {len(df)} rows, {errors_in_batch} errors")

print(f"Total: {total_rows} rows, {total_errors} errors")

FlightSQL Protocol Benefits¶

Micromegas uses Apache Arrow FlightSQL: columnar, binary data transfer with no serialization/deserialization overhead, native compression, and zero-copy reads from network buffers.

Error Handling¶

try:
    df = client.query("SELECT * FROM log_entries;", begin, end)
except Exception as e:
    print(f"Query failed: {e}")

# Check for empty results
if df.empty:
    print("No data found for this time range")
else:
    print(f"Found {len(df)} rows")

Exception types¶

The server classifies each query failure and returns a distinct gRPC status code, which pyarrow's Flight client surfaces as a different Python exception type:

Cause	gRPC code	Exception raised
Bad query (typo'd function/column, syntax error, ...)	`InvalidArgument`	`pyarrow.lib.ArrowInvalid` (a `ValueError` subclass)
Unimplemented feature	`Unimplemented`	`pyarrow.lib.ArrowNotImplementedError` (a `NotImplementedError` subclass)
Query exceeded a resource budget (e.g. memory)	`ResourceExhausted`	`pyarrow.lib.ArrowInvalid` (a `ValueError` subclass, message prefixed `gRPC returned resource exhausted error`)
Query rejected by an admin-managed query deny list rule	`ResourceExhausted`	Same as the row above: `pyarrow.lib.ArrowInvalid`, same `gRPC returned resource exhausted error` message prefix -- the message itself additionally names the rule id and reason and tells you the `remove_query_denial(...)` call that lifts it
`CREATE MATERIALIZED VIEW` of a name that already exists (without `OR REPLACE`)	`AlreadyExists`	`pyarrow.lib.ArrowException`
`DROP MATERIALIZED VIEW` (without `IF EXISTS`) of a name that doesn't exist	`NotFound`	`pyarrow.lib.ArrowKeyError` (an `ArrowException`/`KeyError` subclass)
Genuine server-side bug	`Internal`	`pyarrow._flight.FlightInternalError`

This lets you distinguish "fix my query" from "something broke server-side" without parsing the error message -- except that a resource-budget failure and a deny-list rejection share both the same gRPC code and the same message prefix, so telling those two apart does need one more signal: error_class in the query audit log is "resource" for the former and "denied" for the latter.

try:
    df = client.query(sql, begin, end)
except (ValueError, NotImplementedError) as e:
    # ArrowInvalid (a ValueError) or ArrowNotImplementedError (a NotImplementedError):
    # the query itself needs fixing -- except that a resource-budget failure and a
    # query-deny-list rejection both also raise ArrowInvalid with the same
    # "gRPC returned resource exhausted error" message prefix. Tell all three apart
    # by error_class in the server-side audit log: "user" / "resource" / "denied".
    print(f"Bad query: {e}")
except Exception as e:
    # FlightInternalError / anything else: a genuine server-side problem, not
    # something to fix in the SQL text.
    print(f"Query failed: {e}")

Performance Tips¶

Use Time Ranges¶

Always specify time ranges for better performance:

# ✅ Good - efficient
df = client.query(sql, begin, end)

# ❌ Avoid - can be slow
df = client.query(sql)

Streaming for Large Results¶

Use streaming for queries that might return large datasets:

# If you expect > 100MB of results, use streaming
if expected_result_size_mb > 100:
    for batch in client.query_stream(sql, begin, end):
        process_batch(batch.to_pandas())
else:
    df = client.query(sql, begin, end)
    process_dataframe(df)

Limit Result Size¶

Add LIMIT clauses for exploratory queries:

# Good for exploration
df = client.query("SELECT * FROM log_entries LIMIT 1000;", begin, end)

# Then remove limit for production queries
df = client.query("SELECT * FROM log_entries WHERE level <= 2;", begin, end)

Integration Examples¶

Jupyter Notebooks¶

import matplotlib.pyplot as plt
import seaborn as sns

# Query data
df = client.query("""
    SELECT time, name, value 
    FROM measures 
    WHERE name = 'cpu_usage'
""", begin, end)

# Plot time series
plt.figure(figsize=(12, 6))
plt.plot(df['time'], df['value'])
plt.title('CPU Usage Over Time')
plt.xlabel('Time')
plt.ylabel('CPU Usage %')
plt.show()

Data Pipeline¶

import pandas as pd

def extract_metrics(process_id, hours=24):
    """Extract metrics for a specific process."""
    end = datetime.datetime.now(datetime.timezone.utc)
    begin = end - datetime.timedelta(hours=hours)

    sql = f"""
        SELECT time, name, value, unit
        FROM view_instance('measures', '{process_id}')
        ORDER BY time;
    """

    return client.query(sql, begin, end)

def analyze_performance(df):
    """Analyze performance metrics."""
    metrics = {}
    for name in df['name'].unique():
        data = df[df['name'] == name]['value']
        metrics[name] = {
            'mean': data.mean(),
            'max': data.max(),
            'min': data.min(),
            'std': data.std()
        }
    return metrics

# Use in pipeline
process_metrics = extract_metrics('my-service-123')
performance_summary = analyze_performance(process_metrics)
print(performance_summary)

Next Steps¶

Python API Advanced - Advanced patterns, performance optimization, and specialized tooling
Schema Reference - Understand available views and fields
Functions Reference - Learn about SQL functions
Query Patterns - Common observability query patterns
Performance Guide - Optimize your queries

Python API Reference¶

Installation¶

Basic Usage¶

Connection¶

Connecting via a named profile¶

Simple Queries¶

Client Methods¶

query(sql, begin=None, end=None)¶

query_stream(sql, begin=None, end=None)¶

query_arrow(sql, begin=None, end=None)¶

Dictionary Encoding for Memory Efficiency¶

Using Dictionary-Encoded Properties¶

Compatibility with Standard Functions¶

Working with Results¶

pandas DataFrames¶

pyarrow RecordBatch¶

Connection Configuration¶

FlightSQLClient(uri, headers=None, preserve_dictionary=False, auth_provider=None, client_entrypoint=None)¶

Client Attribution¶

Static Analytics API Keys¶

Schema Discovery¶

prepare_statement(sql)¶

prepared_statement_stream(statement)¶

Process and Stream Discovery¶

find_process(process_id)¶

query_streams(begin, end, limit, process_id=None, tag_filter=None)¶

query_blocks(begin, end, limit, stream_id)¶

query_spans(begin, end, limit, stream_id)¶

Data Management¶

bulk_ingest(table_name, table)¶

materialize_partitions(view_set_name, begin, end, partition_delta_seconds)¶

regenerate_partitions(view_set_name, begin, end, partition_delta_seconds)¶

retire_partitions(view_set_name, view_instance_id, begin, end)¶

Administrative Functions¶

list_incompatible_partitions(client, view_set_name=None)¶

retire_incompatible_partitions(client, view_set_name=None)¶

WebClient — self-service mint¶

Command-Line Interface¶

micromegas-query - Run SQL Queries¶

Config file (~/.micromegas/config.json)¶

micromegas-logout¶

micromegas-grants¶

micromegas-groups¶

micromegas-setup-telemetry¶

Time Utilities¶

format_datetime(value), parse_datetime(value), and parse_time_delta(user_string)¶

Advanced Features¶

Query Streaming Benefits¶

FlightSQL Protocol Benefits¶

Error Handling¶

Exception types¶

Performance Tips¶

Use Time Ranges¶

Streaming for Large Results¶

Limit Result Size¶

Integration Examples¶

Jupyter Notebooks¶

Data Pipeline¶

Next Steps¶

`query(sql, begin=None, end=None)`¶

`query_stream(sql, begin=None, end=None)`¶

`query_arrow(sql, begin=None, end=None)`¶

`FlightSQLClient(uri, headers=None, preserve_dictionary=False, auth_provider=None, client_entrypoint=None)`¶

`prepare_statement(sql)`¶

`prepared_statement_stream(statement)`¶

`find_process(process_id)`¶

`query_streams(begin, end, limit, process_id=None, tag_filter=None)`¶

`query_blocks(begin, end, limit, stream_id)`¶

`query_spans(begin, end, limit, stream_id)`¶

`bulk_ingest(table_name, table)`¶

`materialize_partitions(view_set_name, begin, end, partition_delta_seconds)`¶

`regenerate_partitions(view_set_name, begin, end, partition_delta_seconds)`¶

`retire_partitions(view_set_name, view_instance_id, begin, end)`¶

`list_incompatible_partitions(client, view_set_name=None)`¶

`retire_incompatible_partitions(client, view_set_name=None)`¶

`WebClient` — self-service mint¶

`format_datetime(value)`, `parse_datetime(value)`, and `parse_time_delta(user_string)`¶