Interactive Notebooks for Observability

<img src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9Ii0xMjAgLTUwIDI0MCAxNjAiPgogIDxkZWZzPgogICAgPGxpbmVhckdyYWRpZW50IGlkPSJyaW5nMSIgeDE9IjAlIiB5MT0iMCUiIHgyPSIxMDAlIiB5Mj0iMTAwJSI+CiAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0eWxlPSJzdG9wLWNvbG9yOiNiZjM2MGMiLz4KICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdHlsZT0ic3RvcC1jb2xvcjojOGQzYTE0Ii8+CiAgICA8L2xpbmVhckdyYWRpZW50PgogICAgPGxpbmVhckdyYWRpZW50IGlkPSJyaW5nMiIgeDE9IjAlIiB5MT0iMCUiIHgyPSIxMDAlIiB5Mj0iMTAwJSI+CiAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0eWxlPSJzdG9wLWNvbG9yOiMxNTY1YzAiLz4KICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdHlsZT0ic3RvcC1jb2xvcjojMGQ0N2ExIi8+CiAgICA8L2xpbmVhckdyYWRpZW50PgogICAgPGxpbmVhckdyYWRpZW50IGlkPSJyaW5nMyIgeDE9IjAlIiB5MT0iMCUiIHgyPSIxMDAlIiB5Mj0iMTAwJSI+CiAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0eWxlPSJzdG9wLWNvbG9yOiNmZmIzMDAiLz4KICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdHlsZT0ic3RvcC1jb2xvcjojZTZhMDAwIi8+CiAgICA8L2xpbmVhckdyYWRpZW50PgogICAgPGZpbHRlciBpZD0iZ2xvdyIgeD0iLTUwJSIgeT0iLTUwJSIgd2lkdGg9IjIwMCUiIGhlaWdodD0iMjAwJSI+CiAgICAgIDxmZUdhdXNzaWFuQmx1ciBzdGREZXZpYXRpb249IjIiIHJlc3VsdD0iY29sb3JlZEJsdXIiLz4KICAgICAgPGZlTWVyZ2U+CiAgICAgICAgPGZlTWVyZ2VOb2RlIGluPSJjb2xvcmVkQmx1ciIvPgogICAgICAgIDxmZU1lcmdlTm9kZSBpbj0iU291cmNlR3JhcGhpYyIvPgogICAgICA8L2ZlTWVyZ2U+CiAgICA8L2ZpbHRlcj4KICA8L2RlZnM+CgogIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAsIDApIj4KICAgIDxlbGxpcHNlIGN4PSIwIiBjeT0iMCIgcng9IjEwMCIgcnk9IjM4IiBmaWxsPSJub25lIiBzdHJva2U9InVybCgjcmluZzEpIiBzdHJva2Utd2lkdGg9IjIuNSIgdHJhbnNmb3JtPSJyb3RhdGUoLTIwKSIgb3BhY2l0eT0iMC45Ii8+CiAgICA8ZWxsaXBzZSBjeD0iMCIgY3k9IjAiIHJ4PSI3OSIgcnk9IjMwIiBmaWxsPSJub25lIiBzdHJva2U9InVybCgjcmluZzIpIiBzdHJva2Utd2lkdGg9IjIuNSIgdHJhbnNmb3JtPSJyb3RhdGUoMjUpIiBvcGFjaXR5PSIwLjkiLz4KICAgIDxlbGxpcHNlIGN4PSIwIiBjeT0iMCIgcng9IjU4IiByeT0iMjIiIGZpbGw9Im5vbmUiIHN0cm9rZT0idXJsKCNyaW5nMykiIHN0cm9rZS13aWR0aD0iMi41IiB0cmFuc2Zvcm09InJvdGF0ZSgtOCkiIG9wYWNpdHk9IjAuOSIvPgogICAgPGcgZmlsdGVyPSJ1cmwoI2dsb3cpIj4KICAgICAgPHBvbHlnb24gcG9pbnRzPSIwLC0xMiAyLC00IDksLTQgMywxIDYsOSAwLDQgLTYsOSAtMywxIC05LC00IC0yLC00IiBmaWxsPSIjZmZmZmZmIi8+CiAgICA8L2c+CiAgPC9nPgoKICA8dGV4dCB4PSIwIiB5PSI5MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZmlsbD0iI2ZmZmZmZiIgZm9udC1mYW1pbHk9InN5c3RlbS11aSwgLWFwcGxlLXN5c3RlbSwgJ1NlZ29lIFVJJywgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIyOCIgZm9udC13ZWlnaHQ9IjMwMCIgbGV0dGVyLXNwYWNpbmc9IjQiPm1pY3JvbWVnYXM8L3RleHQ+Cjwvc3ZnPgo=" alt="micromegas" class="plain" style="height: 340px; margin: 0;">

## Interactive Notebooks for Observability

<p style="font-size: 0.6em;">Marc-Antoine Desroches · <a href="mailto:madesroches@gmail.com">madesroches@gmail.com</a><br><a href="https://micromegas.info/">micromegas.info</a></p>

---

## Agenda

1. **Existing tools** — strengths and gaps
2. **The solution** — composable notebooks
3. **Screenshots** — what we built
4. **Live demo**

---

## Grafana Dashboards

<ul>
<li class="fragment"><strong>Self-service</strong>: teams build and maintain their own dashboards</li>
<li class="fragment">Great for <strong>monitoring</strong> — pre-built panels, alerts, familiar interface</li>
<li class="fragment">But <strong>queries can be difficult to write</strong> — big queries, no intermediate results, limited data types, hard to debug</li>
<li class="fragment">Dashboards are <strong>static</strong>: when something looks wrong, you need to explore elsewhere</li>
</ul>

---

## Python + Jupyter

<ul>
<li class="fragment">The Micromegas Python API makes Jupyter a <strong>powerful investigation tool</strong></li>
<li class="fragment">Full SQL access, DataFrames, arbitrary analysis — maximum flexibility</li>
<li class="fragment">But <strong>matplotlib is not quick and intuitive</strong> for throwaway queries</li>
<li class="fragment">10 lines of boilerplate to get a chart you'll look at once and throw away</li>
<li class="fragment">Great for deep analysis, overkill for "just show me what happened"</li>
</ul>

---

## Single-Purpose Tooling

<ul>
<li class="fragment">The analytics web app started here: <strong>dedicated screens</strong> for process list, metrics, logs, performance analytics</li>
<li class="fragment">Each screen does <strong>one thing well</strong> — purpose-built UI, no query writing needed</li>
<li class="fragment"><strong>Not self-service</strong>: every new workflow requires a developer to build a new screen</li>
</ul>

---

## The Gap

Monitoring (Grafana) ← **???** → Deep analysis (Jupyter)

Single-purpose screens sit in between but don't compose

## The Solution

What if we took those **same components** — tables, charts, logs, timelines — and made them **composable**?

<ul>
<li class="fragment">The <strong>iterative, exploratory</strong> nature of Jupyter</li>
<li class="fragment">But with <strong>specialized visualizations</strong> that make results instantly readable</li>
<li class="fragment">No boilerplate — write SQL, see a chart</li>
</ul>

---

## Notebooks

An ordered list of cells that execute top-to-bottom.

<ul>
<li class="fragment"><strong>11 cell types</strong>: tables, charts, logs, timelines, markdown, variables...</li>
<li class="fragment"><strong>Local query engine</strong>: cell results become queryable tables (DataFusion in WASM)</li>
<li class="fragment"><strong>Variables</strong>: parameterize queries, share state via URL</li>
<li class="fragment"><strong>Auto-run</strong>: change a variable, everything updates</li>
</ul>

## The Secret Sauce

**Composability through an in-browser query engine**

<div style="font-size: 0.5em; margin-top: 0.5em; display: grid; grid-template-columns: auto 1fr; column-gap: 1em; row-gap: 0; align-items: center;">
  <div style="border: 1px solid var(--color-border); border-radius: 6px; padding: 0.3em 0.8em; background: var(--color-bg-2); font-family: var(--font-mono); text-align: left;">
    <div>Cell A: <span style="color: var(--color-wheat);">SQL query</span></div>
    <div><code>SELECT ... FROM server</code></div>
  </div>
  <div>→ result registered as table <strong>"A"</strong></div>
  <div style="text-align: center; font-size: 1.3em; line-height: 1.2;">↓</div>
  <div></div>
  <div style="border: 1px solid var(--color-border); border-radius: 6px; padding: 0.3em 0.8em; background: var(--color-bg-2); font-family: var(--font-mono); text-align: left;">
    <div>Cell B: <span style="color: var(--color-horizon);">local query</span></div>
    <div><code>SELECT * FROM A WHERE ...</code></div>
  </div>
  <div>→ runs entirely <strong>in-browser</strong></div>
  <div style="text-align: center; font-size: 1.3em; line-height: 1.2;">↓</div>
  <div></div>
  <div style="border: 1px solid var(--color-border); border-radius: 6px; padding: 0.3em 0.8em; background: var(--color-bg-2); font-family: var(--font-mono); text-align: left;">
    <div>Cell C: <span style="color: var(--color-rust);">visualization</span></div>
    <div><code>Chart / Table / Timeline</code></div>
  </div>
  <div>→ <strong>no</strong> extra server round-trips</div>
</div>

<p style="margin-top: 0.3em; font-size: 0.8em;">This is what makes notebooks more than just a list of queries.</p>

---

## Cell Types at a Glance

Tables, charts, logs, markdown — all in one page

---

## The Table Cell

SQL in, rows out

3.7M rows · 175 MB · <code>SELECT * FROM measures</code>

---

## Charts

Drag to select a time range. Everything re-executes.

---

## Logs

Compact and color-coded automatically.

---

## Variables

Dropdowns, text inputs, computed JS expressions

---

## Horizontal Groups

Side-by-side layout via drag-and-drop

---

## Swimlane

Thread activity over time

---

## Property Timeline

Property changes over time

---

**https://micromegas.info/**