Monitoring¶
Snodra exports OpenTelemetry metrics and traces over OTLP, and serves a live dashboard. This page is the short operator's map; the exhaustive metric tables are in Observability.
Three sources¶
| Source | Prefix | What it tells you |
|---|---|---|
| Server | snodra_hub_* |
Requests, latency, queues, permits, sessions, processor status, auth failures, event browser |
| Client libraries | snodra_* |
Outbox, consumed events, processor lag and state, dead letters, sagas, hub-client latency |
| Event browser | snodra_event_browser_* |
Subscription lag and refresh outcomes (when it runs as its own service) |
Every series carries job (the service name) and instance (the instance id). The server uses
service.name=snodra-server; a library uses its configured service name.
Turn it on¶
docker run --rm -p 7000:7000 -p 7001:7001 \
-e SNODRA_SERVER_OTLP_ENDPOINT=http://otel-lgtm:4317 \
snodra/snodra-server:0.5.0
An empty endpoint disables export. The Rust library records through the OpenTelemetry API only: the
application installs the meter provider before the builder runs. The Kotlin starter uses Micrometer,
so the service's normal Actuator/OTLP setup applies. Set OTEL_METRIC_EXPORT_INTERVAL (milliseconds)
to change the 60-second default.
The four signals to alert on¶
| Signal | Metric | Why |
|---|---|---|
| Capacity | snodra_hub_rejected_total{reason} |
Requests are being refused because a queue or buffer is full |
| Latency | snodra_hub_queue_wait_seconds |
Time spent waiting for a permit, before the handler even starts |
| Backlog | snodra_outbox_pending, snodra_outbox_oldest_age_seconds |
Events are committed but not reaching the event store |
| Consumption | snodra_processor_lag, snodra_dead_letters |
A processor is falling behind or parking records |
Two more are worth a page: snodra_hub_auth_violations_total during an access-control rollout, and
snodra_processor_state for a processor stuck in FAILING or PAUSED.
Gauges need staleness handling¶
An instance that stops exports nothing more, and Prometheus keeps the last gauge value. Read library
gauges as last_over_time(<gauge>[$instance_seen]), as the shipped dashboards do, so a dead pod
drops out instead of looking connected forever.
Dashboard and history¶
The server's dashboard (http://localhost:7001/ui/) shows the live topology, per-edge traffic, queue
depth, permits, processors and dead letters. Set SNODRA_SERVER_GRAFANA_URL to link it to Grafana,
which keeps the history. SNODRA_SERVER_PEERS merges several servers' dashboards and flags where
their registries disagree.
Health and logs¶
The gRPC health service answers on the server's own port; the distroless image probes it with
/app/snodra-server --health-check. Logs are JSON on stdout, filtered by SNODRA_SERVER_LOG. See
Observability for the log events, audit lines and the full metric
reference, and Containers for probes.