Sagas¶
Sagas¶
Sagas, deadlines and the integration toolkit of ADR 0021 (S2): the model of the Kotlin starter (S1), on the same MongoDB
collections and document shapes (sagas, saga_dispatch, saga_dispatch_leases, deadlines, saga_failures,
external_calls, markers in processed_events), so a Kotlin and a Rust instance read each other's documents. The
programming model, the delivery guarantees and the command-failure taxonomy are in
Sagas and deadlines (its "Rust"
subsection maps every Kotlin element to this crate); the decisions in
ADR 0021, S2 rulings 47 to 67. The sandbox's banner saga
(examples/service-c/src/banner.rs) is a complete example.
A saga is a state type and a definition closure. Handlers are synchronous closures that change the state and record
what they want done on the SagaContext; the runtime writes the state, the recorded commands and events and the deadlines
in one transaction, and the relay sends them after the commit, in order, with the idempotency key <sagaId>:<seq>.
use snodra_messaging::SagaDefinition;
use snodra_messaging::external::{ExternalSource, IdempotencyKey, TranslatedEvent};
use snodra_messaging::external::kafka::Message; // record.payload(): rdkafka's trait, re-exported
use serde::de::IgnoredAny;
#[derive(Default, Serialize, Deserialize)] // stored as its serde_json text; must round-trip from Default
struct Banner { job_id: Option<String>, polls: u32 }
fn banner_saga(d: SagaDefinition<Banner>) -> SagaDefinition<Banner> {
d.start_on::<TextShouted>("jobId", |e| Some(job_id_of(&e.text)), |s, e, ctx| {
s.job_id = Some(job_id_of(&e.text));
ctx.send(&PrintBanner { job_id: job_id_of(&e.text), text: e.text.clone() }, None)?; // after commit
ctx.schedule_after::<()>(Duration::from_secs(30), "print-timeout", None)?;
Ok(())
})
.on::<BannerPrinted>("jobId", |e| Some(e.job_id.clone()), |_, _, _| Ok(()))
.ending() // ends after this handler
.on_deadline::<IgnoredAny>("print-timeout", |s, _, ctx| {
ctx.publish(&BannerFailed { job_id: s.job_id.clone().unwrap_or_default(), reason: "TIMEOUT".into() }, None)
})
.ending()
.on_command_failed(|s, e, ctx| { // routed by saga id, no association needed
ctx.publish(&BannerFailed { job_id: s.job_id.clone().unwrap_or_default(), reason: format!("{:?}", e.reason) }, None)?;
ctx.end();
Ok(())
})
}
let app = SnodraMessaging::builder(Config::from_env()?).await?
.saga("examples.Banner", banner_saga)
// The adapter: a command handler made idempotent over external_calls (one call per key).
.idempotent_command::<PrintBanner, _, _>(IdempotencyKey::default(), false, move |c, ctx| {
let printer = printer.clone();
async move { printer.submit(&c, &ctx.headers["idempotency-key"]).await }
})
// A partner's Kafka topic: records become local events (never republished to snodra.events).
.external_source("printer-banner", ExternalSource::new("printer.banner.events"), |record| {
// Malformed never heals: InvalidMessage skips the record at once (a bare `?` is Serialization, retried first).
let n: Notification = serde_json::from_slice(record.payload().unwrap_or_default())
.map_err(|e| MessagingError::InvalidMessage(format!("not a printer notification: {e}")))?;
Ok(Some(TranslatedEvent::new(&BannerPrinted { job_id: n.job_id }, Some(n.delivery_id))?))
})
.start()
.await?;
// A webhook: verify the raw body, then hand the event to the ingress (applied once per dedupe key).
if !snodra_messaging::external::webhook::verify(&secret, None, &headers, &body) { return 401; }
app.ingress().accept("printer-banner", &n.delivery_id, &BannerPrinted { job_id: n.job_id }, &HashMap::new()).await?;
The definition. start_on::<E>(key, value, handler) may start an instance (at most one live instance per start
key = value); start_new_on always starts one; on runs on the live instances associated with key = value; the
*_header forms (start_on_header, start_new_on_header, on_header) read the value from a message header; .ending()
marks the handler just added (event or deadline) as ending the saga; on_deadline::<P>(name, handler) gets the payload as
Option<P> (take IgnoredAny when there is none or it is ignored: () reads only null); on_command_failed(handler) gets
the SagaCommandFailed of a command the instance sent. Handlers of one event type run in registration order. start()
refuses (InvalidMessage naming <saga>.<handler>) a saga without a start handler, two sagas of one name, an empty name,
.ending() with nothing before it, two handlers of one deadline, and a state whose S::default() does not round-trip
through its JSON text.
The context. saga_id(), message() (the MessageContext; for a deadline its id, deadline:<name> and the correlation
id of the message that scheduled it), associate_with, remove_association_with (removing the start association frees the
start key, also from the start handler), end(), send(&command, routing_key), publish(&event, key), schedule_after(duration, name, payload),
schedule_at(time, name, payload) (both return the deadline id), cancel(id), cancel_all(name); at most 999 messages per
handling of an instance. A published event must serialize to a JSON object BSON can hold (not a unit struct, no integer
above i64::MAX): anything else fails the handler with InvalidMessage naming the event. A handler gets no Sender or EventBus, and must not capture one: it would send inside the
saga's transaction, for a state that may roll back, with no order, retry or idempotency key.
Handlers block. A saga handler runs on the consumer's task and holds a tokio worker and the partition while it runs:
keep it to recording. A panic is a handler error: two attempts, then the event is parked in saga_failures
(app.saga_failures(): list(type), retry(id), retry_all(type), discard(id)).
Adapters. idempotent_command::<C>(key, external_idempotent, handler) (and idempotent_event::<E>, whose body runs inside
the ordinary handlers' transaction: prefer commands for slow calls) records each call in external_calls.
IdempotencyKey::default() is the idempotency-key header the relay sets; IdempotencyKey::From(|c, ctx| …) builds one (a
poll puts its attempt in it). Return MessagingError::RetryableExternal for a failure a later attempt may cure (nothing
reached the partner, a 5xx, a 429) and MessagingError::UnknownOutcome when the call may have happened (the connection broke
after the request went out); never retry inside the handler. With external_idempotent = true the partner de-duplicates by
the key, so a call that crashed mid-way is made again after the lease instead of reported as UNKNOWN_OUTCOME. Every handler
of a command a saga sends must be idempotent by idempotency-key: a crash between sending and marking it sent sends it again.
Configuration (Config::from_env, Kotlin's defaults; retentions at least 1 s, intervals at least 1 ms, checked also in a
service without sagas):
| Variable | Default | Meaning |
|---|---|---|
SNODRA_SAGAS_ENABLED |
true | false with sagas registered: a warning, and no runtime, relay or scheduler |
SNODRA_SAGAS_MAX_ATTEMPTS |
5 | attempts of one (event, saga type) on conflicts before it is parked (200 ms doubling to 5 s); a handler error gets two |
SNODRA_SAGAS_ENDED_RETENTION_MS |
30 d | TTL of ended sagas |
SNODRA_SAGAS_DISPATCH_RETENTION_MS |
7 d | TTL of dispatched saga_dispatch entries |
SNODRA_SAGAS_RELAY_POLL_INTERVAL_MS |
500 | how often the relay looks for recorded entries |
SNODRA_SAGAS_RELAY_LEASE_MS |
30000 | one saga's dispatch lease, renewed while a send runs |
SNODRA_SAGAS_RELAY_MAX_ATTEMPTS |
10 | attempts of a command on transient failures before EXHAUSTED |
SNODRA_SAGAS_RELAY_INITIAL_BACKOFF_MS, _MAX_BACKOFF_MS |
1000, 5 min | the relay's backoff (doubling) |
SNODRA_DEADLINES_POLL_INTERVAL_MS |
1000 | how often due deadlines are looked for |
SNODRA_DEADLINES_LEASE_MS |
30000 | how long a claimed deadline stays reserved (at least once) |
SNODRA_DEADLINES_RETENTION_MS |
7 d | TTL of fired and cancelled deadlines |
SNODRA_EXTERNAL_CALLS_LEASE_MS |
5 min | how long an IN_PROGRESS call may take before its outcome is unknown |
SNODRA_EXTERNAL_CALLS_RETENTION_MS |
30 d | TTL of external_calls |
Partner sources (ExternalSource { topic, bootstrap_servers, group_id, properties }; a key the library fixes in
properties, an empty topic or two sources of one name fail start()) and the webhook secret are set in code.
builder.clock(Clock) replaces the clock of the runtime, the relay, the scheduler, the adapters and the ingress
(ManualClock in tests). The runtime, the relay, the scheduler and the gauge sampler run only with a registered saga, the
external_calls indexes only with an idempotent handler, a partner consumer per source; app.ingress() always exists.
Saga commands to a type the service handles itself go to its own handler without the hub. Shutdown stops the relay, the
scheduler and the sampler first (at most 20 s for the three together), then the rest.
Deploys and in-flight adapter calls. At shutdown the relay abandons a send still running after 15 s; the saga's
dispatch lease then lets another instance (or this one, restarted) send the command again with the same key. For a
command delivered to a local idempotent adapter, abandoning the send drops the adapter's future mid-call, so its
external_calls record stays IN_PROGRESS: the re-send is answered "in progress" (retryable, the relay backs off) until
the record's lease ends (SNODRA_EXTERNAL_CALLS_LEASE_MS, 5 min), and then, without external_idempotent, the outcome is
unknown and the saga gets SagaCommandFailed with UNKNOWN_OUTCOME (with the default settings; a relay whose attempts
run out first reports EXHAUSTED). With external_idempotent the call is run again after the lease. Keep an adapter's
I/O timeouts under 15 s, so its calls finish (or fail cleanly) within a deploy's shutdown.
Statistics. app.saga_stats() has the Kotlin meters as counters and gauges with the same names and labels
(snodra.sagas.started{type}, snodra.sagas.ended{type}, snodra.saga.events{type,outcome},
snodra.deadlines.fired{name}, snodra.deadlines.due{name}, snodra.external.calls{handler,outcome},
snodra.saga.command.failures{type,reason}, snodra.sagas.active{type}, snodra.saga.dispatch.pending): typed reads
(started, ended, events, deadlines_fired, deadlines_due, external_calls, command_failures, active,
dispatch_pending) and snapshot() for an exporter. The two database-counted gauges are sampled every 15 s. While a saga
handles a message, a tracing span saga carries saga_type, saga_id and correlation_id.
Testing a saga¶
Enable the feature testing in your dev-dependencies and drive the definition with events and time, in memory, on the
crate's own runtime (no Kafka, MongoDB or hub):
[dev-dependencies]
snodra-messaging-rs = { path = "../snodra-messaging-rs", features = ["testing"] }
use snodra_messaging::saga::testing::SagaTestFixture;
#[test]
fn a_silent_printer_fails_the_banner_after_30_seconds() {
SagaTestFixture::new("examples.Banner", banner_saga) // the closure given to builder.saga
.given_current_time(t0)
.given_events(&[&shouted("SILENT-1")]) // what these cause is not expected below
.when_time_elapses(Duration::from_secs(30)) // fires every due deadline at its own time
.expect_published_events(&[&BannerFailed { job_id: "banner-silent-1".into(), reason: "TIMEOUT".into() }])
.expect_saga_ended()
.expect_no_scheduled_deadlines();
}
Steps: given_events, given_current_time (default 2026-01-01T00:00:00Z), when_event, when_event_with_headers,
when_time_elapses, when_command_failed(&command, reason, message) (the relay's SagaCommandFailed for a command the saga
sent). Expectations: expect_active_sagas, expect_association, expect_dispatched_commands, expect_published_events
(several types in one slice: &[&A { .. }, &B { .. }], compared by wire type and JSON), expect_scheduled_deadline,
expect_no_scheduled_deadlines, expect_saga_ended; store() reads the documents. A step that fails (a handler error or
panic, a state that does not survive its JSON text) and an expectation that does not hold panic, naming the saga; a
failing step leaves nothing behind. The methods are synchronous: call them from a #[test] (or from an async test without
holding the fixture across an .await).
Differences from Kotlin¶
The behaviour is ADR 0021's rulings 1 to 46; where Rust differs, the S2 ruling says why (rulings 47 to 67):
- Sagas are closures registered on the builder (
builder.saga(name, define)), not scanned@Sagaclasses; the name is required. - Handlers of one event type run in registration order (Kotlin: method-name order).
- Handlers get one
&mut SagaContext, no bean parameters (closures capture);send,publishandschedule_*returnResult; capturing aSenderis forbidden by documentation, not by a startup check. - A handler error is an
Error a caught panic; a synchronous handler also holds a tokio worker. - Transactions are an explicit
Txfrom aTxProviderport; the in-memory ports exist behind the featuretesting. - Idempotency keys are
IdempotencyKey::Header/IdempotencyKey::From(fn)instead of SpEL; a store failure before an adapter's body ran isRetryableExternal(Kotlin: a plain failure); a dropped adapter future leaves its recordIN_PROGRESSuntil the lease, a panicking one isFAILED; an unboundExternalCallsHandlefails at call time (Kotlin checks at startup). - A partner translator is a closure returning
Result<Option<TranslatedEvent>, _>;InvalidMessageskips the record at once. - Configuration is
SNODRA_*environment variables (milliseconds); partner sources and the webhook secret are set in code. - Metrics are
SagaStatscounters and gauges (no Micrometer); the logging keys are atracingspan's fields. - A deadline's
payloadTypeisstd::any::type_nameof its payload (Kotlin: the@MessageTypename or the FQCN). SagaTestFixtureis in the crate behind a feature (Kotlin:snodra-messaging-test) and panics instead of throwing.