Skip to content

Protocol

The wire contract: one bidirectional stream per service, a registry of types, and a request/reply envelope.

Protocol summary

A service connects with Hub.Connect, sends Register (service name, instance id, handled command and query types, initial_permits, client_version and optionally credentials), waits for the RegisterAck and then sends Heartbeat messages. A Request (kind, type, id, payload, optional timeout_ms) is routed to one live instance of the service that owns the type; the handler answers with a Reply carrying the same id, which the hub relays to the sender. Each request id gets exactly one terminal message: a Reply or a hub-produced error; a scatter-gather, streaming or subscription query gets ReplyParts and one ReplyEnd instead. The hub forwards the Request (headers included) unchanged and parents its hub.route span on the request's trace headers.

Error codes:

  • Hub-produced: NO_HANDLER, TIMEOUT, HANDLER_DISCONNECTED, INVALID_REQUEST (malformed request: missing id, kind or type, or a duplicate in-flight Request.id, or a request mode this hub does not know), RESOURCE_EXHAUSTED (full queue or session buffer; also a subscription refused by SNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCE or ended by a full SNODRA_SERVER_SUBSCRIPTION_BUFFER), UNAUTHENTICATED or PERMISSION_DENIED (a request access control refuses, see Security) and CANCELLED (as ReplyEnd.error of a stream that outlived SNODRA_SERVER_MAX_STREAM_DURATION or a subscription that outlived SNODRA_SERVER_MAX_SUBSCRIPTION_DURATION).
  • Handler-produced codes in a Reply{error}, a ReplyPart.error or a ReplyEnd.error are forwarded unchanged.

Metrics outcome labels are ok, no_handler, timeout, handler_disconnected, handler_error, invalid_request, resource_exhausted, denied and cancelled (a stream or subscription the sender cancelled, or one that outlived its max duration); snodra_hub_requests_total and snodra_hub_request_duration_seconds also have a mode label (unary, scatter_gather, streaming, subscription), so a gather's or stream's latency (to its ReplyEnd) and a subscription's (to its initial part) stay apart from the unary latency of the same type. any handler Reply{error} counts as handler_error, whatever its code; a capacity rejection by the hub (full queue or session buffer) counts as resource_exhausted.

The generated client stubs are built with build_transport(false); Rust clients create the stub with HubClient::new(channel).

Handshake

  • The first Register of a stream fixes client_version, credentials, initial_permits, load_factor and tags; a later Register replaces the handled types only. A Register with initial_permits 0 is closed with INVALID_ARGUMENT.
  • Every accepted Register is answered with a RegisterAck; the ack of the first one is queued before the echo of the heartbeat that follows the Register, and a client waits for it to know it is registered (the ack of a later Register is dropped, like a heartbeat echo, when the session's outbound buffer is full). It carries hub_id, heartbeat_interval_ms, heartbeat_timeout_ms, default_timeout_ms, max_timeout_ms, context (default) and type_timeouts_ms (the overrides).
  • A session is evicted after SNODRA_SERVER_HEARTBEAT_TIMEOUT (10 s) without a heartbeat; clients heartbeat at the announced interval (3 s).
  • credentials (static token or JWT) is checked in permissive and enforce and ignored in off; it is never logged. A client may send ClientMessage.credentials to replace its session's credential before a JWT expires (Security).
  • Every message of the protocol is read from every session: InstanceStatus (kept, see Multiple hubs), Permits, the multi-part ReplyPart, ReplyEnd, Flow and Cancel (see Query modes) and ControlReply (see Processor control). A request mode this file does not define is INVALID_REQUEST; CANCELLED (9) ends a stream that outlived SNODRA_SERVER_MAX_STREAM_DURATION (or a subscription that outlived SNODRA_SERVER_MAX_SUBSCRIPTION_DURATION).

Flow control

Decision record: ADR 0014, hub flow control and priority.

  • Every session is flow-controlled with the Register.initial_permits it sent (at least 1): the hub forwards it a request only while it has a permit (one per forward) and adds permits on every Permits{count} it sends, never beyond its cap. The cap starts at initial_permits; a Permits.grant > 0 from the session first raises it to max(cap, grant) (a grant never lowers it), which is how a multi-hub client moves a departed hub's share to the surviving hubs (ADR 0016). A stream opens its flow state before its session becomes current, so a request for the session always finds it; a request for a session whose flow is gone fails with HANDLER_DISCONNECTED instead of going out unmetered. Of two concurrent handshakes of one instance id the newer stream wins.
  • With a free permit and nobody waiting a request is forwarded at once; otherwise it waits in the instance's queue, ordered by Request.priority (higher first) and then arrival, with its own deadline (TIMEOUT).
  • A queued request never left the hub, so it is not failed with its instance: when the instance disconnects it is routed again with the same id and its remaining deadline (a keyed command to the new ring owner, an unkeyed one to another candidate, NO_HANDLER when none is left); when the instance reconnects (a new stream replaces the old one) the new session takes the queue over. Requests already forwarded fail with HANDLER_DISCONNECTED (outcome unknown).
  • An instance that re-registers without types (a graceful stop) is no routing target any more (not a candidate, not on the ring) and its queue is routed to the other instances at once; it still answers what it was forwarded.
  • A full queue (SNODRA_SERVER_QUEUE_CAPACITY_PER_INSTANCE entries or SNODRA_SERVER_QUEUE_BYTES_PER_INSTANCE bytes), a hub whose queues together hold SNODRA_SERVER_QUEUE_BYTES_TOTAL bytes, like a full session buffer, answers RESOURCE_EXHAUSTED.
  • Permits, load factor and tags are read from the first Register of a stream.

Routing

  • Keyed commands go to the instance the owning service's consistent-hash ring returns for the key. each instance has 128 x load_factor / 100 virtual nodes (0 reads as 100, clamped to 8..1024), points are xxh64("<instance>#<i>"), a colliding point belongs to the smaller instance id, so every hub builds the same ring. Affinity wins: a full queue rejects the request rather than rerouting it.
  • Unkeyed commands and all queries go to the instance with the best tag-match score (number of Register.tags pairs equal to the sender's), then among those to instances that can take a request now (a permit and an empty queue), then to the least (queued + forwarded) / load_factor, round-robin among ties.
  • Consequences worth knowing: the ring saturates the load factor (clamped to 8..1024 virtual nodes, so 800 and 5000 weigh the same for keys) while the least-loaded choice uses the raw value; at idle (every load 0) unkeyed traffic round-robins whatever the load factors; tags are a strict preference, so a saturated instance with the best match keeps queueing and never spills over to a worse match.

Query modes

Decision record: ADR 0017, query modes and shared query types; details in Query modes on this site.

  • Shared query types. Register.shared_query_types lists query types many services may own (exclusive query_types keep the one-owner rule). A point query on a shared type goes to one live instance of any owner: one pool over every owner, then the routing above, with one round-robin rotation per type.
  • Scatter-gather. A sender sends a query with Request.mode = SCATTER_GATHER. Every live instance of every owner that lists the type (on an exclusive type, those instances of its owner; an old instance of a rolling deploy that adds the type is no target) gets an ordinary unary Request with id <id>#<n>, and flow control, queues and priority apply per target. Each answer reaches the sender as a ReplyPart (request_id = the sender's id, seq in send order, responder service and instance, payload or error), then one ReplyEnd{parts, failed, timed_out} once every target answered, max_results > 0 successful parts arrived, or the deadline passed. Answers after the end are dropped. A gather that ends early (max_results, the deadline, the sender gone) sends Cancel{<id>#<n>, reason} to each still-forwarded target (max results, deadline, sender disconnected); a late answer is dropped.
  • Errors: no live target is an ordinary NO_HANDLER; a command is INVALID_REQUEST. A target that leaves, also while its request is still queued, is a failed part (HANDLER_DISCONNECTED).
  • Accounting: a scatter-gather is one request in flight, on its edge (latency to the ReplyEnd, an error when no part succeeded) and in snodra_hub_requests_total{mode="scatter_gather"}; snodra_hub_scatter_targets{type} records its targets.
  • tests/scatter_gather.rs drives it over gRPC with raw clients.
  • Streaming. A sender sends a query with Request.mode = STREAMING (and initial_credit, max_results, timeout_ms). The hub picks one instance as for a point query (exclusive or shared owner, tags, load, round-robin), but only among the live instances that list the type. That instance gets Request{id = <id>#1, mode = STREAMING, initial_credit = <credit>} (the sender's credit, 64 when it sent 0; max_results is the hub's to enforce, the handler gets 0), through flow control, queues and priority like any request. The handler answers with ClientMessage.part = ReplyPart{<id>#1, payload, seq} and ClientMessage.end = ReplyEnd{<id>#1, error?}; the hub relays each to the sender in order (request_id rewritten to the sender's id, responder_service / responder_instance set, the handler's seq kept), also when the sender's buffer is full (the end never overtakes a part).
  • Credits. The hub keeps the remaining credit (initial credit plus every sender Flow{<id>, credits}, relayed to the handler as Flow{<id>#1, credits}); each part takes one. A part beyond the credit ends the stream with ReplyEnd{error = INVALID_REQUEST "part without credit"} and Cancel{<id>#1, "part without credit"} to the handler. A part carrying error ends it with ReplyEnd{error} (no part is relayed, no cancel: the handler ended). After max_results > 0 parts: ReplyEnd{parts} and Cancel{<id>#1, "max results"}.
  • Ending it. ClientMessage.cancel = Cancel{<id>, reason} from the sender: Cancel{<id>#1, reason} to the handler, nothing more to the sender. No part within timeout_ms (the usual rules; from the forward, reset by every part, paused while the sender's credit is exhausted and restarted by the Flow that raises it; a queue wait is bounded by it too): ReplyEnd{error = TIMEOUT} and Cancel{<id>#1, "idle timeout"}. Open for SNODRA_SERVER_MAX_STREAM_DURATION: ReplyEnd{error = CANCELLED "max stream duration (10m) exceeded"} and a Cancel with that reason. Handler gone: ReplyEnd{error = HANDLER_DISCONNECTED}; sender gone: Cancel{<id>#1, "sender disconnected"}. A Cancel goes only to a handler that was written the request; a stream that ends between the pop of its queued target and the write gets Cancel{<id>#1, "closed before forward"} right after the Request (so the handler never serves a stream the hub forgot). The hub's shutdown closes every session (as for unary requests). a unary Reply to a streaming target is relayed as one part (seq 1, as a streaming handler numbers its first part) and the end (Reply{error} as ReplyEnd{error}).
  • Streaming errors: a request mode the hub does not know INVALID_REQUEST; a command by request kind INVALID_REQUEST; no owner, or no live instance that lists the type, NO_HANDLER; a command by registered kind INVALID_REQUEST; a duplicate id (<id> or <id>#1) INVALID_REQUEST. The gates apply in that order.
  • Streaming accounting: one request in flight (the sender's inFlightAsSender, the handler's inFlightAsHandler, one handler permit from the forward to the end, released on every ending path); one count on its edge with the latency to the end (an error when the end carries one); snodra_hub_requests_total{mode="streaming"} and the duration histogram; snodra_hub_streams_open{service,instance} (the handler's) and snodra_hub_stream_parts_total{service,type}.
  • tests/streaming.rs drives it over gRPC with raw clients. The client APIs are in the libraries' READMEs: snodra-messaging, Streaming queries (Kotlin) and snodra-messaging-rs, Streaming queries (Rust).
  • Subscription. A sender sends a query with Request.mode = SUBSCRIPTION (and initial_credit, max_results, timeout_ms). The targets are every live instance of every owner (exclusive or shared) that lists the type, as they are at the open: an instance registering later is not added (updates are delivered at most once; the hub never resubscribes). Each target gets Request{id = <id>#n, mode = SUBSCRIPTION} (n = 1, 2, ... in target order: service, then instance id; initial_credit and max_results 0) written directly: a subscription is passive, so it takes no permit, waits in no queue and counts in nobody's inFlightAsHandler; a target whose buffer is full at the open is left out (when none could be written the subscription ends with the last failure's code: RESOURCE_EXHAUSTED for a full buffer, HANDLER_DISCONNECTED for a session that is gone). Every <id>#n is written before the <id>#0 initial request, so on the instance that gets both the library sees the update target first; a close cancels only the targets already written. One of the targets, chosen as for a point query (tags, load, round-robin), also gets the initial request Request{id = <id>#0, mode = UNARY} with the sender's timeout_ms and priority, through flow control, queues and priority like any unary request, holding a permit. Its Reply{payload} reaches the sender as ReplyPart{<id>, seq 1, responder_*}; timeout_ms (usual rules) bounds only this request (queue wait plus handler).
  • Updates. A target pushes ClientMessage.part = ReplyPart{<id>#n, payload, seq} per update and ClientMessage.end = ReplyEnd{<id>#n, error?} when its emitter completes. The hub relays the updates to the sender in arrival order with its own seq (2, 3, ... after the initial part; the handlers' seq is dropped), request_id rewritten to the sender's id and responder_service / responder_instance naming the target, in order also when the sender's buffer is full. Updates that arrive before the initial result wait behind it (the initial part is always seq 1). A target's clean ReplyEnd removes it; the last target's clean end ends the subscription with ReplyEnd{parts} once the buffered updates are out. Parts and ends for an unknown id, from an instance that is not that target, or a unary Reply to an update target are ignored (debug log).
  • Credits and buffer. The sender's credit is initial_credit (64 when 0) plus every Flow{<id>, credits}; the initial part takes one, every update one. Updates beyond the credit (or before the initial part) are buffered per subscription, up to SNODRA_SERVER_SUBSCRIPTION_BUFFER (256); one more ends it with ReplyEnd{error = RESOURCE_EXHAUSTED "subscriber buffer (256) exceeded"} and a Cancel to every target. A Flow drains the buffer in order. Nothing is relayed to the targets: updates are pushed, a slow sender is the hub's problem, not the handlers'.
  • Ending it. max_results > 0 counts updates, not the initial part: after that many relayed updates ReplyEnd{parts} and Cancel{<id>#n, "max results"} to every target (what is still buffered is dropped). A ReplyEnd{error} or a part carrying error from any target ends it with ReplyEnd{error} and a Cancel to the rest (the failed target ended by itself). The initial request failing (Reply{error}, TIMEOUT, its handler gone, a capacity rejection) ends it with that error and cancels every target; no part is sent. ClientMessage.cancel = Cancel{<id>, reason} from the sender: Cancel{<id>#n, reason} to every target, and to <id>#0 while the initial request is still forwarded; nothing more to the sender. The sender gone: the same with "sender disconnected". A target gone: its target ends silently; the last one gone: ReplyEnd{error = HANDLER_DISCONNECTED}. Open for SNODRA_SERVER_MAX_SUBSCRIPTION_DURATION (off by default): ReplyEnd{error = CANCELLED "max subscription duration (<d>) exceeded"} and a Cancel with that reason. There is no idle timeout: a subscription without updates is a quiet one. The hub's shutdown closes every session (as for unary requests).
  • Subscription errors: a command by request kind INVALID_REQUEST; no owner, or no live instance that lists the type, NO_HANDLER; a command by registered kind INVALID_REQUEST; a duplicate id (<id>, <id>#0, or a pending <id>#n) INVALID_REQUEST; a target already serving SNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCE (1024) subscriptions RESOURCE_EXHAUSTED. The gates apply in that order, all before anything is sent. Builder call: an update target's id is no pending id (it is matched through its subscription), so a request whose id equals one is routed normally.
  • Subscription accounting: one request in flight (inFlight, the sender's inFlightAsSender; the initial request in its handler's inFlightAsHandler) until the initial part is delivered, then the subscription counts nowhere but in snodra_hub_subscriptions_open{service,instance} (targets per handler instance); one count on its edge at the initial reply (an error when the initial failed, or when the subscription ended before it other than by the sender's cancel), snodra_hub_requests_total{mode="subscription"} and the duration histogram (the time to the initial part), outcome cancelled for a cancel or the max duration before the initial part; snodra_hub_subscription_updates_total{service,type} per update relayed. A subscription that ends after its initial part counts nothing more.
  • tests/subscription.rs drives it over gRPC with raw clients. The client libraries implement the sender and handler sides; their READMEs: snodra-messaging, Subscription queries (Kotlin) and snodra-messaging-rs, Subscription queries (Rust).

Request timeouts

With override the SNODRA_SERVER_TIMEOUT_OVERRIDES entry for the request's type:

Request.timeout_ms Deadline
> 0 min(timeout_ms, override or SNODRA_SERVER_MAX_TIMEOUT_MS)
0 override or SNODRA_SERVER_DEFAULT_TIMEOUT_MS

An override thus also moves the clamp for its type, up (long-running commands) or down. Clients send timeout_ms = 0 unless their caller chose a timeout, so an override applies to them without any client change.