Protocol¶
The wire contract: one bidirectional stream per service, a registry of types, and a request/reply envelope.
Protocol summary¶
A service connects with Hub.Connect, sends Register (service name, instance id, handled
command and query types, initial_permits, client_version and optionally credentials),
waits for the RegisterAck and then sends Heartbeat messages. A Request (kind, type, id, payload,
optional timeout_ms) is routed to one live instance of the service that owns the type; the
handler answers with a Reply carrying the same id, which the hub relays to the sender. Each
request id gets exactly one terminal message: a Reply or a hub-produced error; a
scatter-gather, streaming or subscription query gets ReplyParts and one
ReplyEnd instead. The hub forwards
the Request (headers included) unchanged and parents its hub.route span on the request's
trace headers.
Error codes:
- Hub-produced:
NO_HANDLER,TIMEOUT,HANDLER_DISCONNECTED,INVALID_REQUEST(malformed request: missing id, kind or type, or a duplicate in-flightRequest.id, or a request mode this hub does not know),RESOURCE_EXHAUSTED(full queue or session buffer; also a subscription refused bySNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCEor ended by a fullSNODRA_SERVER_SUBSCRIPTION_BUFFER),UNAUTHENTICATEDorPERMISSION_DENIED(a request access control refuses, see Security) andCANCELLED(asReplyEnd.errorof a stream that outlivedSNODRA_SERVER_MAX_STREAM_DURATIONor a subscription that outlivedSNODRA_SERVER_MAX_SUBSCRIPTION_DURATION). - Handler-produced codes in a
Reply{error}, aReplyPart.erroror aReplyEnd.errorare forwarded unchanged.
Metrics outcome labels are ok, no_handler, timeout, handler_disconnected, handler_error,
invalid_request, resource_exhausted, denied and cancelled (a stream or subscription
the sender cancelled, or one that outlived its max duration); snodra_hub_requests_total and
snodra_hub_request_duration_seconds also have a mode label (unary, scatter_gather,
streaming, subscription), so a gather's or stream's latency (to its ReplyEnd) and a
subscription's (to its initial part) stay apart from the unary latency of the same type.
any handler Reply{error} counts as
handler_error, whatever its code; a capacity rejection by the hub (full queue or session
buffer) counts as resource_exhausted.
The generated client stubs are built with build_transport(false); Rust clients create the stub
with HubClient::new(channel).
Handshake¶
- The first
Registerof a stream fixesclient_version,credentials,initial_permits,load_factorandtags; a laterRegisterreplaces the handled types only. ARegisterwithinitial_permits0 is closed withINVALID_ARGUMENT. - Every accepted
Registeris answered with aRegisterAck; the ack of the first one is queued before the echo of the heartbeat that follows theRegister, and a client waits for it to know it is registered (the ack of a laterRegisteris dropped, like a heartbeat echo, when the session's outbound buffer is full). It carrieshub_id,heartbeat_interval_ms,heartbeat_timeout_ms,default_timeout_ms,max_timeout_ms,context(default) andtype_timeouts_ms(the overrides). - A session is evicted after
SNODRA_SERVER_HEARTBEAT_TIMEOUT(10 s) without a heartbeat; clients heartbeat at the announced interval (3 s). credentials(static token or JWT) is checked inpermissiveandenforceand ignored inoff; it is never logged. A client may sendClientMessage.credentialsto replace its session's credential before a JWT expires (Security).- Every message of the protocol is read from every session:
InstanceStatus(kept, see Multiple hubs),Permits, the multi-partReplyPart,ReplyEnd,FlowandCancel(see Query modes) andControlReply(see Processor control). A requestmodethis file does not define isINVALID_REQUEST;CANCELLED(9) ends a stream that outlivedSNODRA_SERVER_MAX_STREAM_DURATION(or a subscription that outlivedSNODRA_SERVER_MAX_SUBSCRIPTION_DURATION).
Flow control¶
Decision record: ADR 0014, hub flow control and priority.
- Every session is flow-controlled with the
Register.initial_permitsit sent (at least 1): the hub forwards it a request only while it has a permit (one per forward) and adds permits on everyPermits{count}it sends, never beyond its cap. The cap starts atinitial_permits; aPermits.grant > 0from the session first raises it tomax(cap, grant)(a grant never lowers it), which is how a multi-hub client moves a departed hub's share to the surviving hubs (ADR 0016). A stream opens its flow state before its session becomes current, so a request for the session always finds it; a request for a session whose flow is gone fails withHANDLER_DISCONNECTEDinstead of going out unmetered. Of two concurrent handshakes of one instance id the newer stream wins. - With a free permit and nobody waiting a request is forwarded at once; otherwise it waits in the
instance's queue, ordered by
Request.priority(higher first) and then arrival, with its own deadline (TIMEOUT). - A queued request never left the hub, so it is not failed with its instance: when the instance
disconnects it is routed again with the same id and its remaining deadline (a keyed command to
the new ring owner, an unkeyed one to another candidate,
NO_HANDLERwhen none is left); when the instance reconnects (a new stream replaces the old one) the new session takes the queue over. Requests already forwarded fail withHANDLER_DISCONNECTED(outcome unknown). - An instance that re-registers without types (a graceful stop) is no routing target any more (not a candidate, not on the ring) and its queue is routed to the other instances at once; it still answers what it was forwarded.
- A full queue (
SNODRA_SERVER_QUEUE_CAPACITY_PER_INSTANCEentries orSNODRA_SERVER_QUEUE_BYTES_PER_INSTANCEbytes), a hub whose queues together holdSNODRA_SERVER_QUEUE_BYTES_TOTALbytes, like a full session buffer, answersRESOURCE_EXHAUSTED. - Permits, load factor and tags are read from the first
Registerof a stream.
Routing¶
- Keyed commands go to the instance the owning service's consistent-hash ring returns for
the key. each instance has
128 x load_factor / 100virtual nodes (0 reads as 100, clamped to 8..1024), points arexxh64("<instance>#<i>"), a colliding point belongs to the smaller instance id, so every hub builds the same ring. Affinity wins: a full queue rejects the request rather than rerouting it. - Unkeyed commands and all queries go to the instance with the best tag-match score (number
of
Register.tagspairs equal to the sender's), then among those to instances that can take a request now (a permit and an empty queue), then to the least(queued + forwarded) / load_factor, round-robin among ties. - Consequences worth knowing: the ring saturates the load factor (clamped to 8..1024 virtual nodes, so 800 and 5000 weigh the same for keys) while the least-loaded choice uses the raw value; at idle (every load 0) unkeyed traffic round-robins whatever the load factors; tags are a strict preference, so a saturated instance with the best match keeps queueing and never spills over to a worse match.
Query modes¶
Decision record: ADR 0017, query modes and shared query types; details in Query modes on this site.
- Shared query types.
Register.shared_query_typeslists query types many services may own (exclusivequery_typeskeep the one-owner rule). A point query on a shared type goes to one live instance of any owner: one pool over every owner, then the routing above, with one round-robin rotation per type. - Scatter-gather. A sender sends a query with
Request.mode = SCATTER_GATHER. Every live instance of every owner that lists the type (on an exclusive type, those instances of its owner; an old instance of a rolling deploy that adds the type is no target) gets an ordinary unaryRequestwith id<id>#<n>, and flow control, queues and priority apply per target. Each answer reaches the sender as aReplyPart(request_id= the sender's id,seqin send order, responder service and instance, payload or error), then oneReplyEnd{parts, failed, timed_out}once every target answered,max_results > 0successful parts arrived, or the deadline passed. Answers after the end are dropped. A gather that ends early (max_results, the deadline, the sender gone) sendsCancel{<id>#<n>, reason}to each still-forwarded target (max results,deadline,sender disconnected); a late answer is dropped. - Errors: no live target is an ordinary
NO_HANDLER; a command isINVALID_REQUEST. A target that leaves, also while its request is still queued, is a failed part (HANDLER_DISCONNECTED). - Accounting: a scatter-gather is one request in flight, on its edge (latency to the
ReplyEnd, an error when no part succeeded) and insnodra_hub_requests_total{mode="scatter_gather"};snodra_hub_scatter_targets{type}records its targets. tests/scatter_gather.rsdrives it over gRPC with raw clients.- Streaming. A sender sends a query with
Request.mode = STREAMING(andinitial_credit,max_results,timeout_ms). The hub picks one instance as for a point query (exclusive or shared owner, tags, load, round-robin), but only among the live instances that list the type. That instance getsRequest{id = <id>#1, mode = STREAMING, initial_credit = <credit>}(the sender's credit, 64 when it sent 0;max_resultsis the hub's to enforce, the handler gets 0), through flow control, queues and priority like any request. The handler answers withClientMessage.part = ReplyPart{<id>#1, payload, seq}andClientMessage.end = ReplyEnd{<id>#1, error?}; the hub relays each to the sender in order (request_idrewritten to the sender's id,responder_service/responder_instanceset, the handler'sseqkept), also when the sender's buffer is full (the end never overtakes a part). - Credits. The hub keeps the remaining credit (initial credit plus every sender
Flow{<id>, credits}, relayed to the handler asFlow{<id>#1, credits}); each part takes one. A part beyond the credit ends the stream withReplyEnd{error = INVALID_REQUEST "part without credit"}andCancel{<id>#1, "part without credit"}to the handler. A part carryingerrorends it withReplyEnd{error}(no part is relayed, no cancel: the handler ended). Aftermax_results > 0parts:ReplyEnd{parts}andCancel{<id>#1, "max results"}. - Ending it.
ClientMessage.cancel = Cancel{<id>, reason}from the sender:Cancel{<id>#1, reason}to the handler, nothing more to the sender. No part withintimeout_ms(the usual rules; from the forward, reset by every part, paused while the sender's credit is exhausted and restarted by theFlowthat raises it; a queue wait is bounded by it too):ReplyEnd{error = TIMEOUT}andCancel{<id>#1, "idle timeout"}. Open forSNODRA_SERVER_MAX_STREAM_DURATION:ReplyEnd{error = CANCELLED "max stream duration (10m) exceeded"}and aCancelwith that reason. Handler gone:ReplyEnd{error = HANDLER_DISCONNECTED}; sender gone:Cancel{<id>#1, "sender disconnected"}. ACancelgoes only to a handler that was written the request; a stream that ends between the pop of its queued target and the write getsCancel{<id>#1, "closed before forward"}right after theRequest(so the handler never serves a stream the hub forgot). The hub's shutdown closes every session (as for unary requests). a unaryReplyto a streaming target is relayed as one part (seq1, as a streaming handler numbers its first part) and the end (Reply{error}asReplyEnd{error}). - Streaming errors: a request mode the hub does not know
INVALID_REQUEST; a command by request kindINVALID_REQUEST; no owner, or no live instance that lists the type,NO_HANDLER; a command by registered kindINVALID_REQUEST; a duplicate id (<id>or<id>#1)INVALID_REQUEST. The gates apply in that order. - Streaming accounting: one request in flight (the sender's
inFlightAsSender, the handler'sinFlightAsHandler, one handler permit from the forward to the end, released on every ending path); one count on its edge with the latency to the end (an error when the end carries one);snodra_hub_requests_total{mode="streaming"}and the duration histogram;snodra_hub_streams_open{service,instance}(the handler's) andsnodra_hub_stream_parts_total{service,type}. tests/streaming.rsdrives it over gRPC with raw clients. The client APIs are in the libraries' READMEs: snodra-messaging, Streaming queries (Kotlin) and snodra-messaging-rs, Streaming queries (Rust).- Subscription. A sender sends a query with
Request.mode = SUBSCRIPTION(andinitial_credit,max_results,timeout_ms). The targets are every live instance of every owner (exclusive or shared) that lists the type, as they are at the open: an instance registering later is not added (updates are delivered at most once; the hub never resubscribes). Each target getsRequest{id = <id>#n, mode = SUBSCRIPTION}(n= 1, 2, ... in target order: service, then instance id;initial_creditandmax_results0) written directly: a subscription is passive, so it takes no permit, waits in no queue and counts in nobody'sinFlightAsHandler; a target whose buffer is full at the open is left out (when none could be written the subscription ends with the last failure's code:RESOURCE_EXHAUSTEDfor a full buffer,HANDLER_DISCONNECTEDfor a session that is gone). Every<id>#nis written before the<id>#0initial request, so on the instance that gets both the library sees the update target first; a close cancels only the targets already written. One of the targets, chosen as for a point query (tags, load, round-robin), also gets the initial requestRequest{id = <id>#0, mode = UNARY}with the sender'stimeout_msand priority, through flow control, queues and priority like any unary request, holding a permit. ItsReply{payload}reaches the sender asReplyPart{<id>, seq 1, responder_*};timeout_ms(usual rules) bounds only this request (queue wait plus handler). - Updates. A target pushes
ClientMessage.part = ReplyPart{<id>#n, payload, seq}per update andClientMessage.end = ReplyEnd{<id>#n, error?}when its emitter completes. The hub relays the updates to the sender in arrival order with its ownseq(2, 3, ... after the initial part; the handlers'seqis dropped),request_idrewritten to the sender's id andresponder_service/responder_instancenaming the target, in order also when the sender's buffer is full. Updates that arrive before the initial result wait behind it (the initial part is alwaysseq1). A target's cleanReplyEndremoves it; the last target's clean end ends the subscription withReplyEnd{parts}once the buffered updates are out. Parts and ends for an unknown id, from an instance that is not that target, or a unaryReplyto an update target are ignored (debug log). - Credits and buffer. The sender's credit is
initial_credit(64 when 0) plus everyFlow{<id>, credits}; the initial part takes one, every update one. Updates beyond the credit (or before the initial part) are buffered per subscription, up toSNODRA_SERVER_SUBSCRIPTION_BUFFER(256); one more ends it withReplyEnd{error = RESOURCE_EXHAUSTED "subscriber buffer (256) exceeded"}and aCancelto every target. AFlowdrains the buffer in order. Nothing is relayed to the targets: updates are pushed, a slow sender is the hub's problem, not the handlers'. - Ending it.
max_results > 0counts updates, not the initial part: after that many relayed updatesReplyEnd{parts}andCancel{<id>#n, "max results"}to every target (what is still buffered is dropped). AReplyEnd{error}or a part carryingerrorfrom any target ends it withReplyEnd{error}and aCancelto the rest (the failed target ended by itself). The initial request failing (Reply{error},TIMEOUT, its handler gone, a capacity rejection) ends it with that error and cancels every target; no part is sent.ClientMessage.cancel = Cancel{<id>, reason}from the sender:Cancel{<id>#n, reason}to every target, and to<id>#0while the initial request is still forwarded; nothing more to the sender. The sender gone: the same with"sender disconnected". A target gone: its target ends silently; the last one gone:ReplyEnd{error = HANDLER_DISCONNECTED}. Open forSNODRA_SERVER_MAX_SUBSCRIPTION_DURATION(off by default):ReplyEnd{error = CANCELLED "max subscription duration (<d>) exceeded"}and aCancelwith that reason. There is no idle timeout: a subscription without updates is a quiet one. The hub's shutdown closes every session (as for unary requests). - Subscription errors: a command by request kind
INVALID_REQUEST; no owner, or no live instance that lists the type,NO_HANDLER; a command by registered kindINVALID_REQUEST; a duplicate id (<id>,<id>#0, or a pending<id>#n)INVALID_REQUEST; a target already servingSNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCE(1024) subscriptionsRESOURCE_EXHAUSTED. The gates apply in that order, all before anything is sent. Builder call: an update target's id is no pending id (it is matched through its subscription), so a request whose id equals one is routed normally. - Subscription accounting: one request in flight (
inFlight, the sender'sinFlightAsSender; the initial request in its handler'sinFlightAsHandler) until the initial part is delivered, then the subscription counts nowhere but insnodra_hub_subscriptions_open{service,instance}(targets per handler instance); one count on its edge at the initial reply (an error when the initial failed, or when the subscription ended before it other than by the sender's cancel),snodra_hub_requests_total{mode="subscription"}and the duration histogram (the time to the initial part), outcomecancelledfor a cancel or the max duration before the initial part;snodra_hub_subscription_updates_total{service,type}per update relayed. A subscription that ends after its initial part counts nothing more. tests/subscription.rsdrives it over gRPC with raw clients. The client libraries implement the sender and handler sides; their READMEs: snodra-messaging, Subscription queries (Kotlin) and snodra-messaging-rs, Subscription queries (Rust).
Request timeouts¶
With override the SNODRA_SERVER_TIMEOUT_OVERRIDES entry for the request's type:
Request.timeout_ms |
Deadline |
|---|---|
> 0 |
min(timeout_ms, override or SNODRA_SERVER_MAX_TIMEOUT_MS) |
0 |
override or SNODRA_SERVER_DEFAULT_TIMEOUT_MS |
An override thus also moves the clamp for its type, up (long-running commands) or down. Clients
send timeout_ms = 0 unless their caller chose a timeout, so an override applies to them without
any client change.