Configuration¶
Configuration¶
snodra:
messaging:
service-name: workouts # Kafka consumer group, event source, hub registration
type-prefix: workouts # wire type prefix for classes without @MessageType
hub:
address: static://snodra-server:7000
kafka:
bootstrap-servers: broker:19092
spring:
mongodb:
uri: mongodb://mongo:27017/workouts # Spring Boot 4 key; spring.data.mongodb.uri is no longer bound
security:
oauth2:
resourceserver:
jwt:
issuer-uri: https://auth.example/realms/snodra
The issuer-uri is only needed when the service also serves HTTP or wants forwarded tokens verified (see
Security); Kafka-only tools and services behind the hub start without any security configuration.
All keys (snodra.messaging.*; durations accept 200ms, 5s, 7d; relaxed binding, e.g. SNODRA_MESSAGING_HUB_ADDRESS):
| Key | Default | Meaning |
|---|---|---|
service-name |
required | Kafka consumer group, event source, hub Register.service_name. |
type-prefix |
required | Prefix of the wire type name for message classes without @MessageType. |
hub.enabled |
true |
Connect to the hub and expose the hub-backed CommandBus/QueryBus (addition beyond the spec table). |
hub.address |
static://snodra-server:7000 |
The hub or hubs: one gRPC target (host:port, optional static:// prefix), a comma-separated list of them (hub-1:7000,hub-2:7000, one connection per hub), or one dns:///name:port whose addresses are re-resolved every 30 s. See Multiple hubs. A malformed value fails the startup. |
hub.status-interval |
5s |
Period of the InstanceStatus report (connected hub ids, handlers in flight) sent to every live hub; also sent whenever the set of live hubs changes. Must be positive. |
hub.request-timeout |
unset | Per-request timeout sent as timeout_ms. Unset: the hub applies its default (5 s) or its per-type override (SNODRA_SERVER_TIMEOUT_OVERRIDES). Also bounds the wait for a hub stream (5 s when unset). 0s counts as unset. |
hub.reconnect-backoff.min |
500ms |
Initial reconnect delay when the stream drops; doubles per failed attempt. Must be positive and not above max, or the startup fails naming the property. |
hub.reconnect-backoff.max |
5s |
Maximum reconnect delay. Every wait is jittered by ±20 % and capped at this value (at the cap: 80-100 % of it); the delay resets to min once an attempt reached the hub's handshake. Must be positive (a zero would hot-loop handshakes against a refusing hub): checked at binding, a bad value fails the startup naming the property. |
hub.max-subscriptions |
1024 |
Subscription queries this instance serves at once, over every hub (at least 1). The next one is refused with RESOURCE_EXHAUSTED ("subscription limit (N) reached"), which the subscriber sees as a HandlerBusyException. A subscription holds no handler permit, so max-concurrent-handlers does not bound it; the default equals the hub's own SNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCE. |
hub.stream-credit-timeout |
10m |
The longest a streaming handler of this instance waits for credit from the collector before it ends the stream with TIMEOUT ("no credit within ...") and frees its handler slot and hub permit (a hub that forgot the stream would otherwise leave them held forever). Must be positive; the default is the hub's default maximum stream duration (SNODRA_SERVER_MAX_STREAM_DURATION). |
hub.token |
none | Static token sent in Register.credentials.token of every hub stream (hub access control, Hub security). Optional; an empty or blank value is the same as none. Never logged, redacted in toString and in the actuator's configprops and env. Environment: SNODRA_MESSAGING_HUB_TOKEN, or SNODRA_SERVER_TOKEN when this key is unset. |
hub.token-file |
none | File holding the token (a container secret such as /run/secrets/snodra-server-token; surrounding whitespace ignored). Read at startup (an unreadable file fails it, naming the file) and again on every reconnect, so a rotated secret is picked up without a restart. After a first non-empty read, an unreadable or empty file (a non-atomic rotation caught half way) keeps the last token read and logs one WARN; at startup an empty file means no token. Wins over hub.token when both are set. Environment: SNODRA_MESSAGING_HUB_TOKEN_FILE, or SNODRA_SERVER_TOKEN_FILE when this key is unset. |
hub.tls.enabled |
false |
Talk TLS to the hub, for every hub.address form (a list of targets and dns:///). Off: plain text. The hub must serve TLS on that port (SNODRA_SERVER_TLS_CERT / SNODRA_SERVER_TLS_KEY). |
hub.tls.ca-file |
JVM system roots | PEM file with the CA certificate(s) that signed the hub's certificate (a self-signed hub certificate is its own CA). Read on every connection attempt; a file that cannot be used fails the startup. Set without hub.tls.enabled it is ignored with a warning. |
hub.tls.server-name |
the address's host | Name to verify the hub's certificate against, for when the address is an IP or an alias the certificate does not list. |
hub.oauth2.token-uri |
none | Setting it turns OAuth2 client-credentials authentication on (P5b, JWT): the token endpoint, for example https://auth.example/realms/snodra/protocol/openid-connect/token. http is allowed and warns once unless the host is loopback or a single-label name. Cannot be combined with hub.token or hub.token-file (the startup fails naming both). Environment: SNODRA_MESSAGING_HUB_OAUTH2_TOKEN_URI, or SNODRA_SERVER_OAUTH2_TOKEN_URI when this key is unset. |
hub.oauth2.client-id |
snodra.messaging.service-name |
The confidential client's id; it is the token's azp, which the hub requires to equal the service name. Environment: SNODRA_SERVER_OAUTH2_CLIENT_ID. |
hub.oauth2.client-secret |
none | The client secret (sent as HTTP Basic, client_secret_basic). Never logged, redacted in toString and in the actuator's configprops and env. Prefer the file. Environment: SNODRA_MESSAGING_HUB_OAUTH2_CLIENT_SECRET, or SNODRA_SERVER_OAUTH2_CLIENT_SECRET. |
hub.oauth2.client-secret-file |
none | File holding the client secret: read at startup (an unreadable or empty file fails it, naming the file) and again on every token fetch, so a rotated secret applies without a restart. Wins over client-secret. Environment: SNODRA_MESSAGING_HUB_OAUTH2_CLIENT_SECRET_FILE, or SNODRA_SERVER_OAUTH2_CLIENT_SECRET_FILE. A token URI with neither secret fails the startup. |
hub.oauth2.scope |
none | scope of the token request, sent as written (space-separated). Environment: SNODRA_SERVER_OAUTH2_SCOPE. |
hub.oauth2.ca-file |
platform roots | PEM CA certificate(s) trusted on top of the platform roots for an https token URI (not the hub's CA, which is hub.tls.ca-file). A file that cannot be used fails the startup. Environment: SNODRA_SERVER_OAUTH2_CA. |
hub.max-concurrent-handlers |
256 |
Incoming hub requests handled concurrently by this instance, over all hubs (at least 1, else the startup fails); further requests wait for a free slot. Also announced as initial_permits to a hub with flow control (split over N hubs: ceil(max / N) each), which then queues the excess on its side. |
hub.load-factor |
100 |
Relative capacity of this instance (Register.load_factor); weights its share of the hub's routing (keyed ring and load score). A negative value fails the startup; 0 is allowed (the hub reads it as 100). |
hub.tags.<name> |
none | Labels announced in Register.tags, for example hub.tags.zone=eu-1; unkeyed requests prefer instances whose tags match the sender's. |
hub.priorities.<type> |
none (0) | Default Request.priority per message type, for example hub.priorities.orders.PlaceOrder=5; higher is served first when the hub queues requests for a busy handler. SendOptions.priority wins. |
hub.retry.enabled |
false |
Retry failed sends that are safe to retry (see "Retry" below). SendOptions.retry overrides per call. |
hub.retry.max-attempts |
3 |
Total attempts including the first (1 = no retry). The hub.retry.* values are validated at startup even while enabled is false: max-attempts < 1, multiplier < 1 or a negative backoff fails it. |
hub.retry.initial-backoff |
100ms |
Wait before the second attempt. Every wait is jittered by ±20 % and capped at max-backoff (at the cap: 80-100 % of it). |
hub.retry.multiplier |
2 |
Growth of the wait per further attempt. |
hub.retry.max-backoff |
2s |
Upper bound of a single wait. |
kafka.bootstrap-servers |
broker:19092 |
Kafka bootstrap servers. |
kafka.topic |
snodra.events |
Events topic. |
kafka.consumer.enabled |
true |
Set false in tools that only publish. |
kafka.group-id |
service-name |
The service's consumer group: every event processor, default included, uses <this>.<name>. See Event processors. |
kafka.consumer.max-attempts |
5 |
Handler invocations per record (first try included, 1 s apart) before the record goes to the dead letters; shared by every processor. |
kafka.dead-letter.mode |
collection |
What a processor does with a record it cannot handle: collection (park it in dead_letters, sequence its key, DeadLetters admin API), topic (produce it to <topic>.dlt) or log (log and skip: nothing is parked). Case-insensitive; anything else fails the startup. See Dead letters. |
kafka.dead-letter.retention |
30d |
A letter is deleted this long after it was parked (TTL index on expireAt), which also unblocks its key. At least 1s. |
outbox.enabled |
true |
Run the outbox publisher that relays outbox documents to Kafka. |
outbox.poll-interval |
200ms |
Outbox poll period. |
outbox.batch-size |
100 |
Documents per poll. |
outbox.retention |
7d |
Sent outbox documents older than this are deleted (TTL index). |
outbox.max-attempts |
10 |
Send attempts per outbox document before it is marked failed (failedAt) and skipped. |
idempotency.retention |
30d |
processed_events documents older than this are deleted (TTL index). |
security.forward-jwt |
true |
Put the current request's bearer token in the hub request's authorization header. |
sagas.enabled |
true |
Switch for the saga runtime, the dispatch relay and the deadline scheduler. They exist only when the application has a @Saga class (found by scanning its packages) anyway. See Sagas. |
sagas.max-attempts |
5 |
Attempts per saga event (first try included) for conflicts (a version check, a start-key duplicate, a Mongo write conflict or transient transaction error) before the event is parked in saga_failures; also the bound for firing a deadline. Backoff 200 ms doubling to 5 s (not configurable). A handler exception is not cured by waiting: it gets two attempts in all, at most 50 ms apart, and the event is then parked. |
sagas.ended-retention |
30d |
Ended sagas are deleted this long after they ended (TTL index on sagas.endedAt). Must be positive. |
sagas.dispatch-retention |
7d |
Dispatched saga_dispatch entries are deleted this long after they were sent (TTL index on dispatchedAt). Must be positive. |
sagas.relay.poll-interval |
500ms |
How often the dispatch relay looks for entries that saga handlers recorded. Must be positive. |
sagas.relay.lease |
30s |
How long a relay instance holds one saga's dispatch order before another instance may take over (a crashed instance's lease expires); renewed before each entry and, every third of the lease, while a send runs. Needs synchronised clocks (NTP) across instances. Must be positive. |
sagas.relay.max-attempts |
10 |
Attempts per command for transient failures (first try included, at least 1) before it becomes SagaCommandFailed(EXHAUSTED). |
sagas.relay.initial-backoff |
1s |
Wait before the second attempt of a command; doubles per attempt up to max-backoff. |
sagas.relay.max-backoff |
5m |
Upper bound of one wait between attempts of a command. |
deadlines.poll-interval |
1s |
How often the deadline scheduler looks for due deadlines. Must be positive. |
deadlines.lease |
30s |
How long a claimed deadline stays reserved for the instance that claimed it; an expired claim lets it fire again (at least once). Must be positive. |
deadlines.retention |
7d |
Fired and cancelled deadlines are deleted this long after (TTL index on deadlines.finishedAt). Must be positive. |
external.calls.enabled |
unset | Whether the idempotency interceptor and the external_calls indexes exist. Unset: they exist when a class in the application's packages has an @IdempotentHandler method. true when the adapter lives outside those packages, false to switch it off. |
external.calls.lease |
5m |
How long an IN_PROGRESS external call may take before its outcome counts as unknown (a crash in the middle of the call). Must be positive. |
external.calls.retention |
30d |
external_calls documents older than this are deleted (TTL index on updatedAt). Must be positive. |
external.kafka.sources.<name>.topic |
required | A partner's topic, consumed into local Snodra events (never republished to snodra.events). Each source needs an ExternalRecordTranslator bean named <name>; a source without one fails the startup naming it. |
external.kafka.sources.<name>.bootstrap-servers |
kafka.bootstrap-servers |
The partner's cluster. |
external.kafka.sources.<name>.group-id |
<service-name>.external.<name> |
The source's consumer group. |
external.kafka.sources.<name>.properties.* |
none | Extra consumer properties (for example security.protocol); bootstrap servers, group id, deserializers and auto-commit are fixed by the library: an entry for one of those keys fails the startup, naming it. auto.offset.reset is earliest unless set here. Record retries use kafka.consumer.max-attempts. |
Security¶
(The hub's own access control, the service token and TLS are in Hub security below.)
Cross-service commands and queries are service-to-service calls on a private network, so they are trusted by default: a request with no token is anonymous and is still handled. A handler that needs an authenticated user opts in:
@CommandHandler
@RequiresAuthentication(expression = "hasRole('snodra-user')")
suspend fun handle(cmd: LogWorkout, ctx: MessageContext): LogWorkoutResult = ...
@QueryHandler
@RequiresAuthentication // isAuthenticated()
fun handle(q: GetWorkout, ctx: MessageContext): GetWorkoutResult = ...
@CommandHandler
fun handle(cmd: RecomputeStats) { ... } // no annotation: any caller, including event handlers and jobs
The expression is Spring Security SpEL evaluated by the receiving library before the handler runs:
isAuthenticated(), hasRole('x'), hasAnyRole(...), hasAuthority('ROLE_x'), principal (a MessagePrincipal
with userId, roles, verified), plus #message (the decoded command or query) and #ctx (the MessageContext),
e.g. "hasRole('snodra-user') and #message.ownerId == #ctx.userId". Realm roles from the token's realm_access.roles
become ROLE_<role> authorities. A syntax error fails startup naming the handler; @RequiresAuthentication is not
allowed on @EventHandler (events carry no identity). Annotated handlers are listed at startup at INFO.
Outcomes, on the hub path and the local short-circuit alike:
| Caller | Result at the sender |
|---|---|
| no identity (no token, or an event handler / job) | UnauthenticatedException (wire code UNAUTHENTICATED) |
| identity present, expression false or fails to evaluate | AccessDeniedException (wire code PERMISSION_DENIED) |
| identity present, expression true | handler runs |
A handler that throws AccessDeniedException (this library's or Spring's) also yields PERMISSION_DENIED, so a
nested send denied deeper in a chain reaches the original caller as AccessDeniedException.
How the receiver resolves the forwarded identity (ForwardedIdentityResolver):
Forwarded authorization header |
JwtDecoder bean |
Result |
|---|---|---|
| absent | any | anonymous; MessageContext.userId = null, request proceeds |
Bearer <jwt>, valid |
present | identity {sub, realm roles}, verified = true |
Bearer <jwt>, invalid or expired |
present | UNAUTHENTICATED (a forged or stale token is a bug or an attack) |
Bearer <jwt>, signed |
absent | claims read without verification, verified = false; warned once per process |
Bearer <jwt>, unsigned (alg: none) or encrypted |
absent | UNAUTHENTICATED (only signed tokens are read unverified) |
| malformed | any | UNAUTHENTICATED |
an unverified identity counts as authenticated. The peer that forwarded it validated it at its own
ingress and is trusted by the network model; verification is for catching forged or expired tokens where the
service can, not for distrusting the peer. Without a JwtDecoder the checks are therefore advisory: any client on
the private network can mint a signed token with any claims. Use principal.verified in an expression when a
handler must insist on a verified identity ("isAuthenticated() and principal.verified"). On the local
short-circuit the current Spring Authentication is normalised the same way: sub and realm_access.roles are
read from the token (JwtAuthenticationToken, BearerTokenAuthentication), so hasRole('x') means the realm role
regardless of how the application maps authorities for its own REST endpoints.
Notes:
@RequiresAuthenticationis read from the handler method itself and is not inherited from an interface or superclass method.permitAll(),denyAll()andisAnonymous()are accepted syntax but meaningless here: an anonymous caller is rejected withUNAUTHENTICATEDbefore any expression runs, so leave the annotation off a handler open to anyone.- A string result such as
'true'is not a grant; the expression must evaluate to the Booleantrue. - An expression that throws (for example
#message.owner.idon a nullowner) is a denial, not a handler error.
The resolved identity is installed in the SecurityContext while the handler runs, so CallerContext works inside
handlers and nested sends re-forward the same token. snodra.messaging.security.require-jwt was removed; setting it
fails startup with a hint.
Hub security: token, ACL and TLS¶
Since P5a the hub can authenticate services and restrict what they register and send, and serve its gRPC port over TLS
(decision record: ADR 0018, hub access control; operator how-to: https://snodra.damoreira.com/concepts/security/). All of it is opt-in
on the hub (SNODRA_SERVER_AUTH_MODE=off|permissive|enforce, default off); this section is the client's side. It is
unrelated to Security above, which is about the user's forwarded JWT.
The token. Give each service its own random token (openssl rand -hex 32) and the same one in the hub's ACL file
(which holds only its SHA-256):
snodra:
messaging:
hub:
token: ${BILLING_HUB_TOKEN} # or: token-file: /run/secrets/snodra-server-token
The token goes in Register.credentials.token of every stream (read once per stream; the hub reads it from the first
Register only). It is never logged, HubSettings.toString() and SnodraMessagingProperties.toString() print ****, and with
spring-boot-actuator on the classpath the starter contributes a SanitizingFunction that masks it (and
SNODRA_SERVER_TOKEN) in /actuator/configprops and /actuator/env even with show-values=ALWAYS and whatever
sanitization the application configured. An empty token is no token (a Register without credentials, which off and
permissive hubs accept). token-file wins when both are set (it is the source that can rotate); the
file is read at startup, where a missing file fails the startup, and on every reconnect, so replacing the secret and
letting the connection reconnect (or fail once) is a rotation. An empty file after a first good read (a rotation that
replaces the file non-atomically can be caught half way) is treated like an unreadable one: the last token read is kept
and one WARN is logged, so a Register never goes out without its token just because of a race. At startup an empty
file still means no token. A file that disappears later likewise keeps the last token read and logs one warning.
What the hub does with it, and what you see:
Hub (enforce) |
Stream or request | In the client |
|---|---|---|
| no token, an empty token, an unknown token (a JWT on a hub with no issuer: "jwt credentials are not configured") | the hub closes the stream with UNAUTHENTICATED |
registration rejection |
service-name the token does not grant, or a handled type outside its handle |
the hub closes the stream with PERMISSION_DENIED (naming the type) |
registration rejection |
a Register naming the instance id of a live session of another service (enforce and permissive; off keeps replacing it) |
the hub closes the stream with PERMISSION_DENIED (instance id … belongs to another service); the owner's session is untouched |
registration rejection |
a request whose type is outside send (unary, scatter-gather, streaming, subscription) |
an immediate Error{PERMISSION_DENIED}, nothing routed |
AccessDeniedException, never retried |
| a request after the token left the ACL | an immediate Error{UNAUTHENTICATED} |
UnauthenticatedException, never retried |
A registration rejection (ALREADY_EXISTS, INVALID_ARGUMENT, UNAUTHENTICATED, PERMISSION_DENIED, the last also for an instance id that belongs to another service): the status is
in HubConnection.lastRejection, awaitRegistered throws HubRegistrationRejectedException naming the code and the hub's
message, the connection reconnects at the backoff cap (80-100 % of hub.reconnect-backoff.max, no tight loop), logs at
ERROR once per distinct message and then at DEBUG, and keeps retrying (a fixed ACL or token file heals it without a
restart; lastRejection clears when a stream completes its handshake). The service keeps running; the hub counts every attempt as a
violation. In permissive nothing is refused: the hub logs and counts what enforce would refuse (except an instance-id
takeover, below, which it refuses too).
Instance ids belong to a service. A Register that names the instance id of a live session of another service is
refused with PERMISSION_DENIED (instance id … belongs to another service) in enforce and permissive, and the owner's
session is left alone, so a service with a valid token cannot hijack another one's stream and its queued requests. A
reconnect of the same service still replaces its own older stream. Make instance ids unique per instance (the default is
random) and never reuse another service's.
The back-off is validated. hub.reconnect-backoff.max must be positive and min positive and not above max; a bad
value fails the startup naming the property (a zero max would hot-loop handshakes against a refusing hub).
TLS. hub.tls.enabled=true with hub.tls.ca-file (the CA that signed the hub's certificate; the JVM's trust store when
absent) and, if the address is not a name on the certificate, hub.tls.server-name. It applies to every address form.
Plain text against a TLS hub (or TLS against a plain-text one) never hangs: the stream fails at once as
UNAVAILABLE (io exception) or INTERNAL (http2 exception) and the first WARN line that applies (not
necessarily the very first attempt: a hub that was still starting is a plain refusal, which carries no hint) says to check
snodra.messaging.hub.tls.enabled and tls.ca-file; later attempts log it at DEBUG. A client certificate (mTLS, P5c) is
its own section; DefaultHubChannelFactory.customizeTls stays a hook for a subclass.
Turning it on. enforce last: (1) run the hub with SNODRA_SERVER_AUTH_MODE=permissive and an ACL file; (2) give every service its
token (and the CA, if the hub serves TLS); (3) watch snodra_hub_auth_violations_total{reason,service} and the dashboard until
it stays flat; (4) switch the hub to enforce; (5) turn on TLS before tokens cross an untrusted network. A service
whose token is missing or wrong in enforce fails visibly (HubRegistrationRejectedException, an ERROR line) instead of
reconnecting in a tight loop.
Authenticating with a JWT (OAuth2 client credentials)¶
Since P5b a service can present a short-lived OAuth2 access token instead of a static token (decision record: ADR 0019,
hub JWT authentication). The hub validates it against the realm's keys (SNODRA_SERVER_JWT_ISSUER, ...), requires
its azp to equal the service name and takes the rights from the [[service]] table of that name in its ACL file.
The provider, in two lines. One confidential client per service with service accounts enabled and nothing else (no standard
flow, no direct access grants), its client id = the service name; add an audience mapper and set the hub's
SNODRA_SERVER_JWT_AUDIENCE. Never give a [[service]] table to the user-facing client.
snodra:
messaging:
hub:
oauth2:
token-uri: https://auth.example/realms/snodra/protocol/openid-connect/token
client-secret-file: /run/secrets/snodra-client-secret # client-id defaults to snodra.messaging.service-name
The environment forms are SNODRA_SERVER_OAUTH2_TOKEN_URI, _CLIENT_ID, _CLIENT_SECRET, _CLIENT_SECRET_FILE, _SCOPE and _CA
(relaxed binding also gives SNODRA_MESSAGING_HUB_OAUTH2_*); the properties win. A token URI together with a static hub.token
or hub.token-file, or without any secret, fails the startup. So does any hub.oauth2.* setting (or SNODRA_SERVER_OAUTH2_* variable)
without a token URI: a stray client id or secret would otherwise leave the service silently unauthenticated.
How it behaves.
- The
client_credentialsgrant is aPOSTto the token URI (application/x-www-form-urlencoded, 10 s, answer capped at 1 MiB, redirects never followed), the client authenticated with HTTP Basic (id and secret form-urlencoded first, RFC 6749 2.3.1). The answer needsaccess_token;expires_inmissing means 60 s (oneWARN); atoken_typeother thanBeareris refused. - One token per process, shared by every hub connection (N hubs, one fetch), fetched single-flight: connections that start
together wait for the same request. Each new stream (connect, reconnect) gets a token before it sends
Register. - A cached token is used while more than its refresh margin is left; the provider renews it in the background at that
point: margin = 20 % of the lifetime, at least 5 s, and at most half of it (a 6 s token is renewed after 3 s; the Rust library
uses the same rule). A lifetime over 10 years (a huge
expires_in) is cut to 10 years. The renewer looks at the wall clock at least every 5 s, so a clock that jumped ahead while the machine slept does not postpone the renewal. The renewal goes to every live stream (one that has its ack) asClientMessage.credentials. The hub closes a JWT session atexpplusSNODRA_SERVER_JWT_EXPIRY_GRACE(30 s) without one. - A renewal that fails keeps the current token and retries with a back-off from 1 s to 30 s (one
WARNper distinct error). - A stream the hub closes (or a
Registerit refuses) withUNAUTHENTICATED("token expired", "unknown signing key", "credentials expired", a refused refresh) drops the cached token: the reconnect fetches a new one instead of presenting the same one until its margin runs out.PERMISSION_DENIED(wrongazp, no[[service]]entry) keeps it: a new token would not help and the identity provider is not asked again for every retry. - No token, no stream. When the endpoint cannot give a token (401
invalid_client, down, TLS, an unusable answer) the stream is not opened: the attempt fails locally,HubConnection.lastRejectionholdsUNAUTHENTICATEDandawaitRegisteredthrowsHubRegistrationRejectedException("no access token for ...: invalid_client: ..." with the endpoint'serroranderror_description, clipped to 200 characters), the connection retries with the normal reconnect back-off, logsERRORonce per distinct error and thenDEBUG, and recovers by itself when the endpoint does. A hub that is not ready (UNAVAILABLE"jwks not loaded yet") is an ordinary reconnect. - With a static
hub.token-filethe same message rotates the token without a reconnect: when the file's content changes the new token is sent asClientMessage.credentialson every live stream (checked once per heartbeat). - The client secret and the access tokens never appear in logs, exception messages,
toString()or the actuator'sconfigpropsandenv(the starter'sSanitizingFunctionmaskshub.oauth2.client-secretandSNODRA_SERVER_OAUTH2_CLIENT_SECRET).
What you can read. HubConnections.authStatus() returns kind (NONE, TOKEN, JWT), the current token's tokenExpiresAt, the
lastTokenError (null once a fetch worked), tokenFetches by result and credentialsRefreshesSent (one process-wide counter: it never drops when a hub connection goes away): wire it into your own
health indicator. With Micrometer on the classpath the starter binds snodra_messaging_hub_token_fetches_total{result}
(ok, http_error, oauth_error, io_error, bad_response) and snodra_messaging_hub_credentials_refreshes_sent_total.
Divergences from the Rust library (same design, ecosystem differences): the Kotlin starter reports through
HubConnections.authStatus() and Micrometer counters where Rust has an accessor. A token URI host with an underscore in
its name is refused by the JDK's URI parser (the host would be null), so it fails the startup here; the Rust library accepts it.
Authenticating with a client certificate (mTLS)¶
Since P5c the hub can take a client certificate as the service's credential (decision record: ADR 0020,
hub mutual TLS; issuing certificates and the hub's settings: https://snodra.damoreira.com/concepts/security/, "mTLS mode"). The TLS handshake
proves the service holds the key, the certificate names the service, and the rights are the hub's [[service]] table of that name, the
same table a JWT uses. No secret crosses the wire and no identity provider is involved.
snodra:
messaging:
service-name: billing-service
hub:
tls:
ca-file: /etc/snodra/hub-ca.pem # the CA that signed the hub's certificate (or the system roots)
client-cert-file: /etc/snodra/tls/client.pem # certificate chain, leaf first
client-key-file: /etc/snodra/tls/client.key # unencrypted PKCS#8 PEM
Environment names (relaxed binding): SNODRA_MESSAGING_HUB_TLS_CLIENT_CERT_FILE, SNODRA_MESSAGING_HUB_TLS_CLIENT_KEY_FILE.
- Both files or neither. One alone fails the startup naming the missing one. A client certificate implies TLS:
hub.tls.enabledneed not be set, and an explicithub.tls.enabled=falsenext to the certificate settings fails the startup (the Rust library does the same withSNODRA_SERVER_TLS_CLIENT_CERT/_KEY). - One credential kind per service. A client certificate together with a static token (
hub.token,hub.token-file,SNODRA_SERVER_TOKEN,SNODRA_SERVER_TOKEN_FILE) or OAuth2 (hub.oauth2.*,SNODRA_SERVER_OAUTH2_*) fails the startup naming both: a stale token next to a good certificate would be refused by the hub. With a certificate the library sends no credentials at all. - The SAN rule. The leaf must carry exactly one URI SAN
snodra://service/<name>(the name is 1 to 128 characters of[A-Za-z0-9._-]; DNS, IP and other URI SANs are ignored; two such SANs, even equal ones, are refused as the hub refuses them), and<name>must besnodra.messaging.service-name. The hub would refuse anything else withPERMISSION_DENIEDatRegister; the starter says so at startup instead. Also checked at startup: both files exist, are not empty and parse, the key belongs to the certificate, and the certificate has not ended. The key must be PKCS#8 PEM (-----BEGIN PRIVATE KEY-----); RSA (BEGIN RSA PRIVATE KEY), SEC1 (BEGIN EC PRIVATE KEY) and encrypted keys are refused with the conversion hint (openssl pkcs8 -topk8 -nocrypt). The JDK reads RSA, EC and Ed25519 keys. - How to issue one. Follow
https://snodra.damoreira.com/concepts/security/, "mTLS mode" (openssl, or cert-manager withuris: [snodra://service/billing-service]andusages: [client auth]). Issue short-lived certificates (hours to days): there is no revocation list, so the lifetime is the revocation story. - Failures at the handshake are transport errors. With
SNODRA_SERVER_TLS_CLIENT_AUTH=required, no certificate or one the hub does not trust fails the TLS handshake, which TLS 1.3 shows when the stream opens:UNAVAILABLE("ssl exception"), not a registration rejection. The library retries with its normal back-off and logs oneWARNthat names the fingerprint and what to check (the hub'sSNODRA_SERVER_TLS_CLIENT_CA, the certificate's end,hub.tls.ca-file). A certificate that names no or several services, or another service thanservice-name, isPERMISSION_DENIEDatRegister: a rejection (HubRegistrationRejectedException). A client that presents no certificate to a hub that requires one gets the same transport failure; itsWARNsays that a hub which requires client certificates does that to a client that presents none, and to sethub.tls.client-cert-fileandclient-key-file.
Rotation without a restart. The files are read again on every (re)connect and once per check tick (3 s, the hub's heartbeat period).
A new connection attempt whose files cannot be used presents the last good certificate instead, with one WARN per problem. When the pair changed and is valid (the same checks as at startup, not expired, already valid), the instance first waits a random delay, uniform in
[0, min(30 s, 10 % of what is left of the certificate its streams present)] (none when that certificate has ended), so the replicas of a service
that read the renewed file within the same tick do not all deregister at once; the files are read again after the wait and the roll goes to the
pair that is current then. Then the hub connections roll over to it
one hub at a time: (1) the hub is told this instance is stopping (a Register with no types, as at shutdown), so it routes new work
to other instances; (2) the library waits up to 10 s for the work in flight on that stream (requests it sent and has no answer for,
requests it is handling; not the streams and subscriptions this instance itself consumes, see below); (3) the stream is closed gracefully (half-close, the hub's end awaited for 2 s); (4) it reconnects at once
(no back-off) with the new pair and waits up to 30 s for it to register again; only then the next hub is rolled, in target order. Write
the key first and the certificate second, or both with one rename each: a pair that does not match (a half-written replacement) is not
adopted and the current connection stays; the problem is logged at DEBUG when first seen and at ERROR (once) when the next check finds
it again, and the next check adopts the pair when it is complete.
- Failed requests. Requests accepted before the roll finish on the old stream. A sender that picks a hub in the middle of its roll gets
NO_HANDLERfrom it when this instance was the only owner there, so a unary send now fails over to another live hub that has a handler (every hub answeringNO_HANDLERreturns the answer at once). A hub that has another live hub leaves this instance's own routing at once; the only hub leaves it after its work in flight finished. - One hub: a gap. With a single hub there is nobody to take over: from the hub being told until the new stream is registered, the
instance has no stream and takes no work. That is up to the drain (10 s at most, usually much less: only requests that are really in
flight are waited for) plus the reconnect. Its own sends wait for the new stream within the call's deadline; requests other services send to
it meanwhile get
NO_HANDLERfrom the hub (give such a service two instances, or two hubs). The random delay above keeps the replicas of a service from all being in that gap at the same moment. - Streams and subscriptions. A streaming handler counts as work in flight (it is waited for, up to 10 s, then cut like any lost stream).
A subscription this instance serves does not hold the roll: it ends with the old stream (the hub tells the subscriber
HANDLER_DISCONNECTED), as at any reconnect. The same holds on the consumer side: streams (queryBus.stream) and subscriptions (queryBus.subscribe) this instance itself consumes through the hub being rolled are not waited for either (they have no end of their own, so waiting would always cost the full 10 s); the roll cuts them, the stream or theupdatesflow fails withHubConnectionLostException, and the consumer subscribes again, as after any lost stream. A scatter-gather in flight, which ends within its deadline, is waited for. - Requests not yet written when the stream closes follow the usual lost-stream rules (
HubUnavailableException, sent again on another hub). - A roll that does not complete (the new stream does not register within 30 s) stops the rollover for that pair with one
ERROR: the other hubs keep the old pair until the files change again (the hub that did not register keeps trying with the new pair on its own).
Expiry. The hub closes a certificate session at notAfter with UNAUTHENTICATED "certificate expired" and no grace. A service whose
files are renewed before that never meets it. Otherwise: with less than a tenth of its lifetime (24 hours at most) left and no renewed
file, the library logs one WARN per certificate; when the hub closes the session at notAfter the library treats it as a normal
reconnect (it is not a registration rejection: no back-off at the cap) and reads the files again; if they still hold the expired
certificate, the attempt fails locally, without a handshake, as a registration rejection (UNAUTHENTICATED "client certificate ... expired
at ...": one ERROR, then retried at the back-off cap), and a renewed file is picked up by the next attempt.
What you can read. HubConnections.authStatus() has kind = CERTIFICATE, certificateService, certificateFingerprint (the first 8 hex characters of
the leaf's SHA-256, the only form of the certificate that is logged or shown; the hub's topology shows the same value), certificateExpiresAt
(notAfter) and certificateRotations (pairs adopted since start). With Micrometer the starter binds
snodra_messaging_hub_certificate_rotations_total. Logs name a certificate only by that fingerprint and the service; no key material
is ever logged, and toString() of the certificate objects shows neither key nor certificate.
Divergences from the Rust library (same design, ecosystem differences): the Kotlin starter reports through authStatus() and a
Micrometer counter where Rust has hub_auth(); the status fields are certificate* next to the existing tokenExpiresAt; the JDK's
PKIX validation rules apply to the hub's certificate (the Rust library uses rustls).
Configuration keys¶
| Key | Default | Meaning |
|---|---|---|
snodra.messaging.service-name |
required | Logical name; Kafka consumer group; Register.service_name |
snodra.messaging.type-prefix |
required | Prefix for classes without @MessageType |
snodra.messaging.hub.enabled |
true |
Set false to skip the hub connection (publish-only tools, unit tests) |
snodra.messaging.hub.address |
static://snodra-server:7000 |
The hub or hubs (plaintext gRPC): one target (host:port, static:// optional), a comma-separated list (static://hub:7000,hub-2:7000; static:// optional per entry, blanks and repeats dropped), or one dns:///name:port (port required, not combinable with other entries) whose addresses are re-resolved every 30 s. One connection per hub, see Several hubs. A malformed value fails startup |
snodra.messaging.hub.status-interval |
5s |
Period of the InstanceStatus report (connected hub ids, handlers in flight) to every live hub; also sent whenever the set of live hubs changes. Must be positive |
snodra.messaging.hub.request-timeout |
unset | Per-request timeout chosen by this service. Unset: the hub decides (timeout_ms = 0: its default, 5 s, or the type's override) and the client guards with the hub's bound for the type + 1 s. Set: sent as timeout_ms (the hub still clamps it, and its per-type override can then only shorten it), so leave it unset unless the service needs its own deadline; 0s counts as unset |
snodra.messaging.hub.reconnect-backoff.min |
500ms |
Exponential backoff floor when the stream drops; the backoff returns to it once an attempt reached the handshake |
snodra.messaging.hub.reconnect-backoff.max |
5s |
Exponential backoff ceiling; every wait gets ±20 % jitter and never exceeds this (at the ceiling: 80-100 % of it) |
snodra.messaging.hub.max-concurrent-handlers |
256 |
Incoming requests handled at once by this instance over all hubs (at least 1, else startup fails); also sent as Register.initial_permits to every hub (split over N hubs, see Several hubs), which queues the excess. One permit goes back per completed handler: at once while the hub holds min(16, max-concurrent-handlers / 2) or fewer of them, else batched at 16 or after 50 ms |
snodra.messaging.hub.load-factor |
100 |
Relative capacity of this instance (Register.load_factor; 0 reads as 100, negative fails startup): weights its share of keyed commands (saturating at 800 on the ring) and its load score once instances are busy |
snodra.messaging.hub.tags.<name> |
none | Register.tags; unkeyed requests prefer instances whose tags match the sender's |
snodra.messaging.hub.priorities.<type> |
none (0) | Default Request.priority per message type; SendOptions.priority wins |
snodra.messaging.hub.max-subscriptions |
1024 |
Subscription queries this instance serves at once, over every hub (at least 1, else startup fails). The next one is refused with ReplyEnd{RESOURCE_EXHAUSTED, "subscription limit (N) reached"}, which the subscriber sees as a HandlerBusyException. A subscription holds no handler permit, so max-concurrent-handlers does not bound it; the default equals the hub's SNODRA_SERVER_MAX_SUBSCRIPTIONS_PER_INSTANCE. See Subscription queries |
snodra.messaging.hub.token |
none | Static token sent in Register.credentials.token of every hub stream (hub access control, Hub token and TLS). An empty or blank value is no token. Never logged, redacted in toString and in the actuator's configprops and env. Environment: SNODRA_MESSAGING_HUB_TOKEN, or SNODRA_SERVER_TOKEN when the key is unset |
snodra.messaging.hub.token-file |
none | File holding the token (a container secret such as /run/secrets/snodra-server-token). Read at startup (an unreadable file fails it) and again on every reconnect; if a later read fails the last good token is kept. Wins over hub.token when both are set. Environment: SNODRA_MESSAGING_HUB_TOKEN_FILE, or SNODRA_SERVER_TOKEN_FILE when the key is unset |
snodra.messaging.hub.tls.enabled |
false |
TLS to every hub of hub.address (every address form). The hub must serve TLS (SNODRA_SERVER_TLS_CERT / SNODRA_SERVER_TLS_KEY). Environment: SNODRA_MESSAGING_HUB_TLS_ENABLED |
snodra.messaging.hub.tls.ca-file |
JVM trust store | PEM file with the CA that signed the hub's certificate; read on every connection attempt. Set without tls.enabled it is ignored with a warning. Environment: SNODRA_MESSAGING_HUB_TLS_CA_FILE |
snodra.messaging.hub.tls.server-name |
the address's host | Name the hub's certificate must carry (and the SNI), for an address that is an IP or an alias the certificate does not list. Environment: SNODRA_MESSAGING_HUB_TLS_SERVER_NAME |
snodra.messaging.hub.retry.enabled |
false |
Opt-in retry of the transient errors (see "Send options, priority and retry"); SendOptions.retry overrides per call. The retry.* values below are validated at startup even while this is false (attempts >= 1, multiplier >= 1, no negative backoff) |
snodra.messaging.hub.retry.max-attempts |
3 |
Total attempts, the first one included |
snodra.messaging.hub.retry.initial-backoff |
100ms |
Wait before the second attempt |
snodra.messaging.hub.retry.multiplier |
2 |
Growth of the wait per further attempt |
snodra.messaging.hub.retry.max-backoff |
2s |
Upper bound of a single wait |
snodra.messaging.kafka.bootstrap-servers |
broker:19092 |
Kafka |
snodra.messaging.kafka.topic |
snodra.events |
Events topic |
snodra.messaging.kafka.consumer.enabled |
true |
Set false in tools that only publish |
snodra.messaging.kafka.consumer.max-attempts |
5 |
Handler attempts per record before it is logged and skipped |
snodra.messaging.outbox.enabled |
true |
Outbox collection + publisher |
snodra.messaging.outbox.poll-interval |
200ms |
Publisher poll period |
snodra.messaging.outbox.batch-size |
100 |
Documents per poll |
snodra.messaging.outbox.retention |
7d |
Delete sent documents older than this |
snodra.messaging.idempotency.retention |
30d |
Delete processed_events older than this |
snodra.messaging.security.forward-jwt |
true |
Put the current request's bearer token in headers.authorization |
The keys under snodra.messaging.sagas.*, deadlines.* and external.* are listed with their meaning in
Sagas and deadlines.
Environment variable form follows Spring's relaxed binding, for example
SNODRA_MESSAGING_HUB_ADDRESS. Map keys taken from environment variables are lowercased with _ read
as ., so snodra.messaging.hub.priorities.<type> cannot be set that way for a mixed-case type name
(orders.CancelOrder): put it in application.yaml, or in SPRING_APPLICATION_JSON when only the
environment is available.