Observability Routing — egress consumer that drains the buffer to Mimir / Loki / SIEM

This document is the authoritative bounded-context reference for the Observability Routing context — the egress half of the observability pipeline. Where the ingest front dooradmits, validates, and buffers every batch onto a per-Domain JetStream stream, this context drains those streams back out: it consumes one buffered batch at a time, resolves the routes the originating Project states, selects the records each destination receives, transforms them into the wire shape that destination expects, and exports them. A Project that states no route for a signal is served by the platform backends that have always carried it: Grafana Mimir (metrics), Grafana Loki (logs and audit), and the audit SIEM (logs and audit). The domain root that pins the ubiquitous language is ../../../internal/observability/routing.

Routing is a pure backend consumer. Unlike the ingest front door it mounts no /v1 HTTP route and installs no NSK middleware — a batch does not enter this pipeline off a request, it enters off a JetStream durable consumer. The context owns no aggregate the way ingest owns Batch, and it owns neither the Route nor the Sink: it owns the three built-in transforms, the outbound Exporter port and its three net/http adapters, the export-outcome classification, the read-through cache a delivery resolves its routes with, the consumer lifecycle that keeps one durable per destination and stream, and the application service that drives one destination-bound consume loop from buffered delivery to ack or nak. Its only HTTP surfaces are outbound: the POSTs each Exporter makes to its downstream backend.

Ubiquitous language

The terms below travel verbatim across the domain root, the three transforms, the Exporter port and its adapters, and the application service. Documentation, message headers, and metric labels adopt the exact spelling.

Term	Definition	Code anchor
Route	The per-Project aggregate that states which records of one signal reach which Sinks. It lives in the sibling Telemetry routes context; this engine resolves a delivery's Project to that Project's routes and selects the records each targeted Sink receives.	../../../internal/observability/routes/route.go (Route)
Sink	A destination a Route delivers to: one of the three platform backends (Grafana Mimir for metrics, Grafana Loki for logs and audit, the audit SIEM for normalised audit events) or a tenant-authored destination a Domain declared. A delivery that resolves to no route for its signal goes to the platform backend that has always carried that signal.	../../../internal/observability/sinks/sink.go (Sink)
Resolved route	The read model a delivery consults: one Route flattened to its predicate and to the targets the Project may currently use. An empty target set is not a fallback (see route resolution).	../../../internal/observability/routes/types.go (ResolvedRoute)
Exporter	The outbound port — Export(ctx, ExportRequest) error — that ships one already-transformed payload to a single Sink and classifies the outcome. A nil error means the export was accepted; a non-nil error MUST wrap either ErrExportRetryable (the consumer redelivers) or ErrExportRejected (the consumer drops).	../../../internal/observability/routing/export.go (Exporter)
ExportRequest	The unit of work an Exporter ships: the transformed, Sink-specific wire body (Payload) plus the routing metadata that addresses and labels the destination — Domain, Project, Node, Signal, and the optional per-request Headers a transform produced (the SIEM envelope).	../../../internal/observability/routing/export.go (ExportRequest)
Router	The application service that drives one Sink-bound consume loop: it parses the subject and the buffer-stream headers, resolves the Project's routes, parses the batch body when this destination carries it at all, runs the Sink-specific transform, hands the transformed payload to the Sink's Exporter, and then acks or naks the delivery according to the export outcome.	../../../internal/observability/routing/services/router.go (Router)
Sink-bound consumer	A durable JetStream consumer that binds exactly one Sink to one ingest stream. Its name is a wire identifier operators see in nats consumer ls: the five built-in names are fixed constants, a tenant sink's name is minted per sink and signal by TenantConsumerName.	../../../internal/observability/routing/consumers.go (TenantConsumerName)
Export-outcome sentinels	The two sentinels every export attempt is classified into — ErrExportRetryable (a transport error, a 429, or any 5xx: the consumer redelivers) and ErrExportRejected (any other non-2xx, i.e. a non-429 4xx: terminal, the consumer drops). Callers detect them with errors.Is.	../../../internal/observability/routing/errors.go (ErrExportRetryable, ErrExportRejected)

The default destinations

The three platform backends are where a signal goes when the originating Project states no route for it. A deployment holding no route rows at all therefore behaves exactly as it did before routes existed: every metrics batch reaches Mimir, every logs and audit batch reaches Loki and the SIEM. The mapping below is the default, not a fixed topology, and a Project's routes replace it per signal.

Each backend reads one or more ingest streams, transforms the buffered batch into a different wire format, and POSTs it to a different endpoint with a different tenant / auth header set. The transform is not interchangeable: a metrics batch becomes remote-write protobuf, a logs / audit batch becomes a Loki JSON push body, and the same logs / audit batch becomes verbatim NDJSON for the SIEM.

Sink	Source stream(s)	Wire format	Endpoint (POST)	Tenant / auth headers
Mimir (mimir)	metrics	snappy-compressed Prometheus remote-write protobuf	<MimirURL>/api/v1/push	Content-Encoding: snappy, Content-Type: application/x-protobuf, X-Prometheus-Remote-Write-Version: 0.1.0, X-Scope-OrgID: <domain>
Loki (loki)	logs, audit	JSON push body	<LokiURL>/loki/api/v1/push	Content-Type: application/json, X-Scope-OrgID: <domain>
SIEM (siem)	logs, audit	verbatim NDJSON	<SIEMURL> (the full endpoint — no path suffix appended)	the SIEM envelope (see SIEM forwarder)

The remote-write specification Mimir consumes is the public Prometheus remote-write protocol; the Loki push body matches the public Grafana Loki push API. X-Scope-OrgID carries the originating Domain as the multitenancy key for both Mimir and Loki, so each tenant's series and streams land in that tenant's isolated store rather than collapsing into one shared namespace.

The endpoint construction differs across the three: Mimir and Loki append a fixed path (/api/v1/push and /loki/api/v1/push) to the configured base URL, while the SIEM URL is treated as the full endpoint and POSTed to verbatim — the SIEM endpoint a customer configures is the exact ingest URL, not a base to which a path is added.

A tenant sink's delivery carries no built-in transform. The selected records travel raw, in the shape the ingest parser validated, and the protocol adapter owns the wire format. The ExporterRegistry puts those adapters in front of the platform's own backends: a built-in sink delegates to the built-in provider unchanged, and a tenant sink resolves through the factory registered for its type. Both tenant-declarable types resolve to an adapter the platform ships: otlp (see OTLP mapping) and syslog (see Syslog mapping).

The registry is wiring only. It states nothing about what a destination can receive: routes.AcceptedSignals remains the capability table both route validation and the consumer lifecycle read, and the Sink aggregate's settings validation remains the per-type configuration schema. Registering a protocol is therefore three things: an adapter package, an entry in the composition root's factories map, and the entries in those two static tables. A factory keyed on a platform backend is refused at construction, because a built-in sink is answered by the built-in provider and never reaches the factories.

Route resolution

Every delivery resolves the routes of the Project the ingest publisher stamped onto the batch, and the routes it finds for the delivery's signal decide which records this destination receives. Three cases:

• No route for the signal. The whole batch goes to the destination when it is that signal's default backend, and nowhere otherwise. This is the path every delivery took before routes existed.
• Routes for the signal. Only the records a route targeting this destination selects travel, each at most once even when two routes select it. The platform backends are ordinary targets here: a Project's route reaches Loki only if it names the loki sink.
• A project header that names no Project. An empty or non-UUID header resolves to no routes without reading the source, because an id that cannot address a Project cannot address a route set either. The delivery takes the default path rather than redelivering forever against an id no source will recognise.

Both fail-closed corollaries of the replace rule are enforced here. A route whose usable targets are all gone selects records for nobody, and its slice of the batch goes dark rather than flowing back into the platform backends. A record whose predicate field does not read (an undecodable body, or a severity keyword outside the roster ingest admits) does not match a route that states a predicate. The Telemetry routes page carries the reasoning for both.

Resolution decides before the batch is parsed. Which routes name this destination is a property of the routes and the destination alone — no record is involved — so it is answered once per delivery, ahead of the parse. A destination no route names, and that carries no default for the signal, acks the batch as skipped without decoding any of its up to ten thousand records. Only once a destination is known to carry the batch do the per-record predicates run.

One consequence is worth stating: a malformed batch is counted dropped only by the destinations that would have carried it, and skipped by the rest, where before it was counted dropped by every destination it reached.

The resolver is a read-through TTL cache keyed by Project, with a 30-second default. Without it every delivery would put one database round trip in front of the batch; with it a route the operator just changed reaches delivery within one TTL. A source failure propagates instead of serving the expired entry, and nothing is written to the cache: the delivery is undecided, so it is naked for redelivery and counted on resolution_failures_total rather than exported or dropped.

The resolution is bounded by the per-export HTTP timeout (10s by default). It ends in a connection-pool acquisition, and an unbounded one blocks forever on a saturated pool or a database failover; because the consume callback dispatches sequentially, one such call would wedge the whole durable's loop without a log line or a counter to show for it. A resolution that overruns the ceiling naks on the export backoff instead.

Predicate fields are decoded lazily and at most once per record, so a batch whose routes state no predicate is never decoded at all and its records reach the Exporter as the bytes the ingest parser validated.

Consumer topology

The pipeline drains the three ingest streams through one durable consumer per (sink, stream) pair. The configured platform backends get five fixed durables, whose names are the wire identifiers an operator sees in nats consumer ls:

Durable consumer	Sink	Source stream
obs-route-mimir	Mimir	metrics
obs-route-loki-logs	Loki	logs
obs-route-loki-audit	Loki	audit
obs-route-siem-logs	SIEM	logs
obs-route-siem-audit	SIEM	audit

Why one consumer per (sink, stream), not one shared consumer per stream. JetStream tracks a single ack cursor per durable consumer. If Loki and SIEM both read the logs stream through one shared consumer, a slow or down sink (the SIEM during an outage, say) holding un-acked messages would head-of-line block the healthy sink (Loki) reading the same stream — stalling healthy egress behind a sick one. A dedicated consumer per (sink, stream) gives each sink its own ack cursor, so a sink that falls behind only delays itself, and a sink enabled later starts from DeliverAll over whatever the stream still retains.

Each consumer is configured AckExplicit + DeliverAll with no MaxDeliver. A poison batch is therefore dropped by the Router's terminal-ack logic, never by a JetStream MaxDeliver dead-letter: the Router acks on a successful export and on a terminal drop, and naks (with a delay) only on a retryable outcome, so a batch the downstream will never accept is acked-and-dropped exactly once rather than circulating until a delivery ceiling expires it.

Deterministic capped backoff. A retryable outcome schedules the redelivery with a deterministic delay of min(5s · 2^(n-1), 60s), where n is the server-tracked delivery count. The sequence is 5s, 10s, 20s, 40s, 60s, 60s, … — exponential until it clamps at the one-minute ceiling — so a transient downstream outage backs off rather than hot-looping, without ever exceeding a minute between attempts.

The 24h replay horizon. The ingest streams' MaxAge is fixed at 24h. That window bounds how far back a freshly-enabled or recovered consumer can replay: a sink wired up (or brought back) after an outage catches up over only the last 24h of retained telemetry, and anything older has already aged out of the stream. The horizon is the same one the ingest doc calls out as deliberately fixed, not an operator knob.

At-least-once delivery and the SIEM dedup note. Because there is no MaxDeliver and a nak redelivers, delivery is at-least-once: a batch may be exported more than once (a successful export whose ack is lost, or a downstream that accepted a batch but returned a retryable status). A downstream SIEM should therefore treat ingestion idempotently and de-duplicate on the (node, sent_at, record) tuple — the originating Node id, the batch send time, and the verbatim record — so a redelivery does not double-count an audit event.

Tenant consumers and their lifecycle

A declared tenant sink gets one durable per signal its type receives, named obs-route-sink-<sink id>-<signal>. The split is the same one the five fixed names encode: one ack cursor per sink and stream, so a sink that falls behind on logs never blocks its own audit egress. The obs-route-sink- prefix is what an operator greps nats consumer ls with to tell tenant durables from the built-in five, and it is what the lifecycle matches on to find the durables it owns.

Each tenant durable is filtered to its Domain's subject for that signal (obs.<signal>.<domain id>). The buffer streams carry every Domain, so an unfiltered tenant consumer would read every other tenant's traffic. The five built-in durables are deliberately unfiltered: a platform backend receives every Domain's telemetry, which is the fan-out the pipeline has always had.

A Manager reconciles the desired set at boot and on a 30-second interval while the process runs. One reconcile lists the declared sinks, computes the durables the configured backends and those sinks require, provisions and binds a consume loop to each one that has none, and stops the loops that are no longer wanted. The interval bounds how long a sink an operator just declared waits for its consumer, and how long a deleted sink's durable survives. A failed listing stops nothing: an empty desired set computed from a failed read would tear down every healthy loop on a transient database blip.

A tenant loop is rebound when the sink it routes for changes id or revision. The revision is the row's updated_at, and it moves when an operator re-points the sink's endpoint, tightens its TLS posture, rotates its credential or changes its protocol. The Exporter is derived from the sink row on every reconcile, so a loop that was left alone across such a rewrite would keep shipping to the old destination, over the old TLS posture, with the old credential, until the process restarts.

A built-in loop is rebound on the id alone, which changes exactly once — when the backend's row appears after the loop was bound, since the routing reconcile runs before the built-in sink seed. It deliberately does not key on the revision: a built-in binding's Exporter comes from the egress configuration by type and never from the row, while the seed's upsert touches updated_at on every boot. Keying on it would tear all five built-in loops down and back up once per boot — and, because the built-in rows are global rather than per-replica, one replica booting would churn every other replica's loops within an interval, redelivering whatever was in flight each time.

A rebind stops and restarts the loop only; the durable and its ack cursor are untouched.

A loop the server ended is rebound too. Every reconcile first drops the loops that have already ended on their own, before it compares anything else. A durable removed server-side — a nats consumer rm, a stream recreated during a broker upgrade — makes the NATS client treat the next status as terminal: it stops the subscription and never rebinds it. The loop reports that end on the channel the consume bind hands back, so the pass sees the entry as missing, recreates the durable and binds a fresh loop to it. Without that step the identity comparison above would match the dead entry and skip it forever, so audit deliveries would pile up on the stream until the 24h retention dropped them while /readyz stayed green.

A tenant durable is deleted, a built-in durable never is. When a tenant sink disappears, nothing is left to redeliver to and the durable would otherwise accumulate on the stream forever. A built-in durable whose backend URL an operator unset is stopped and left in place: its name is the key of its ack cursor, so deleting and recreating it would replay the stream's whole retention into that backend on the next enable.

A reconcile closes with an orphan sweep: it lists each buffer stream's consumers and deletes every obs-route-sink--prefixed durable the desired set no longer holds. That catches the sink deleted while the process was down, whose durable no running loop knows about. Names without the prefix are never touched, so the five fixed durables and any consumer another component owns stay where they are.

A destination the platform has no exporter for gets no consumer at all rather than one that buffers deliveries until an adapter appears. A durable created early would start delivering immediately and hold its messages un-acked until the 24h retention drops them, which stalls the consumer behind its in-flight ceiling and leaves an operator an un-drained durable to explain. Created later it costs nothing: it starts from DeliverAll over whatever the stream still retains, the same window a sink enabled later gets.

Mimir mapping

The Mimir transform decodes each buffered metric record to {group, name, value, timestamp, labels} and turns each surviving record into one remote-write TimeSeries.

The label set. Each surviving record carries five reserved labels the transform sets itself, plus the record's sanitized user labels:

Reserved label	Value
__name__	the sanitized metric name
group	the record's metric group
domain	the originating tenant
project	the originating Project id
node	the originating Node id

A user label whose sanitized name collides with one of the five reserved names is dropped and counted (the record_drops_total-adjacent reserved-collision count), so a caller can never spoof the tenant-scoping labels Mimir keys on.

Sanitization. Label and metric names are coerced into the Prometheus identifier grammar — a metric name to [a-zA-Z_:][a-zA-Z0-9_:]* and a label name to [a-zA-Z_][a-zA-Z0-9_]*. Note : is permitted in a metric name but not in a label name; a result that is empty or begins with a digit is _-prefixed. Label values are not sanitized — they are opaque UTF-8 that Mimir accepts verbatim, and coercing a value would silently corrupt a legitimate URL or path.

Sorting and timestamps. A series' labels are sorted lexicographically by name before encoding, and each record's timestamp is emitted in milliseconds (UnixMilli).

The closed drop-reason set. A record that fails one of the parse steps becomes a drop, in record order, carrying one of three reasons:

Drop reason	Meaning
malformed_value	the value field is present but not a JSON number
malformed_timestamp	the timestamp field is not a JSON string, or does not parse as RFC3339Nano
undecodable	the record bytes are not a JSON object at all

Surviving records still ship even when some siblings drop. Only when every record is malformed does the transform yield a nil payload; the Router then reaches a terminal disposition with nothing to export and the batch is dropped (counted dropped), never POSTed.

Why the wire shape is hand-encoded. The WriteRequest / TimeSeries / Label / Sample four-message subset is hand-encoded with google.golang.org/protobuf/encoding/protowire and snappy-framed with github.com/klauspost/compress/s2, rather than vendoring github.com/prometheus/prometheus/prompb. Those four messages are field-stable and have not changed across remote-write versions, and the transform only ever serialises them — it never reflects over them — so pulling a large, transitively heavy generated proto package and its descriptor machinery for four fixed messages was rejected in favour of a std-lib-plus-protowire/s2 core.

Loki mapping

The Loki transform turns one buffered logs or audit batch into a single Loki stream whose labels are exactly {signal, domain, project, node}. Each record contributes one [<unix-ns-string>, <line>] value entry, where:

• the line is the raw record bytes verbatim — the transform does not re-serialize the record, so the line Loki stores is byte-for-byte what the Node produced; and
• the nanosecond timestamp is the record's own RFC3339Nano timestamp field, or the batch sentAt when that field is missing, not a JSON string, or unparseable — in which case the record is still emitted and a timestamp-fallback is counted. Records are never dropped on the Loki path; the fallback count is a quality signal, not an error count.

X-Scope-OrgID carries the Domain as the Loki tenant.

Why the label set is fixed at four bounded keys. Promoting a log's severity (or an audit event's source) to a fifth stream label was rejected: Loki indexes one stream per distinct label-value combination, so a per-record-varying label multiplies stream cardinality within a batch and across the tenant — the failure mode Loki operators most need to avoid. severity, source, and every other field stay queryable inside the JSON line through LogQL's json parser, so nothing is lost by keeping them out of the index.

The SIEM forwarder

The SIEM forwarder ships the buffered records verbatim: it joins the raw record byte slices with \n and a trailing \n into one NDJSON body, with no re-serialization of any record. This preserves byte-exact audit provenance and leaves all parsing and normalization to the SIEM, which owns its own schema — re-encoding here would both lose provenance and risk a lossy round-trip through Go's JSON encoder.

The body is POSTed to the configured SIEM endpoint under the SIEM ingest envelope:

Header	Carries
Content-Type	application/x-ndjson
X-Plexsphere-Signal	the source Signal (logs / audit)
X-Plexsphere-Domain-Id	the originating tenant
X-Plexsphere-Project-Id	the originating Project id
X-Plexsphere-Node-Id	the originating Node id
Authorization	Bearer <token> — only when a SIEM token is configured

The bearer is attached by the transform, not the exporter, so the token policy lives in exactly one place and the exporter stays a pure request shipper.

Production-TLS expectation. The outbound SIEM connection's TLS is satisfied by configuring an https:// SIEM endpoint URL at the composition root. The pure transform and the exporter hold no TLS policy of their own — the shared platform HTTP client carries the transport configuration, and choosing an https scheme for the endpoint is what puts the SIEM POST on TLS.

OTLP mapping

An otlp sink is a tenant-declared destination that speaks OTLP/HTTP, and it is the one tenant protocol the platform ships an adapter for. The adapter lives in ../../../internal/observability/routing/otlp: it encodes the delivery, POSTs it, and classifies the answer. It is built per sink, so the endpoint, the dataset, the transport posture and the credential reference all come off the sink row rather than out of the egress configuration.

One request per delivery. A delivery becomes exactly one ResourceMetrics or one ResourceLogs, with a single resource and a single scope. Every record in a batch shares the same attribution, so the attribution is the resource; the signal is the scope, because that is what tells an audit event from a log line on the shared logs endpoint. One resource per record would repeat the same three attributes for every point, and carrying the attribution as a record attribute would let a destination's resource-level queries miss it.

Resource attribute	Carries
plexsphere.domain.id	the originating tenant
plexsphere.project.id	the originating Project id
plexsphere.node.id	the originating Node id

An attribution field the delivery left empty contributes no attribute rather than an empty one. The keys are namespaced so they never collide with a semantic-convention key the destination already interprets.

Signal	Scope name	Path
metrics	plexsphere.metrics	<endpoint>/v1/metrics
logs	plexsphere.logs	<endpoint>/v1/logs
audit	plexsphere.audit	<endpoint>/v1/logs

The request URL is built from the endpoint's scheme and host, with the signal path appended, so nothing an endpoint carries past the host can reach the wire. It must be a base URL: an endpoint carrying a path, a query — including a bare trailing ? — a fragment or userinfo is refused at construction, because the adapter would otherwise POST to that path with the signal path appended after it — the vendor-published signal URL pasted whole reads as a 404, which the classifier calls terminal, so every batch would be destroyed silently. The destination is left without an exporter instead, which the registry logs.

An export follows no redirect. The destination is the address the sink states, which is the only one those checks judged, so a 3xx is handed back rather than chased. Following one would also hand the destination the Authorization header on the next hop: net/http decides that by comparing host names alone, so a 307 from https to the same host over http carries the bearer token in the clear, on every batch. The unfollowed 3xx is classified retryable, not terminal: the destination is misconfigured, but an operator correcting the endpoint is what repairs it, and a terminal verdict would ack every batch off the stream before anyone read the drop counter — the same reason the dial guard's refusal is retryable.

Three headers ride on the POST:

Header	Carries
Content-Type	application/x-protobuf, the binary OTLP/HTTP encoding
Dash0-Dataset	the sink's dataset, set only when the sink states one
Authorization	Bearer <token>, set only when the sink states a credential

Dash0-Dataset is one vendor's spelling of the dataset idea. A destination that does not read that header files the telemetry under its own default and reports no error, so a sink's settings.dataset is only meaningful for a destination that does. The header name is not configurable today.

A dataset that is not a legal HTTP field value is refused at construction, on the same grounds as the endpoint: net/http would refuse the request itself, and that refusal reads as a transport error — a batch redelivered until the retention horizon with retries_total climbing and no terminal outcome to alert on. A credential whose stored token carries such a byte is refused the same way, retryably, so rewriting the secret repairs it.

Metrics. Each record becomes one gauge data point: the value as a double and the record's RFC3339Nano timestamp, filed under one metric per distinct name — the name verbatim. Samples sharing a name share their metric, which is the shape an agent's flush window arrives in, so the name and its framing are written once rather than once per sample. OTLP names are free UTF-8, so the Prometheus identifier grammar has no authority here and the sanitization the Mimir transform applies stays where Prometheus is. The data point's attributes are the record's metric group plus its user labels in name order; a user label literally named group is dropped, so a caller cannot overwrite the group the sample was admitted under. A record that does not decode as an object, whose value is not a number, or whose timestamp does not parse is skipped, and its siblings still ship. A batch in which every record is skipped is a terminal rejection, so the Router counts it dropped rather than redelivering records no retry repairs.

Skips are reported as one warning per batch carrying the tally per reason, not one line per record. A batch holds up to 10 000 records, so a producer emitting a systematically malformed field would otherwise turn every export into 10 000 log lines at ingest rate — for free, since the batch is rejected terminally and acked. The per-record record_drops_total counter stays a built-in-transform metric: the Router hands a tenant sink its records raw, so the adapter's skips are visible in the log tally rather than on that counter.

Logs. Each record becomes one log record whose body is the message. The severity keyword travels as the severity text and maps onto the OTLP severity scale:

Severity keyword	OTLP severity number
debug	5
info	9
notice	10
warning	13
err	17
crit	18
alert	19
emerg	21

The roster is the closed one the ingest front door admits, so a keyword outside it only arrives through wire corruption; such a record keeps its raw text and gets severity number 0, which OTLP reads as unspecified. The observed timestamp is always set from the adapter's clock. The origin timestamp is the record's own, and is 0 (unset) when it does not parse, which is the fallback OTLP defines, so a bad clock on a node costs no telemetry. unit and hostname become attributes when the record carries them.

Audit. The body is the record's raw JSON as a string, so the byte-exact provenance the SIEM forwarder preserves survives here too, and source, action and outcome become attributes. An audit record carries severity number 0 and no severity text: an audit trail records what happened, not how bad it was, so inventing a level would put a judgement in the data the source never made.

The credential contract. A credentialed sink is served only over a transport that keeps the token confidential: the endpoint's scheme must be https and the sink's TLS posture must not switch verification off. The platform reads that token out of its own KV store under its own read authority, so shipping it over cleartext — or over a session that accepts any certificate — would hand a third-party API token to whoever is on the path, on every batch. A sink that must reach a destination with a private certificate states a CA bundle instead; that path stays open. A sink that states no credential is unaffected and may address an http endpoint.

A sink that states a credential has its bearer token read out of the KV store at export time. The payload must carry a token key holding a non-empty string; every other key is ignored, so an operator may store the token beside whatever else the destination's onboarding asked them to keep. The token is read on every export and never cached, so a rotation takes effect on the very next batch, with nothing to expire and no process to restart. Every failure of the read is retryable (an unreachable or sealed store, a path that holds no payload yet, a payload without the key, a value that is not a non-empty string), so the batch stays on the stream and the operator has the retention window to write the secret or unseal the store. No message carries the token value; each names what was wrong with it. The read and the POST share one per-export deadline, taken when the export begins, so a store that accepts a connection and then stalls cannot hold the sink's consume loop past the AckWait that redelivers the message and stack redeliveries behind a handler still waiting — and a slow store plus a slow destination still cost one timeout between them, not one each. That one budget is apportioned: the read runs first and may spend a third of it, so a degraded store costs the read its share and leaves the POST the remainder rather than an expired context — a healthy destination still delivers while the store is slow.

The dial-time guard. A tenant sink is delivered over a transport of its own: the destination's certificate is verified against the sink's own CA bundle, and every connection the transport opens is re-checked against the refused address set — the one the Sink aggregate states, covering loopback, unspecified, private, link-local, interface-local, the 100.64.0.0/10 mesh range, 192.0.0.0/24, 198.18.0.0/15 and fec0::/10. Both halves read the same predicate, so neither can drift into admitting what the other refuses.

The transport takes no proxy, not even a configured HTTP_PROXY. A proxied dial hands the guard the proxy's address while the proxy resolves and connects to the destination on the far side, which would disarm the rule for every sink at once; and an egress proxy on a private address would itself be refused, failing every delivery retryably with no terminal outcome. A deployment that needs proxied egress reaches its destinations through a network-level gateway instead. The Sink aggregate refuses an endpoint that names one of those numerically, but a host name says nothing at authoring time about what it resolves to at delivery time. The hook runs after the name is resolved and before the socket connects, on every attempt, so a name that resolves inward only some of the time is caught on the attempt it does. An address it cannot classify, one whose host is not a literal IP, is refused for the same reason.

A refused dial surfaces as an ordinary transport error, which the classifier counts retryable, so the batch stays on the stream. A terminal verdict was rejected: DNS is dynamic, and a resolver that briefly answers with a stale or poisoned record would then discard every batch in flight, leaving the operator with the data gone rather than waiting. The retry backoff already bounds the redial rate.

Classification. The adapter classifies through the same ClassifyResponse the built-in exporters use, so a transport error, a 429 and a 5xx leave the batch on the stream while any other non-2xx drops it. A 2xx is an accepted export, including one whose body carries an OTLP partial-success message — the spec forbids retrying one, since the rejected records are ones the destination will never take. That rejection is read off the body and logged with the signal, the count and the leading 256 bytes of the destination's own message, so records thrown away by an otherwise successful export are not counted as delivered without a trace. The message is truncated because the destination writes it and one line is logged per accepted export: a tenant pointing a sink at a receiver of their own would otherwise turn small batches into kilobyte-sized log lines at ingest rate. The body is drained either way so the connection can be reused.

The POST body is not compressed. Alternative considered: gzip, which this hop would benefit from most, since it is the one leg that leaves the platform network. Rejected for now because compression is not negotiated: a destination that does not decode it answers a non-429 4xx, which the classifier calls terminal, so the batch is destroyed — the adapter would be trading egress bytes for silent data loss at destinations it cannot probe in advance. A per-sink setting is where this belongs.

Syslog mapping

A syslog sink is a tenant-declared destination that receives RFC 5424 messages over TLS, and it is the second tenant protocol the platform ships an adapter for. The adapter lives in ../../../internal/observability/routing/syslog: it renders the delivery's records, frames them, and writes the frames to the destination. It is built per sink, so the endpoint and the transport posture come off the sink row rather than out of the egress configuration, and it is written with the standard library alone: no syslog client library enters the module for it.

Logs and audit events have a mapping; metrics do not. A metric sample is a number on a time series, and rendering it as a line of text would invent a reading of the data the destination never asked for. routes.AcceptedSignals keeps a syslog sink off the metrics stream, so no metrics consumer is provisioned for one; a metrics delivery that reached the adapter anyway is rejected terminally.

One buffer, one connection per delivery. Each record becomes one message, and each message is framed the way RFC 5425 states: the message's length in octets, one space, then the message. The frames are concatenated in record order into one buffer, and that buffer is written to a single TLS connection the export dials and closes. The count is a byte length rather than a rune count, because it is the number of octets the receiver reads off the stream before it looks for the next frame.

The connection is dialled per export rather than held open across exports. RFC 5425 has no application-level acknowledgment, so the only evidence a delivery produces is whether the write succeeded. On a held-open connection a peer that died between two flushes surfaces on the write of the next batch: that batch is classified failed and redelivered while the batch actually lost was acked long before, so the failure lands on the wrong data. The price of dialling per export is one TLS handshake per flush window, and a flush window is a batch rather than a record.

Facility and severity. PRI is the facility times eight plus the severity, and the two streams travel under different facilities:

Signal	Facility	Severity	PRI
logs	16 (local0)	the record's own keyword	crit renders <130>
audit	13 (log audit)	notice (5)	always <109>

A collector files and filters by facility before it reads anything else, and the two streams have different consumers: a log line belongs in the operator's log store, an audit event in whatever keeps the audit trail. One shared facility with MSGID telling the streams apart was rejected, because MSGID is a field a receiver has to be configured to read while facility routing is what every syslog receiver already does, so the shared facility would put the tenant's audit trail in the operator's log store by default.

A log record travels under the severity its own keyword names. The closed roster the ingest front door admits is syslog's own, and so is its ordering, so the mapping is the identity:

Severity keyword	Syslog severity code
emerg	0
alert	1
crit	2
err	3
warning	4
notice	5
info	6
debug	7

Every audit message carries severity notice. PRI is mandatory, so the "no severity" the OTLP mapping records for an audit event has no representation here, and notice is defined as a normal but significant condition, which is what an audit record is. info was rejected: it files the audit trail with ordinary chatter, and info is the first level a severity floor at the receiver discards.

The header fields. They are written in the order the RFC states, and a field with nothing to put in it carries the NILVALUE -:

Field	Carries
VERSION	1, the format's only version
TIMESTAMP	the record's origin timestamp, converted to UTC and rendered as RFC 3339 with at most six fractional digits
HOSTNAME	a log record's hostname; - on an audit message
APP-NAME	a log record's unit; an audit event's source
PROCID	-
MSGID	the signal token (logs, audit)

An origin timestamp that does not parse renders as - and the record is still delivered: a timestamp the producer got wrong is a reason to lose the stamp, not the line, which is the posture the Loki mapping and the OTLP one take on the same field.

HOSTNAME and APP-NAME travel under the RFC's grammar, which admits one to 255 bytes (HOSTNAME) or 48 bytes (APP-NAME) of printable US-ASCII with no space. A value outside it degrades to -: a unit name carrying a non-ASCII byte, a hostname longer than the field. Nothing is lost, because both mappings carry the same value as a structured-data parameter as well, whose grammar admits it.

PROCID names no process, because the batch is written by the platform's delivery leg rather than by a process on the host the records came from. An audit message names no host, because an audit event records what the platform did rather than what a host reported.

The structured-data element. A message carries one element, [plexsphere@32473 …]:

Parameter	Set on	Carries
domain	both	the originating tenant
project	both	the originating Project id
node	both	the originating Node id
unit	logs	the record's unit, when it states one
hostname	logs	the record's hostname, when it states one
source	audit	the event's source
action	audit	the event's action
outcome	audit	the event's outcome

A field the delivery or the record left empty contributes no parameter rather than an empty one, and a message left with no parameter at all carries - in place of the whole element. A ", a \ or a ] inside a value is escaped with a preceding backslash: those are the three characters RFC 5424 states a parameter value escapes. The value is walked once, so an escape the adapter writes is never escaped again.

The element is identified by a private SD-ID, which RFC 5424 requires to carry a private enterprise number, and the number is 32473. The platform holds no registered IANA private enterprise number, and 32473 is the one RFC 5612 reserves for documentation and examples, so the element is named without claiming an assignment nobody made. A registered SD-ID such as origin was rejected: the registered ids have fixed parameter sets and none of them carries tenant attribution, so the delivery's domain, project and node would travel as parameters the registered id does not define. The id is one constant, so a registered enterprise number replaces it in one edit.

MSG. A log message travels as the record's message; an audit event travels as its raw JSON verbatim, so the byte-exact provenance the SIEM forwarder preserves survives here too. MSG carries no leading byte-order mark. RFC 5424 states that a UTF-8 MSG SHOULD start with one and that a MSG without it is of unknown encoding, a classification no receiver refuses a message over. Emitting the mark was rejected because collectors treat MSG as opaque bytes: several render the three bytes at the head of every line, so the cost is a message that looks corrupted in the operator's log store and the benefit is a label nothing reads.

Skips and rejection. A record the mapping cannot read is skipped and its siblings still ship, for one of two reasons:

Skip reason	Meaning
undecodable	the record bytes do not decode as a log line or an audit event
unknown_severity	a log record's severity keyword is outside the roster PRI can carry

The roster is the closed one the ingest front door admits, so an unknown keyword only arrives through wire corruption, and PRI is mandatory: there is nothing to render it as, so the record is skipped rather than delivered under a guessed severity. A batch in which every record is skipped is a terminal rejection, so the Router counts it dropped rather than redelivering records no retry repairs.

Skips are reported as one warning per batch carrying the tally per reason, on the same grounds the OTLP adapter reports its own: a batch holds up to 10 000 records, and a producer emitting a systematically malformed field would otherwise turn every export into 10 000 log lines at ingest rate. They reach no counter, because the Router hands a tenant sink its records raw and record_drops_total stays a built-in-transform metric.

No credential, and the transport. A sink that states a credential is refused when the exporter is built, so the destination is left without an exporter and no consumer is provisioned for it. The transport is server-authenticated TLS: the destination proves itself with a certificate the sink's CA bundle chains to, and the platform proves nothing. The client-side convention a syslog collector speaks is mutual TLS, which needs a credential payload carrying a certificate and its private key, a shape the credential contract does not state. Reading the credential and sending it as a token, in a structured-data parameter or ahead of the first frame, was rejected: no collector reads either place, so the platform would put the tenant's secret on the wire with every batch for a receiver that ignores it, and the sink would look authenticated while being anonymous.

The connection floors at TLS 1.2. A stated CA bundle replaces the system trust store rather than extending it; an empty bundle keeps the system store; a bundle that yields no certificate refuses the sink at construction, where an operator can act on it, instead of looking like a transient handshake failure on every batch for as long as the row stands. The name the destination's certificate has to match is the host the endpoint names, which this adapter sets itself because it dials raw TCP and wraps the connection rather than handing a config to net/http. The sink aggregate lets an operator state that the certificate is not to be verified, and that posture is carried through as authored rather than second-guessed here.

Every resolved address is re-checked at dial time against the refused set the Sink aggregate states, on the same terms the OTLP adapter's guard applies it: the aggregate judges the endpoint as authored, this adapter judges the address it resolves to on every attempt. An address it cannot classify, one whose host is not a literal IP, is refused too.

Classification. There is no ClassifyResponse here. ClassifyResponse is HTTP-shaped and syslog over TLS has no status channel, so a failed dial, handshake, write or close carries no evidence that the next attempt would fail the same way: every transport failure is retryable, the batch stays on the stream, and the backoff redelivers it. The one terminal outcome the adapter produces is a batch in which no record could be encoded, a property of the records rather than of the destination. Classifying a dial failure as terminal was rejected, since it drops the batch on a transient network fault and the telemetry is gone before anyone reads the drop counter.

Error classification

Every export attempt is classified by ClassifyResponse into exactly one outcome:

Outcome	Condition	Consumer action
success	a 2xx	ack
ErrExportRetryable	a transport error, a 429, or any 5xx	nak with the backoff delay; the batch stays on the stream
ErrExportRejected	any other non-2xx (a non-429 4xx)	ack-and-drop (terminal)

Why a non-429 4xx is terminal. Per the Prometheus remote-write spec a 4xx means the receiver will never accept this payload, so a 4xx that is not 429 is permanent. Retrying it forever would wedge the consumer behind a poison batch it can never drain; treating every error as retryable was rejected for exactly that wedge — a single malformed batch would stall the whole consumer indefinitely. A 429 and a 5xx (rate limit, backend restart) are transient, and a transport error is by nature retryable, so all three are safe to redeliver.

The Router's terminal and retryable paths follow from that classification:

• an unparseable subject or an unparseable batch → terminal ack-and-drop (a wire-contract violation never parses on retry);
• an all-records-malformed Mimir transform (nil payload) → terminal ack-and-drop;
• no route naming this destination, and no default carrying the signal → terminal ack, counted skipped, decided before the parse;
• no record selected by the routes that do name it → terminal ack, counted skipped: the delivery is handled, and it is neither an export nor a loss;
• a failed route resolution → NakWithDelay(backoff), counted on resolution_failures_total; the delivery is undecided, so neither exporting nor dropping is defensible;
• ErrExportRejected → terminal ack-and-drop;
• ErrExportRetryable → NakWithDelay(backoff); the batch stays on the stream for a later delivery.

Observability and metric semantics

The application service emits seven Prometheus collectors in the plexsphere namespace, subsystem observability_routing:

Metric	Type	Labels	Buckets	Meaning
plexsphere_observability_routing_batches_total	CounterVec	sink, signal, outcome	—	Batches driven to a terminal disposition. outcome is the closed three-value set exported / dropped / skipped, where skipped means the batch was acked without an export because the Project's routes selected none of its records for this sink.
plexsphere_observability_routing_records_total	CounterVec	sink, signal	—	Records exported.
plexsphere_observability_routing_record_drops_total	CounterVec	sink, signal, reason	—	Records dropped during transformation. reason is the closed Mimir drop vocabulary (malformed_value / malformed_timestamp / undecodable).
plexsphere_observability_routing_timestamp_fallbacks_total	CounterVec	sink, signal	—	Records that fell back to the batch send time for their timestamp (the Loki path).
plexsphere_observability_routing_retries_total	CounterVec	sink, signal	—	Retryable export outcomes that scheduled a redelivery.
plexsphere_observability_routing_lag_seconds	HistogramVec	sink, signal	0.25, 1, 5, 15, 60, 300, 900, 3600	Age in seconds of telemetry at export time, clamped to ≥ 0.
plexsphere_observability_routing_resolution_failures_total	CounterVec	sink, signal	—	Deliveries whose route resolution failed. Like a retryable export this is not a terminal disposition: the delivery is redelivered, so batches_total stays untouched.

What the sink label carries. A built-in destination is labelled by its slug (mimir, loki, siem), so a dashboard keyed on sink="mimir" reads the same series it always did. A tenant destination is labelled by its sink id, not by its slug: a slug is unique per Domain only, so two Domains naming their destination central would collapse into one time series and hide each other's drops. The id is unbounded in principle and bounded in practice, because sinks are declared configuration objects an operator creates by hand rather than a per-request dimension: the series count tracks the number of destinations the platform delivers to.

The lag metric. Lag is the gap between the consumer clock at export time and the batch's sentAt. It is clamped to ≥ 0 so a producer clock running ahead of the consumer yields a 0 sample rather than a nonsensical negative age.

A retryable outcome does not increment batches_total. A retryable export is not a terminal disposition: it bumps retries_total and schedules a redelivery, leaving batches_total untouched. The batch only reaches a terminal batches_total outcome (exported, dropped or skipped) on a later delivery — so batches_total counts each batch's final disposition exactly once while retries_total counts the redeliveries along the way.

The no-node_id-label cardinality rule. No collector carries a node_id label. A Node id is unbounded — every agent that ever reports mints a new series — so a node_id label would multiply each collector's cardinality by the fleet size and eventually overwhelm the metrics backend. The surviving labels are bounded: signal and outcome are three-value closed sets, reason is a closed drop-cause vocabulary, and sink grows only with the destinations an operator declares. Per-Node attribution lives where high-cardinality attribution belongs — on the structured slog line and the audit trail — not on the metric labels. This mirrors the same rule the ingest bundle enforces.

Composition root

The production wiring is assembled in ../../../cmd/plexsphere/observability_routing_factory_prod.go. It is a pure backend consumer: it mounts no /v1 route and installs no NSK middleware, so its wiring carries only a boot reconcile with its readiness probe (gating /readyz on durable-consumer provisioning) and a Consume-loop launcher run under the same goroutine supervisor as the other reconcile sweeps.

Env knobs. Each read is strings.TrimSpace-d; an empty sink URL disables that Sink.

Env var	What it tunes	Default
PLEXSPHERE_OBS_MIMIR_URL	Grafana Mimir remote-write endpoint; empty disables the Mimir sink	(none)
PLEXSPHERE_OBS_LOKI_URL	Grafana Loki push endpoint; empty disables the Loki sink	(none)
PLEXSPHERE_OBS_SIEM_URL	audit SIEM ingest endpoint; empty disables the SIEM sink	(none)
PLEXSPHERE_OBS_SIEM_TOKEN	bearer the SIEM exporter attaches; only meaningful alongside the SIEM URL	(none)
PLEXSPHERE_OBS_NATS_URL	the shared JetStream buffer the routing consumers drain; threaded from the same source as the ingest front door by main.go (one source of truth for the buffer cluster); one half of the opt-in gate	(none)
PLEXSPHERE_DSN	the database the engine reads a Project's routes and the platform's declared sinks out of; threaded from the same source as every other surface by main.go; the other half of the opt-in gate	(none)
PLEXSPHERE_SECRETS_OPENBAO_*	the secret store a credentialed tenant sink's bearer token is read from; loaded once by the secrets surface and threaded onto this config by main.go (see the credential store)	(none)
PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS	waives the destination rule, so a tenant sink may address and be delivered to the platform's own network; only the literal true enables it (see the internal-destination opt-in)	(unset — the rule is in force)

The per-export HTTP round-trip timeout defaults to 10s — a code default applied via WithDefaults, not an env knob.

The internal-destination opt-in

PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS is a dev-stack setting. Leave it unset in any real installation: the platform dials a tenant-stated destination from inside its own network, and an endpoint that may name that network turns the delivery leg into a request forger against the cloud metadata service, the API server, or any in-cluster listener that acts on a request body.

It exists because the rule makes the local dev stack unable to demonstrate egress at all. Every collector a reader can reach from their own machine — a container, the LAN, the host the kind cluster runs on — resolves into a refused range, so a destination that receives anything cannot be built there. The local dev stack therefore states it, and the lesson Route logs to your own collector is what it is stated for.

One variable governs both halves of the destination rule, and both are read from it independently:

• the authoring half, in the sinks HTTP composition root, which passes it to the sinks application service. The service waives the rule on every aggregate it constructs, so plexctl sink create stops answering 422 sink_invalid for an internal endpoint;
• the delivery half, here, which builds the tenant exporters on a dialer without the internal-address guard.

Coupling them is deliberate: a deployment that relaxed one alone would either refuse the sink it can deliver to, or accept the sink it cannot.

Only the literal true (any casing, surrounding whitespace trimmed) enables it. 1, yes and a misspelling all leave the refusing posture in place, so a typo cannot silently disarm the rule.

The exporter adapters themselves carry no switch. The opt-in is acted on by the composition root handing them a dialer and a transport built without the guard, through the same seams a test uses. A guard that could switch itself off would be a security control on a knob production code carries, where a wiring mistake disarms it silently; an unguarded connection is instead something a reader can see being assembled.

Row reconstruction is not governed by the variable. The repository waives the rule on every row it hydrates, because a sink legitimately authored while the opt-in was on has to stay listable, routable and deletable after it is switched off. Its deliveries are refused at the dial all the same.

Boot-error behaviour. The opt-in gate is "PLEXSPHERE_OBS_NATS_URL and PLEXSPHERE_DSN are both set". Both are load-bearing for every delivery: the pipeline drains the JetStream buffer, and it resolves each batch against the Project's routes and the declared sinks in Postgres. Together they are also sufficient, because a Domain may declare a sink of its own at any moment and the registry builds the adapter for it, so no platform backend has to be configured for the pipeline to have work. The cases:

• both set → the factory is built, whether or not a built-in sink URL is present. With none, the built-in provider holds no exporters and the reconcile provisions nothing until the first tenant sink appears;
• a built-in sink URL set but PLEXSPHERE_OBS_NATS_URL empty → boot error ErrObservabilityRoutingNATSRequired (the operator named a destination but gave the pipeline no buffer to drain);
• a built-in sink URL set but PLEXSPHERE_DSN empty → boot error ErrObservabilityRoutingDSNRequired. The engine resolves every delivery against the Project's routes and the platform's declared sinks, both of which live in Postgres, so a routing pipeline without a database can decide nothing;
• no built-in sink URL, and the NATS URL or the DSN empty → the factory returns a nil factory and the routing pipeline boots fully inert, so an ingest-only deployment without a database keeps booting;
• a malformed sink URL, or a SIEM token set without a SIEM URL → boot error wrapping ErrInvalidConfig, naming the offending knob.

Two alternatives were rejected. An explicit enable flag: its only correct setting is derivable from the two URLs the pipeline cannot run without, and an operator who sets both but forgets the flag gets silence instead of telemetry. The previous "at least one built-in sink URL is set" gate: it leaves a deployment that routes only to tenant-declared destinations unable to route by construction, because with no Mimir, Loki or SIEM URL the factory returned nil and no consumer was ever provisioned for the tenant sinks the registry now serves.

Validation runs at build time, so a misconfigured operator sees the failure before /readyz lights green rather than as a runtime export failure on the hot path.

The credential store behind a credentialed sink

A tenant sink that states a credential needs a secret store to read its bearer token from. The routing factory builds one from the PLEXSPHERE_SECRETS_OPENBAO_* family the secrets surface already loads, and the mount and path are the sink's own: its CredentialRef carries the mount its Domain's material lives under and the path DeriveCredentialPath computed for it, never a value taken from configuration, so one Domain's sink cannot address another's material. A pinned version reads exactly that version; version 0 reads the latest. A syslog sink takes no credential: the adapter refuses a sink that states one when the exporter is built, so no read of the store ever happens for a syslog destination.

The OpenBao client is dialled eagerly, at boot, and a failure refuses the boot. That adds no fragility the deployment does not already carry, because the secrets surface dials the same backend from the same config and refuses to boot when it is unreachable. Dialling lazily on the first export was rejected: it moves a misconfiguration from boot time, where it stops the rollout, to delivery time, where the only trace is a warning log per skipped sink.

An unset address, or a config naming no auth strategy, is not a misconfiguration: the platform simply has no secret store. The source stays unwired, the OTLP factory refuses exactly the sinks that state a credential (the registry logs a warning for each and provisions no consumer for it), and credential-less sinks keep delivering.

The reconcile probe and the consume launcher. The reconcile runs the durable consumers of the enabled backends and the declared tenant sinks into line once at boot, and that boot pass failing refuses startup. The /readyz probe registered under the name observability-routing then reports the outcome of the most recent reconcile rather than running another one, so a NATS or database outage that prevents the consumers from being reconciled flips readiness to 503 within one reconcile interval rather than failing silently.

The probe reads a recorded outcome on purpose. /readyz is unauthenticated and polled every few seconds per replica, and a reconcile lists every declared sink, sweeps three streams for orphans and deletes durables — so a probe that drove one would turn readiness traffic into write traffic against shared infrastructure, and would make answering /readyz depend on Postgres. It is the same trade-off the built-in Sink seed records for its own probe.

A recorded outcome would otherwise never go stale, so the probe also bounds its own freshness: it refuses before any pass has completed, and refuses once the last completed pass is older than one pass budget plus two intervals (90 seconds at the defaults). That is what makes a green answer mean "the invariant is still being re-established" rather than "it was, once". Each pass gets a deadline of its own for the same reason — a saturated pool or a blackholed connection inside a pass would otherwise hold the reconcile mutex, and reconciliation, forever.

The freshness bound covers the reconcile, not the consume loops: a loop the server ended is caught by the pass itself, which drops it and binds a new one. The bound is measured on a monotonic reading rather than the wall clock, so a chronyd step or a VM resumed from a snapshot cannot refuse a healthy probe on every replica at once — or, stepping backwards, disable the bound for the length of the offset.

The ConsumeStart launcher runs one JetStream Consume loop per durable under the goroutine supervisor, dispatching each delivery to the Router, keeps reconciling on the 30-second interval, and returns ctx.Err() on cancellation after stopping every loop. No reconcile failure takes that loop down — not even its own first pass. Returning would stop every loop that is draining fine because one listing failed, and, since the probe no longer drives a reconcile and the supervisor spawns the launcher exactly once, it would end reconciliation for the process lifetime: no consumer for a newly declared sink, no sweep for a deleted one, and a /readyz pinned at 503 with no path back. The failure is logged and surfaces through /readyz; the next tick retries.

Cross-references

• ./ingest.md — the ingest front door whose per-Domain JetStream buffer this context drains; it owns the Batch model, the three-stream topology, the subject / header wire contract this pipeline reads back off the stream, and the 24h retention horizon.
• ./routes.md — the Route aggregate this engine resolves per delivery: the replace-per-signal rule, the two predicates, and the resolution that narrows a route's targets to the usable ones.
• ./sinks.md — the Sink aggregate a route targets and the three platform backends this engine binds from its egress configuration.
• ../index.md — the bounded-contexts landing page.
• ../../../internal/observability/routing — the bounded-context root that pins the ubiquitous language.
• ../../../cmd/plexsphere/observability_routing_factory_prod.go — the composition root that validates the egress config, provisions the durable consumers, builds the exporters, and launches the consume loops.
• ../../reference/api/observability.md and ../../../api/openapi/plexsphere-v1.yaml — the upstream producer's HTTP surface: the ingest operations that feed the buffer this context drains. Routing itself exposes no /v1 HTTP API — it is a backend consumer with no inbound HTTP surface.