Appearance
Telemetry routes
This document is the bounded-context reference for the Route aggregate: one Project's statement of where its incoming telemetry goes. The domain root that pins the ubiquitous language is ../../../internal/observability/routes.
A Project that states no route still gets its telemetry delivered, because the platform's built-in destinations take it. A route is what an operator writes to send a stream somewhere else. The context owns the aggregate and its value objects, the three domain events, the persistence, the read model the delivery engine consumes, and the application service that authors a route.
The name sits close to Observability Routing, which names something else. That context is the egress machinery: it drains the buffered batch, transforms it and hands it to an exporter. This context is the per-Project configuration that machinery reads.
Ubiquitous language
The terms below travel verbatim across the domain root, the events, the application service, the persistence adapter and the delivery engine.
| Term | Definition | Code anchor |
|---|---|---|
| Route | The aggregate root: one Project's rule for one signal, holding the predicate that selects records and the sinks that receive them. Identified by a UUIDv7. | ../../../internal/observability/routes/route.go (Route) |
| Signal | The stream a route governs, one of metrics, logs and audit. The three are independent and a route governs exactly one. | ../../../internal/observability/ingest/signal.go (Signal) |
| Severity floor | The least severe log keyword a route still carries. A floor of warning carries warning and everything more severe than it. A logs route may state one; no other signal may. | ../../../internal/observability/routes/types.go (Severity) |
| Name prefix | The leading substring of a metric series name a route selects on. A metrics route may state one; no other signal may. | ../../../internal/observability/routes/route.go (validatePredicate) |
| Target | One sink a route delivers a matched record to. Every route names at least one. | ../../../internal/observability/routes/route.go (validateTargets) |
| Resolved route | The flattened read model the delivery engine consumes: one route with its targets already narrowed to the ones the Project may use. | ../../../internal/observability/routes/types.go (ResolvedRoute) |
Aggregate invariants
NewRoute and HydrateRoute share one builder, so every rule below holds for an operator-authored route and for a row read back out of the database. HydrateRoute adds three: a persisted row states its own id, creation and update timestamps, and a zero in any of them is a corrupt row rather than a value to default.
- Owner. A route names the Project whose telemetry it selects.
- Signal. One of
metrics,logsandaudit, and exactly one. - Predicate per signal. A severity floor is stated on a logs route only and names a member of the eight-keyword roster (
emerg,alert,crit,err,warning,notice,info,debug). A name prefix is stated on a metrics route only and is at most 256 runes. A predicate stated on another signal is refused rather than carried along and ignored at delivery time. - Targets. At least one sink and at most sixteen, no zero id, and no sink named twice. A route with no target would deliver nowhere while still superseding the default for its signal, a zero id names no sink, and a repeated id would hand the same record to the same destination twice. The ceiling bounds the other factor of the delivery cost the per-Project route limit bounds: a batch costs routes x targets x records, so capping only the route count leaves the per-batch fan-out open. The accessor returns a copy, so a caller cannot reach into the aggregate through the slice it passed in.
Every violation wraps the single ErrInvalidRoute sentinel and names the offending field, so a caller branches on one sentinel with errors.Is while the message still points at the field the writer has to fix. The write path adds the repository's ErrRouteNotFound and ErrRouteConflict. The domain sentinel carries no HTTP status, which stays a transport-layer concern.
A route replaces the default for its signal
A Project holding at least one route for a signal gets exactly its own routes for that signal. The built-in default applies only to the signals the Project left unrouted, so the three signals are decided independently: a Project with a logs route keeps its logs out of Loki and out of the SIEM entirely, while its metrics still reach Mimir and its audit records still reach both platform backends.
Adding to the default was the alternative, and it cannot express what an operator asks for when they route a stream elsewhere. A route that only widened delivery would leave every record in the platform backends as well, and there would be no way to move a stream, only to copy it.
Two corollaries follow, and both fail closed.
A route whose targets are all unusable goes dark. Resolution narrows a route's targets to the ones the Project may currently use, and that set may come out empty, for instance when the only tenant sink it named lost its enablement. The route still supersedes the default, so its slice of the batch is delivered nowhere. Falling back would resume delivery to the platform backends at the exact moment the operator revoked the only destination they named, which turns a withdrawal into a widening.
A record whose predicate field does not read does not match. A log record whose body does not decode, or whose severity keyword is outside the roster, leaves the field empty, and no floor and no non-empty prefix accepts an empty value. Letting such a record through would put exactly the records the floor exists to hold back past the floor.
An audit route moves a compliance destination
The replace rule applies to audit exactly as it does to metrics and logs: a Project that states an audit route stops sending that Project's node audit telemetry to the platform's own SIEM and Loki, and sends it only where the route says.
The audit signal is nonetheless different in kind. The SIEM is configured once per platform, its two durables are deliberately unfiltered across Domains, and it is the platform operator's compliance destination. So whoever may author an audit route can move a Project's node audit telemetry off that destination, silently and from the next resolver TTL onward.
Teeing audit — delivering to the routed targets in addition to the platform backends — was the alternative, and it was rejected because it makes one signal disobey the rule the other two follow, and because a Project that has been told to ship its audit records to its own collector gains no way to stop the duplicate copy.
The consequence is a permission consequence, and it is stated here so the transport tier that lands the route surfaces inherits it rather than guesses: the permission to author an audit route is a platform-level one and must not be tenant-grantable. Authoring a metrics or logs route is an ordinary per-Project operation; authoring an audit route moves a compliance destination and belongs with the operator who configured that destination. Nothing in this context enforces the split — the service carries no authorizer at all — which is exactly why it is written down before the surface that will enforce it exists.
The platform's own audit chain is unaffected. What a route can move is node-originated audit telemetry. The hash-chained plexsphere.audit_log_entry the control plane writes for its own mutations is a different record in a different context and no route touches it.
The two predicates
A route states at most one predicate, and each predicate reads one field of the record:
| Predicate | Signal | Field read | Matches |
|---|---|---|---|
| Severity floor | logs | severity | The record is at least as severe as the floor, under the syslog ordering where emerg is most severe and debug least. |
| Name prefix | metrics | name | The series name starts with the prefix. |
An audit route carries neither, so every audit record the Project produces matches it, and so does a logs or metrics route that states no predicate.
Both predicates rest on what ingest admission already pins on the way in. A log record carries a severity from the closed syslog roster, and a batch carrying any other keyword is refused at the front door; a metric sample carries a non-empty name. So the roster a floor orders is the roster the pipeline admits, and a prefix always has a name to compare against. A value arriving at the predicate outside those bounds is wire corruption rather than a legitimate record the operator did not anticipate, which is why both predicates refuse it instead of passing it on.
The severity roster is this context's own copy of the set the ingest parser admits. That set is unexported, so the two layers agree by contract rather than by a shared declaration: a change to one roster is a change to both.
Which signals a sink type receives
A destination takes the records it can represent, and no route may state otherwise:
| Sink type | Receives |
|---|---|
otlp | metrics, logs, audit |
syslog | logs, audit |
loki | logs, audit |
siem | logs, audit |
mimir | metrics |
syslog, loki and siem take records that carry a severity and a message, which is logs and audit; mimir stores metric samples only; an otlp endpoint speaks all three. An unknown type receives nothing, so a caller holding a type it does not know delivers nowhere rather than everywhere.
One capability table, exposed as AcceptedSignals, answers this question for every reader. Route validation refuses a route whose signal a target cannot receive, the engine reads it to decide which built-in destination carries an unrouted signal, and the consumer lifecycle expands a tenant sink onto one durable consumer per signal it receives. Three separate tables would give the three readers room to drift apart.
Resolution and the two staleness windows
The delivery engine reads routes through one statement, ResolveProject, which returns every route of one Project with its targets already narrowed to the ones the Project may use: the built-in sinks, which are reachable by every Project without a grant, and the tenant sinks the Project holds an approved enablement for. A route whose targets have all become unusable comes back with an empty target set rather than disappearing from the result, which is what makes the fail-closed corollary above reachable.
The narrowing runs in SQL, in the same statement that reads the routes, so the answer comes from one snapshot of the database. Resolving the routes first and then asking the sinks and enablement repositories which targets survive would cost one round trip per route on the delivery path and would compose an answer out of three reads that no single state of the database ever had. Because the predicate runs on every read, a grant revoked after a route was written invalidates no route: the target simply stops being resolved.
Two windows bound how long the running engine can act on a state the operator has already changed, and both are 30 seconds by default:
- The resolver cache. Every delivery resolves the routes of its Project, so the engine keeps each Project's resolution for one TTL before it re-queries. A route the operator adds, retargets or deletes reaches delivery within one TTL. A source failure propagates instead of serving the expired entry: the delivery is undecided, so the engine redelivers it rather than shipping by a route set that may already be revoked.
- The reconcile interval. The consumer lifecycle recomputes its desired consumer set on that interval. It bounds how long a sink an operator just declared waits for its durable consumer, and how long a deleted sink's durable survives.
Neither window is an operator knob. Both are constructor options the composition root leaves at their defaults, and tests set them to drive the behaviour without waiting.
Persistence
Two tables back the context: ../../../internal/platform/db/migrations/0085_routes.sql.
plexsphere.routes holds one row per route: the owning Project, the signal, and the two nullable predicate columns. A NULL in either is "no predicate on this axis". The three signal CHECK literals are exactly the ingest signal constants and the eight severity_floor literals are exactly the aggregate's roster, so the storage CHECKs and the closed sets cannot drift. Two further CHECKs state the storage half of the predicate-per-signal invariant, so a direct UPDATE cannot store a row that no longer hydrates. routes_project_idx backs the read the delivery engine runs on every batch.
plexsphere.route_sinks holds the target set, one row per route and sink pair. Its primary key over the pair is the set semantics themselves, the same rule the aggregate enforces on its target slice. route_sinks_sink_idx backs the referential check a sink deletion runs, which the composite primary key cannot serve because it leads with the route.
The two parent references take opposite postures. project_id cascades: a Project deletion is the canonical tear-down of derived state, and a route is meaningless without the Project whose telemetry it selects. The cascade leaves no graph litter behind, because a route writes no ReBAC tuple in the first place. sink_id restricts, so deleting a sink a route still targets is refused with a foreign-key violation. A cascade here would be a write path that appends no outbox row: it would silently shrink every route that named the sink, and a route whose last target vanished reads as live while delivering nothing. The refusal gives the operator a structural answer instead of telemetry that stops arriving without a trace.
Each of the three mutations runs several statements, the route row, its target set and the outbox row, inside one transaction, so no consumer reads an event for a write that rolled back and no reader sees a route whose targets landed only in part. The down arm drops both tables in reverse-reference order: they hold routing configuration only, and the decision trail lives in the outbox and the audit chain, which the drop does not touch.
Domain events
Three events record the row changes: obsroutes.RouteCreated, obsroutes.RouteUpdated and obsroutes.RouteDeleted. Each carries the route identity together with the denormalised Project, signal and target set, plus the acting principal, so a consumer indexes a delivery path without a follow-up join. The deletion carries the same field set as the other two, because a route holds no destination address and no credential coordinate: its attributes are a signal keyword and a list of sink references the reader is already entitled to see.
All three map to no ReBAC tuple. A route grants nothing; it selects among the sinks the enablement layer has already authorised, so nothing follows from a route write for the graph. The arms are registered in ../../../internal/authz/sync/mapping_observability.go as explicit no-ops rather than left out, so an unmapped event type stays an error the sync arm reports instead of a silent drop.
Authoring a route
RouteService is the entry point through which a caller authors a Project's routes. It builds the aggregate, checks the three facts the aggregate cannot see for itself, persists the result and records one audit row per mutation. Create, Update and Delete mutate; GetByID and ListByProject read and record nothing.
The three checks run in order of specificity, so a route naming two broken targets is refused for the more fundamental breakage rather than for whichever target comes first in the set:
- The sink exists. A target the sink reader cannot resolve propagates the sinks context's not-found sentinel.
- The Project may use it. A tenant sink the Project holds no approved enablement for is refused with
ErrSinkNotUsable. Built-in sinks are skipped, because they are reachable without a grant and are registered as no enablement, so probing for one would refuse every route to the platform's own backends. - The type receives the signal. A target whose type cannot receive the route's signal (a logs route to the platform's metric backend, for one) is refused with
ErrSignalNotAccepted.
Create additionally holds the Project to a ceiling of 100 routes, refusing past it with ErrRouteLimitReached. The count is not an authoring concern but a delivery one: the engine walks a Project's routes for every buffered batch a consumer bound to its Domain receives, and a batch carries up to ten thousand records, so an unbounded route set turns into a per-batch cost every other Project on those consumers pays. A hundred is far above what an operator hand-authoring destinations for three signals reaches. The check counts the Project's routes on the way in, so two authors racing at the ceiling can overshoot it by the number of racers — it bounds the set, it does not serialise writes.
The second check has no delivery-time counterpart, and the absence is deliberate: usability is enforced on every read by the resolution described above. What the authoring probe buys is that an operator learns at the form they wrote, rather than from telemetry that quietly never arrives.
The service carries no authorizer and runs no permission check of its own, following the sibling sinks service: authorization is gated at the transport tier before the call arrives. A route only selects among destinations the enablement layer has already authorised, and every tenant target it names is checked against that layer's answer, so a route can never widen the reach a grant established. What is left to gate is who may author a Project's routes, an ordinary per-operation permission on the owning Project.
Each mutation emits one names-only audit row under routes.create, routes.update or routes.delete. The row names the owning Project rather than the route, so an operator asking what happened to a Project's telemetry configuration reads one object and a delete row still points at something that outlives the route it announces. Its caveat context is the list of field names the row is about (route_id, project_id, signal, sink_ids) and never a value of any of them. A deletion reads the route before removing it, so the row's object is bound to the route that was actually deleted rather than to a Project the caller supplied.
An update reads the stored route first for the same reason, and refuses with ErrRouteProjectMismatch when the stated post-image names a different Project than the stored route belongs to. The persisted project_id is immutable — the update statement addresses the row by id alone and never writes that column — so a post-image carrying another Project would be written onto the owner's row while the usability check, the announced event and the audit row were all computed against the caller's: the owner's audit trail would show no mutation at all. The Project a route belongs to is read from the route, never taken on the caller's word.
Operator surfaces
Five /v1 operations carry the aggregate: CreateTelemetryRoute and ListTelemetryRoutes under /v1/projects/{id}/telemetry-routes, and GetTelemetryRoute, UpdateTelemetryRoute and DeleteTelemetryRoute under /v1/telemetry-routes/{id}. Their gates and the closed Problem.code taxonomy are in ../../reference/api/sinks.md.
The CLI family is plexctl route. The Console carries routes on the Project detail's Telemetry tab, which composes the target list from the Project's approved enablements plus the built-in sinks, so a target the server would refuse is never offered.
The delivery engine does not go through the application service. It reads the repository's resolution directly, so a route is honoured on the delivery path regardless of which surface wrote it.
A route targeting an otlp sink delivers. The production exporter registry puts the tenant protocol adapters in front of the three platform backends, and the platform ships the OTLP/HTTP adapter, so a Project's route to a Domain's own OTLP destination is stored, resolved and shipped. The routing page carries the wire mapping.
A route targeting a syslog sink delivers too. The platform ships the RFC 5424 adapter beside the OTLP one, so a Project's logs and audit records reach a Domain's own collector as octet-counted messages over one TLS connection per delivery. A syslog sink receives logs and audit only, which is what the table above states, and it takes no credential: the destination is authenticated by its certificate and the platform authenticates itself to it with nothing. The routing page carries the wire mapping.
Cross-references
./routing.md— the egress engine that resolves these routes per delivery, provisions the per-sink consumers and exports each batch../sinks.md— the Sink aggregate a route targets, its tenant and built-in shapes, and the three platform backends../sink-enablement.md— the grant that makes a Domain's tenant sink usable inside one Project, which is what resolution narrows a route's targets against../ingest.md— the front door that admits the records a predicate reads and buffers the batches the engine drains.../index.md— the bounded-contexts landing page.../../../internal/observability/routes— the domain root that pins the ubiquitous language.