Telemetry sinks

This document is the bounded-context reference for the telemetry Sink aggregate: the named destinations a Domain's routing layer may deliver to. The domain root that pins the ubiquitous language is ../../../internal/observability/sinks.

A sink is either tenant-authored or built in, and a route treats the two the same way: a named destination it may target. The context owns the aggregate and its value objects, the persistence, the boot seed that asserts the platform's own backends, and the three domain events the authz sync consumes.

Operator surfaces

Six /v1 operations carry the aggregate: CreateSink and ListDomainSinks under /v1/domains/{id}/sinks, ListBuiltInSinks under /v1/sinks/built-in, and GetSink, UpdateSink and DeleteSink under /v1/sinks/{id}. Their gates, the reference-only credential contract and the closed Problem.code taxonomy are in ../../reference/api/sinks.md.

The CLI family is plexctl sink. The Console carries the Domain-scoped /sinks page, which lists a Domain's tenant sinks beside the built-in roster and drives the same six operations. A sink is granted to a Project through the sink enablement lifecycle, whose requests are decided on the approvals queue.

Ubiquitous language

The terms below travel verbatim across the domain root, the application service, the seed, the persistence adapter and the authz sync arm.

Term	Definition	Code anchor
Sink	The aggregate root: one named telemetry destination, addressed by a kebab-case slug within its scope and identified by a UUIDv7. It carries a display name, a type, an optional endpoint, a TLS posture, a credential reference and per-type settings.	../../../internal/observability/sinks/sink.go (Sink)
Tenant sink	A sink one Domain owns. It names the owning Domain, speaks one of the two tenant-declarable forwarding protocols (otlp, syslog), and delivers to an endpoint the tenant operates.	../../../internal/observability/sinks/sink.go (checkTenantShape)
Built-in sink	A platform-provided sink. It carries no Domain, no endpoint, no credential and no TLS posture, names one of the platform's own backends (mimir, loki, siem), and is addressed by the slug that equals its type.	../../../internal/observability/sinks/sink.go (checkBuiltInShape)
Credential reference	The KV-v2 coordinates of a tenant sink's connection material: the mount, the path under it, and the version to read (0 means the latest). It holds the coordinates only; the material stays in the KV store and never enters this context. The zero value states that the destination needs no credential.	../../../internal/observability/sinks/types.go (CredentialRef)

Aggregate invariants

NewSink and HydrateSink share one builder, so every rule below holds for an operator-authored sink and for a row read back out of the database. HydrateSink adds three: a persisted row states its own id, creation and update timestamps, and a zero in any of them is a corrupt row rather than a value to default.

• Display name. Non-empty once surrounding whitespace is disregarded, at most 256 runes. The name is stored exactly as supplied: trimming decides whether it is empty, it never rewrites the value.
• Slug. Lowercase kebab-case matching ^[a-z0-9]+(-[a-z0-9]+)*$, at most 63 characters so it fits inside a single DNS label. Leading and trailing whitespace is refused, not trimmed. The pattern is kept byte-identical to the CHECK on plexsphere.sinks.slug, so the domain layer and the SQL layer agree on what a valid slug is.
• Tenant shape. A tenant sink names its owning Domain, its type is otlp or syslog, and it carries an endpoint. A zero credential reference stays legal: a destination that needs no credential is a destination all the same.
• Built-in shape. A built-in sink carries no Domain, no endpoint, no credential and no TLS posture, its type is one of mimir, loki, siem, and its slug equals its type. sinks_shape_chk states the same exclusivity, but the implication runs one way only: a shape the CHECK refuses never reaches the SQL layer, while a shape the CHECK admits is not necessarily one the aggregate admits. A CHECK cannot express a URL or host-and-port grammar, so an endpoint written by anything other than the aggregate can satisfy the constraint and still fail hydration.
• Endpoint per type. An otlp destination is addressed by an http or https URL with a host, at most 2048 characters. A syslog destination is addressed by a host and port pair whose port lies between 1 and 65535.
• Destination. An endpoint whose host is a literal IP inside the platform's own network is refused: loopback, the unspecified address, link-local (which carries the cloud metadata service), the private ranges a cluster's pod and service networks live in, the carrier-grade NAT range 100.64.0.0/10 that the platform's own WireGuard mesh addresses every node and control-plane listener on, the non-routable 192.0.0.0/24 and 198.18.0.0/15, and the deprecated IPv6 site-local fec0::/10. A tenant authors the endpoint and the platform dials it from inside the cluster, so without the rule the delivery leg becomes a request forger pointed at the platform's own control plane. The rule covers literal addresses only: a host name that resolves into one of those ranges is a question for the dial, not for a constructor that runs on every row read back, so the delivery leg re-checks the address it actually connects to. The rule yields to one thing only: a deployment that states PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS, which the local dev stack does and a real installation does not (see the internal-destination opt-in). The waiver is granted per construction by the application service and is never stored, so it cannot outlive the posture that granted it; row reconstruction waives it unconditionally, so a sink authored under the opt-in stays readable after the opt-in is gone.
• TLS CA bundle. An empty bundle means the system trust store. A stated bundle is at most 128 KiB, decodes as PEM and holds at least one CERTIFICATE block, so a truncated paste or a private key pasted into the wrong field is refused where it was authored rather than at the first delivery attempt. Every block is walked because a bundle legitimately carries several. The skip-verify flag is a posture an operator may state and carries no rule. The bound matters because validation stops at the first certificate: without it the rest of an oversized bundle would be stored unparsed and re-walked on every read of the Domain's sinks.
• Dataset. Settings.Dataset names the logical stream an otlp destination files telemetry under, bounded at 256 runes. It travels as the Dash0-Dataset request header, which is one vendor's spelling of the idea: a destination that does not read that header files the telemetry under its own default and reports no error, so the setting is only meaningful for a destination that does. The value must be a legal HTTP field value; the delivery leg refuses a sink whose dataset is not, rather than wedging on a request net/http will not send. Every other type carries empty settings: a dataset stated elsewhere is refused instead of being carried along and ignored at delivery time.
• Credential coordinates. Mount and path are non-empty and free of surrounding whitespace, and the version is not negative. Any /-delimited .. segment in either is refused, so a poisoned row cannot walk a KV read outside the intended mount prefix — both are guarded because a KV-v2 read composes them as <mount>/data/<path>, and a traversal in the mount leaves the KV space altogether.
• Credential scope. A tenant sink's KV path is not stated freely: it must be DeriveCredentialPath(domainID, sinkID), the path the aggregate computes from the owning Domain and the sink id. The platform, not the tenant, holds the KV read authority the delivery leg dereferences the reference with, so a freely stated path would make the platform a confused deputy — one Domain could name another Domain's secret, pair it with an endpoint it controls, and have the material shipped there. A writer authoring a credentialed sink therefore mints the id first, derives the path, writes the material at it, and passes both to NewSink. The mount stays caller-supplied: it names the operator's KV mount, which the domain layer does not know.

Every violation wraps the single ErrInvalidSink sentinel and names the offending field, so a caller branches on one sentinel with errors.Is while the message still points at the field the writer has to fix. The write path adds ErrSinkNotFound, ErrSinkSlugTaken and ErrSinkConflict; the domain sentinels carry no HTTP status, which stays a transport-layer concern.

One rule the aggregate cannot state lives beside it as a package constant, because it is about a sink's SIBLINGS rather than about the sink: a Domain holds at most MaxSinksPerDomain tenant sinks, and a create past the ceiling is refused with ErrSinkLimitReached. The reader pays for the count — the Domain listing renders the whole set in one unpaginated response, each row can carry a 128 KiB CA bundle, and every route picks its targets from the same roster — so the number cannot be left to the author.

The ceiling is decided in the persistence adapter, not in the application service, and that placement is the rule rather than an implementation detail. The adapter counts and inserts inside one transaction under a per-Domain advisory lock, so authors racing at the ceiling serialise and exactly one of them takes the last slot. A count taken in the service would sit outside the create transaction: every racer would read the same pre-commit count, every racer would pass, and the number of racers is the caller's to choose — which is no ceiling at all. The count itself is a SELECT count(*), not a listing: the rule needs one integer, and reading a hundred rows of up to 128 KiB each to take their length would make the refusal cost more than the write it declines.

Built-in sinks and the boot seed

A fresh install carries three platform-provided sinks:

Slug	Type	Display name	Backend
mimir	mimir	Grafana Mimir	The platform's metric backend.
loki	loki	Grafana Loki	The platform's log backend.
siem	siem	Audit SIEM	The platform's audit-event backend.

BuiltInSinks builds all three through NewSink, so the built-in shape is asserted in the seed rather than at the first write. ReconcileBuiltIns upserts each one through the repository's UpsertBuiltIn path. The upsert is idempotent: a later pass finds each row by slug and keeps the id the first pass minted, so a reference taken to a built-in sink stays valid across restarts. It appends no outbox row, so a restart costs the event stream nothing.

The seed is registered in ../../../cmd/plexsphere/app.go before the listener binds, so /readyz reflects the boot-time invariant from the first tick. It runs once, at boot; a failure there refuses startup. The standing observability-sinks-seeds probe then runs VerifyBuiltIns, a read-only listing that names any declared backend the table no longer holds.

The split matters: /readyz is unauthenticated and polled every few seconds per replica, so a probe that re-seeded would turn request volume into three row rewrites per tick against a three-row table, record the time of the last probe in updated_at instead of the time of the last change, and make readiness depend on Postgres accepting writes — which a replica attached to a read-only standby cannot do while still serving every read. The drift the probe exists to catch is visible to the read just as well. The probe is boot-latched, so it gates readiness until its first success and degrades afterwards rather than evicting every warmed-up replica on one Postgres wobble.

A built-in sink is also undeletable: DeleteSink carries an AND NOT built_in predicate and the repository reports ErrSinkConflict for the refusal. Without it a caller holding a built-in id would remove a backend every Domain's routes may target, and the next boot would re-insert the slug under a brand-new id — a healthy-looking seed with every prior reference broken.

A nil ObservabilitySinksFactory keeps the seed inert, which is the opt-out for a binary running without a database.

All three rows exist whether or not the routing environment configures an endpoint for them. A built-in row records that the destination exists as something a route may target; whether the platform is currently configured to reach it stays a property of the routing configuration, which reads its own MimirURL, LokiURL and SIEMURL knobs and reports the enabled set. Seeding only the configured backends would delete a tenant's chosen destination out from under it the moment an operator unset a URL.

The built-ins are persisted rows rather than a virtual special case for two reasons. A route reads one destination list, so a built-in sink needs the same identity and the same listing path as a tenant sink: one UUID, one row, one repository port. A Route aggregate that later references a sink then carries a single foreign key into one table, instead of a discriminator plus a union over a real table and a synthetic set.

The sink definition

The sink object type lives in ../../../schema/authz.zed:

definition sink {
  relation parent: domain

  relation owner:    user | serviceaccount | group#member
  relation assigner: user | serviceaccount | group#member
  relation uses:     project | project#operator

  permission manage  = owner + parent->manage
  permission assign  = owner + assigner
  permission use     = uses + owner + assigner
  permission observe = owner + assigner + parent->manage + uses->read
}

A sink hangs off its Domain through parent, so manage and observe resolve for a Domain admin without a per-object grant. owner and assigner admit serviceaccount because a Domain-admin group reaches a service account through group#member. A relation that refused the subject type would make SpiceDB reject the whole two-tuple batch the create arm writes, and the committed row would be left with no tuples at all. use is a plain union rather than the (uses & parent->use) intersection cloudcredential#use carries: domain declares no use permission for the arrow to reach, so the intersection would deny every granted Project. The Domain-side control is structural instead, because only a Domain-parented sink exists to be granted at all. The rationale for each of the three choices sits next to the definition in the schema file.

Event-to-tuple mapping

internal/authz/sync.MapEvent translates a discriminated outbox row into the relationships to write and the delete filters to apply. The sink arms are registered in ../../../internal/authz/sync/mapping_observability.go, and this table mirrors the truth there.

Sink event	Write	Delete
obssinks.SinkCreated	sink:<sink_id> <-[parent]- domain:<domain_id>, plus sink:<sink_id> <-[owner]- <created_by>	—
obssinks.SinkUpdated	— (attribute-only change, no graph effect)	—
obssinks.SinkDeleted	—	wholesale (sink:<sink_id>), every tuple under the sink

Three properties are worth reading aloud:

• SinkCreated seeds two tuples. The parent edge lights up manage = owner + parent->manage and the parent->manage operand of observe, so a sink whose creator leaves the Domain stays manageable. The owner edge gives the creating principal per-object manage, assign and observe. created_by is the ReBAC subject the repository denormalised onto the payload, so it already carries its user: or serviceaccount: prefix and is used verbatim — the event constructor refuses any other shape, because the two tuples go into one atomic batch and a subject SpiceDB rejects would fail both, leaving the already-committed row with no tuples at all.
• SinkDeleted purges wholesale. The filter is keyed on (sink, <id>) only: the aggregate is gone, so the Domain parent edge, the creator's owner grant and every uses tuple go with it. The narrow per-subject filter the assignment-revoked arms use would be wrong here, as no consuming Project may keep an edge into an object that no longer exists.
• uses is written by the grant lifecycle. The relation carries the Projects a tenant sink has been enabled for, and the sink-enablement arms are what move it: a granted or an approved enablement writes sink:<sink_id> <-[uses]- project:<project_id>, and a revoked one deletes that Project's edge narrowly, leaving every sibling Project's grant in place.

Only the tenant shape reaches these arms. A built-in sink is seeded through UpsertBuiltIn, which emits no event, so the three platform backends carry no tuples at all.

Persistence

One table holds both shapes: ../../../internal/platform/db/migrations/0083_sinks.sql. A route targets a sink without caring which kind it is, so a split into builtin_sinks and tenant_sinks would force every route reference to carry a discriminator and every read to union two tables. sinks_shape_chk expresses the same exclusivity in one place and keeps the foreign key a route needs single-valued. sinks_credential_chk keeps the KV triple internally consistent: mount and path are set together, and a version needs a path. endpoint and tls_ca_pem carry the same length bounds the aggregate does, because every read of a Domain's sinks loads both in full.

domain_id references plexsphere.domains ON DELETE RESTRICT and is NULL exactly for a built-in sink, so a Domain delete refuses while a sink still references it. A cascade would be the one write path to this table that appends no outbox row: Postgres would prune the rows inside the tenancy delete transaction, no SinkDeleted event would be written, and every parent edge and owner grant of the pruned sinks would stay in SpiceDB forever with nothing left in Postgres that could emit the purge.

Uniqueness is split across two partial indexes rather than one index on (domain_id, slug), because a NULL domain_id compares unequal to itself and a single index would admit two built-in sinks under the same slug:

Index	Predicate	Scope it enforces
sinks_domain_slug_uidx	WHERE domain_id IS NOT NULL	A Domain cannot hold two tenant sinks with the same slug.
sinks_builtin_slug_uidx	WHERE domain_id IS NULL	A slug is unique among the built-in sinks, which have no Domain to scope them.

A third, non-unique sinks_domain_idx backs the FK's referential check and the per-Domain enumeration.

Create, Update and Delete each write the row and append their outbox row inside one transaction, under the aggregate type Sink, so no consumer sees an event for a write that rolled back. A Delete against a sink that is already gone — or against a built-in one, which the statement declines — reports zero rows affected, rolls back and announces nothing; one read tells the two cases apart so the caller reads ErrSinkNotFound or ErrSinkConflict rather than a single ambiguous answer.

Update carries the compare-and-swap precondition in the statement itself: WHERE id = $1 AND updated_at = $13, with the caller passing the updated_at it read the sink at. A caller states a whole post-image, so a write from a stale read would restore every mutable field its author never looked at — the TLS posture and the credential reference among them — while the event it emits reads as an ordinary update. The predicate has to live in the statement: one evaluated in application memory decides against a row that can move before the write reaches it. The zero-row outcome is ambiguous the same way Delete's is, so one read tells the two apart and the caller gets ErrSinkStaleWrite or ErrSinkNotFound. UpsertBuiltIn runs as a single statement and appends no outbox row. SinkUpdated names the sink and the acting principal only, never the changed attributes, so no destination address or credential coordinate reaches a bus that a projector and an audit log both read.

The down arm refuses the downgrade with SQLSTATE 0A000. The table holds the endpoints, transport posture and credential references an operator deliberately authored per Domain; dropping it discards every authored sink, and a downgrade-then-upgrade cycle would restore an empty table. An operator performing a legitimate wipe-and-reinstall drops the Postgres database instead.

How the export engine addresses a sink

The export engine knows one sink type, the aggregate on this page. It holds no enum of its own: the closed mimir | loki | siem set it once switched over is the three built-in rows, and SinkType is what an export path now branches on to pick a transform.

The two shapes are addressed differently, and the difference is which state the binding depends on. A built-in destination is bound from the egress configuration: setting its URL enables it, and the engine drains its stream whether or not the boot seed has written its row yet. A tenant destination is bound through route resolution: a route names the sink by id, resolution narrows that target to the ones the owning Project may use, and the consumer lifecycle provisions the sink's own durable consumers off the sink rows.

The metric label follows the same split: a built-in delivery carries the slug, a tenant delivery carries the sink id, because a slug is unique per Domain only.

Cross-references

• ./ingest.md — the ingest front door that admits, validates and buffers each batch onto a per-Domain JetStream stream.
• ./routing.md — the egress consumer that drains those streams to the sinks a Project's routes name, and to Grafana Mimir, Grafana Loki and the audit SIEM when it names none.
• ./routes.md — the Route aggregate that targets a sink, and the capability table that says which signals each type receives.
• ../index.md — the bounded-contexts landing page.
• ../../../internal/observability/sinks — the domain root that pins the ubiquitous language.
• ../../../schema/authz.zed — the ReBAC schema that carries the sink definition.