Deliver telemetry to Dash0

Dash0 receives OTLP/HTTP, which is the one tenant protocol the platform ships an exporter adapter for. A Dash0 destination is therefore an ordinary tenant sink of type otlp: a Domain declares it, a Project is granted the use of it, and a route sends that Project's telemetry there instead of to the platform's own backends.

This guide takes the local dev stack as the running installation. The wire mapping the adapter produces is on the routing page.

What a Dash0 sink needs

Three values, all read off the Dash0 organization settings:

Value	Where it goes
the OTLP/HTTP ingress endpoint	the sink's --endpoint
the dataset	the sink's --dataset, sent as the Dash0-Dataset header
the auth token	the KV store, under the token key

The endpoint is a base URL: the exporter appends /v1/metrics for metrics and /v1/logs for logs and audit events, and tells the last two apart by the OTLP scope name. Paste the ingress host, not the signal URL a vendor page shows — an endpoint that already carries a path is refused when the exporter is built, and the sink is left without one rather than POSTing to /v1/logs/v1/logs. The dataset header is sent only when the sink states a dataset, and Authorization: Bearer <token> only when the sink states a credential.

A credentialed sink must be https. The platform reads the token out of its own KV store and puts it on the wire on every batch, so a sink that states a credential is refused unless its endpoint names https and its TLS posture leaves verification on. A destination with a private certificate states a CA bundle (--ca-pem-file) instead of switching verification off. Dash0-Dataset is Dash0's own header: another OTLP receiver ignores it and files the telemetry under its default, without reporting anything.

The token never reaches the sink. The sink holds the KV mount, the derived path and the version to read; the material stays in the store and is read on every export.

Before you start

• A running dev stack (make dev). Its overlay sets both PLEXSPHERE_OBS_NATS_URL and PLEXSPHERE_DSN, which is the routing pipeline's opt-in gate, so the pipeline is live whether or not a platform backend is configured.
• The server wired to the secret store. A sink that states a credential needs PLEXSPHERE_SECRETS_OPENBAO_MOUNT, PLEXSPHERE_SECRETS_OPENBAO_ADDRESS and one auth strategy (the PLEXSPHERE_SECRETS_OPENBAO_APPROLE_ROLE_ID and ..._APPROLE_SECRET_ID pair, or ..._K8S_ROLE). The dev overlay sets none of them, so add them to ../../../deploy/local/base/plexsphere/configmap.yaml and roll the deployment first. Without them the credential source stays unwired: the adapter for a credentialed sink is refused, the server logs a warning naming the sink, and no consumer is provisioned for it. A credential-less sink keeps delivering.
• kubectl pointed at the dev cluster (kind-plexsphere). The dev overlay applies into the default namespace, so no -n flag is needed below. Only the token write goes through kubectl; the sink, the grant and the route go through plexctl.
• Permissions on the three objects the steps touch: manage on the Domain to declare the sink, assign on the sink to grant it, and deploy on the Project to state the route. The seeded admin operator holds all three.

Read the ids you will substitute

export PATH="$PWD/bin:$PATH"
export PLEXSPHERE_URL=http://localhost:8080

eval "$(make -s dev-ids)"   # exports DOMAIN_ID for the seeded acme-corp Domain
plexctl login --domain-id "$DOMAIN_ID"

plexctl project list --domain "$DOMAIN_ID"

uuidgen | tr '[:upper:]' '[:lower:]'   # the sink id, minted before anything else

Every <placeholder> below is one of these: the domain id, the project id, the sink id you just minted, and the Dash0 endpoint and dataset. The enablement and the route take server-minted ids, so the sink id is the only one you mint by hand; the next section says why. Any canonical UUID works; the platform mints UUIDv7 so creation order sorts.

Write the token into the secret store

The sink does not say where its token lives. The declare request carries the KV mount and, optionally, the version to read; it carries no path and no material. The platform derives the path from the owning Domain and the sink id, as domains/<domain-id>/sinks/<sink-id>, so no caller can point a sink at another tenant's material. That is why the sink id is minted first: the secret is written at the derived path before the sink exists.

The dev stack runs OpenBao in -dev mode. KV v2 is mounted at secret, the service answers on openbao:8200 inside the cluster, and the root token is the dev-only literal in ../../../deploy/local/base/openbao/secret.yaml.

kubectl exec deploy/openbao -- sh -c "
  export BAO_ADDR=http://127.0.0.1:8200
  export BAO_TOKEN=dev-root-token-do-not-use
  bao kv put secret/domains/<domain-id>/sinks/<sink-id> token='<dash0-auth-token>'
"

Only the token key is read, and it must hold a non-empty string. Every other key in the payload is ignored, so the entry may carry whatever else the Dash0 onboarding asked you to keep. The path is the part most likely to be wrong; confirm it holds a version without printing the token:

kubectl exec deploy/openbao -- sh -c "
  export BAO_ADDR=http://127.0.0.1:8200
  export BAO_TOKEN=dev-root-token-do-not-use
  bao kv metadata get secret/domains/<domain-id>/sinks/<sink-id>
"

Declare the sink

plexctl sink create \
  --domain-id           "$DOMAIN_ID" \
  --id                  <sink-id> \
  --slug                dash0 \
  --display-name        "Dash0" \
  --type                otlp \
  --endpoint            <dash0-endpoint> \
  --credential-kv-mount secret \
  --dataset             <dash0-dataset>

--credential-kv-mount is the whole credential the request carries. --credential-kv-version pins the sink to one revision, which a later rotation then does not reach; omitted, the sink reads whichever version is latest. --ca-pem-file is the one flag that carries bytes rather than coordinates, for a destination whose certificate does not chain to the system trust store; without it verification runs against that store.

The aggregate checks the body before anything is stored, and the refusal names what it refused. A slug another sink of the Domain already holds fails with 409 sink_slug_taken. 422 sink_invalid covers the rest: a slug that is not kebab-case or is longer than 63 characters, an empty display name, an endpoint that is not an http or https URL with a host, one whose host is a literal address inside the platform's own network, and a CA bundle that decodes to no certificate. The Sink context page has the full set.

The https-with-a-credential rule is not one of them. It is enforced where the exporter is built rather than where the sink is written, so a credentialed sink on an http endpoint, or one whose endpoint carries a path, is created and then left without a consumer. That reads as no metric series at all, which the counters section below covers.

Grant the sink to the Project

Route resolution narrows a Project's targets to the built-in sinks plus the tenant sinks it holds an approved enablement for. A requested enablement grants nothing.

The sink's owner places the grant in one step:

plexctl sink enablement grant \
  --sink-id    <sink-id> \
  --project-id <project-id>

The enablement lands already approved, so the sink is usable in the Project the moment the command returns. The single step is deliberate: requesting and then approving one's own request is refused by the four-eyes guard, so without this route a sink's owner could not place it at all.

The other direction is the consuming Project asking, and a second principal deciding:

plexctl sink enablement request \
  --project-id <project-id> \
  --sink-id    <sink-id>

plexctl approval list --kind sink_enablement
plexctl approval approve <enablement-id>

Either way, read the result back and confirm the state is approved:

plexctl sink enablement list --project-id <project-id> --all

A grant response carrying sync_pending: true is a committed grant whose authorization edge has not landed yet. Do not retry it: a second grant for the same pair is refused with 409 duplicate_live_sink_enablement while the first is live, and the committed event drives the same edge through the sync arm on its own.

State the route

A route governs exactly one signal, and it replaces the platform's default for that signal. After the command below the Project's logs go to Dash0 and stop reaching Loki, unless the route names the loki sink too.

plexctl route create \
  --project-id <project-id> \
  --signal     logs \
  --sink-id    <sink-id>

An otlp sink receives all three signals, so repeat the command with --signal metrics and --signal audit for the rest of the Project's telemetry. Neither predicate flag is stated here, which sends every record on the signal; --severity-floor narrows a logs route to a syslog severity and above, and --name-prefix narrows a metrics route to a metric-name prefix. Each is accepted on its own signal only, and stating one on another signal fails with 422 telemetry_route_invalid.

Every target is checked where the route is written, so a route that would deliver nowhere is refused rather than stored: a --sink-id naming no sink fails with 422 route_sink_not_found, a tenant sink the Project holds no approved enablement for with 422 route_sink_not_usable, and a sink whose type cannot receive the signal with 422 route_signal_not_accepted.

To keep Loki carrying the same logs alongside Dash0, name both targets. Read the built-in id off the roster:

plexctl sink list --built-in

plexctl route create \
  --project-id <project-id> \
  --signal     logs \
  --sink-id    <sink-id> \
  --sink-id    <loki-sink-id>

A built-in sink needs no enablement, so it passes the usability check by construction. On a route that already exists, restate both targets with plexctl route update <route-id>: the body is the full post-image, so a target the command does not name is dropped.

Read the delivery metrics

Two 30-second windows stand between the rows and the first delivery: the consumer lifecycle reconciles the desired durables every 30 seconds, and the route resolver caches a Project's routes for 30 seconds. Give it a minute, then sample the counters. The dev stack serves them unauthenticated:

curl -s http://localhost:8080/metrics | grep '<sink-id>'

The sink label of a tenant destination is the sink id, not the slug, because a slug is unique per Domain only and two Domains naming their destination dash0 would otherwise share one time series.

Metric	Reads
plexsphere_observability_routing_batches_total	batches driven to a terminal disposition, by signal and outcome (exported, dropped, skipped)
plexsphere_observability_routing_records_total	records exported, by signal
plexsphere_observability_routing_retries_total	retryable outcomes that scheduled a redelivery
plexsphere_observability_routing_resolution_failures_total	deliveries whose route resolution failed
plexsphere_observability_routing_lag_seconds	age of the telemetry at export time

A climbing outcome="exported" is the delivery working. The three ways it goes wrong read differently:

• retries_total climbs and nothing is exported. The export keeps failing retryably: a transport error, a 429, a 5xx, a 3xx, a dial the internal-address guard refused, or a credential the store would not yield. The batch stays on the stream, and the server log names the reason. A 3xx reads this way because an export follows no redirect: a destination that starts answering one is repaired by correcting the endpoint, not by chasing an address no check has seen, and the batch waits on the stream while you do.
• outcome="skipped" climbs. The routes selected none of this sink's records. A missing route, or an enablement that is not approved, looks like this.
• No series at all for the sink id. No consumer was provisioned, because the exporter could not be built. A credentialed sink with no secret store wired reads this way, and so do an endpoint carrying a path, a query or userinfo, a credentialed sink on an http endpoint or with verification switched off, and a dataset holding a control character. The server log names the sink and the reason on every reconcile.
• outcome="dropped" climbs. The destination answered a non-4294xx, which is terminal: the batch is acked and gone. A 404 here is usually the endpoint — check it is the ingress base URL and not a signal URL. A 401 or 403 is the token.
• outcome="exported" climbs but the destination shows less. Read the server log for destination rejected part of an accepted batch: a 200 carrying an OTLP partial-success message is an accepted export by the spec, and the rejected count and the destination's reason are logged rather than counted.

Rotate the token

Write the new value at the same path. The token is read from the store on every export and is never cached, so the next batch carries it: there is nothing to restart and no cache to wait out.

kubectl exec deploy/openbao -- sh -c "
  export BAO_ADDR=http://127.0.0.1:8200
  export BAO_TOKEN=dev-root-token-do-not-use
  bao kv put secret/domains/<domain-id>/sinks/<sink-id> token='<new-dash0-auth-token>'
"

A sink that states --credential-kv-version keeps reading the pinned version, so a rotation reaches it only once plexctl sink update restates the new one. That command sends the full post-image, so restate every field the sink holds, not just the version, and pass the updated_atplexctl sink get prints as --expected-updated-at: it is the compare-and-swap token that turns a concurrent edit into a 409 sink_cas_conflict instead of a silent revert. Omit --credential-kv-version to follow the latest version and let a rotation reach the sink with no API call at all.

Remove the destination

Removal runs in reference order: delete the routes that target the sink, revoke the enablement, then delete the sink. A sink a route still targets is not deletable, and the refusal is what keeps a route from silently shrinking to no target.

plexctl route delete <route-id> --yes
plexctl sink enablement revoke <enablement-id> --yes \
  --reason "Dash0 trial concluded"
plexctl sink delete <sink-id> --yes

A 502 revocation_sync_pending on the revoke is not a completed revocation: the row moved to revoked while the grant is still live in the authorization graph, so the Project keeps delivering. Retrying is refused, and the removal replays through the committed event.

The material at the derived KV path is not deleted with the sink. Remove it with bao kv metadata delete at the same path when the destination is gone for good.

Related

• OTLP mapping: the resource attributes, scope names, severity table, credential contract and dial-time guard the adapter applies.
• Telemetry Sinks HTTP API: the fifteen operations these commands drive, their ReBAC gates, and the closed Problem.code taxonomy behind every refusal quoted above.
• plexctl sink and plexctl route: the full flag reference for both command families.
• Telemetry sinks: the aggregate the sink hydrates into and every invariant it enforces.
• Sink enablement: the four-eyes grant lifecycle behind the grant and the request paths.
• Telemetry routes: the replace-per-signal rule and the two predicates.