Route logs to your own collector

Run your first agent followed a log line into the platform: you appended it to a file, the agent forwarded it, and plexctl logs query read it back out of the platform's own Loki. Every lesson so far has worked that way — data goes in, and you read it where the platform keeps it.

This lesson turns that around. You will stand up a syslog collector of your own, tell the platform it is a legitimate destination, and route your Project's logs to it. At the end you will tail a file inside a container you started and find a line you typed in your own shell, delivered there by the platform over TLS.

You will:

• run a TLS syslog collector as one Docker container on the same network the cluster runs on,
• declare it as a tenant sink your Domain owns, verified against a certificate authority you mint yourself,
• grant the sink to your Project, the one step that makes a Domain's destination usable inside a Project,
• state a route that sends the Project's logs to your collector and keeps the platform's own Loki receiving them,
• and watch one line travel from your shell, through the agent, through the platform, and out to your collector.

This lesson takes about fifteen minutes.

Before you start

This lesson continues Run your first agent. You need its plexd-agent container still running and still forwarding from agent-logs/, because that agent is what produces the log records this lesson routes. If you stopped it, run that lesson again before this one.

It adds no tools beyond the ones that lesson already used: Docker, jq, and openssl, which ships with macOS and every Linux distribution.

Recreate the shell environment and sign in:

export PATH="$PWD/bin:$PATH"
export PLEXSPHERE_URL=http://localhost:8080

eval "$(make -s dev-ids)"
plexctl login --domain-id "$DOMAIN_ID"

You also need the edge-fleet Project id again — the Project your agent registered into:

PROJECT_ID=$(plexctl project list --domain "$DOMAIN_ID" --output json \
  | jq -r '.items[] | select(.slug == "edge-fleet") | .id')
echo "$PROJECT_ID"

01a02106-1d58-7b87-82c2-53eaa3938eb1

Why this works on the dev stack and not in production

A tenant states where its telemetry goes, and the platform dials that address from inside its own network. So the platform refuses a destination that names its own network: loopback, the private ranges, the mesh range, and the rest. Without that rule, anyone allowed to declare a sink could point the delivery leg at the cloud metadata service or the API server and read the answer off the delivery outcome.

Every collector you can reach from your own machine sits in exactly those refused ranges. So the dev stack — and only the dev stack — sets PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS, which waives the rule at both ends: the sink may be declared, and the delivery may dial it. A real installation leaves it unset, and the how-to guide walks the same flow with a collector on a routable address.

Step 1 — Give the collector a certificate

The platform speaks syslog over TLS and nothing else, so the collector needs a certificate before it can accept anything. Mint a self-signed one — it is its own certificate authority, which is all a first delivery needs:

mkdir -p collector-tls
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout collector-tls/collector.key -out collector-tls/collector.pem \
  -subj "/CN=syslog-collector" \
  -addext "subjectAltName=IP:172.18.0.250" \
  2>/dev/null
cp collector-tls/collector.pem collector-tls/ca.pem

The subject alternative name is the part that matters. The platform dials an address, wraps the connection in TLS itself, and verifies the certificate against the exact host the sink's endpoint names — so the certificate has to carry that address, and 172.18.0.250 is the address you will pin the container to in the next step.

Step 2 — Start the collector on the cluster's network

kind runs its nodes as containers on a Docker network called kind. Put the collector on that same network and the cluster can reach it, at an address you choose rather than one Docker hands out:

docker run --detach --name syslog-collector \
  --network kind --ip 172.18.0.250 --platform linux/amd64 \
  --volume "$PWD/collector-tls:/etc/rsyslog-tls:ro" \
  --env ENABLE_TLS=on \
  --env ENABLE_TCP=off --env ENABLE_UDP=off --env ENABLE_RELP=off \
  --env WRITE_ALL_FILE=on --env WRITE_JSON_FILE=on \
  --env TLS_AUTH_MODE=anon \
  --env TLS_CA_FILE=/etc/rsyslog-tls/ca.pem \
  --env TLS_CERT_FILE=/etc/rsyslog-tls/collector.pem \
  --env TLS_KEY_FILE=/etc/rsyslog-tls/collector.key \
  rsyslog/rsyslog-collector:2026-04

cac36c4f214f94c00547409a93c01e85d190ccc6a70e1190d6d8cc97794c2d27

The image needs no configuration file: ENABLE_TLS=on switches on an RFC 5425 input on port 6514, which is the framing the platform writes, and the three other ENABLE_* switches turn off the inputs you are not using. TLS_AUTH_MODE=anon tells the collector not to ask the client for a certificate, which matches how the platform connects — it verifies the collector, and proves nothing about itself. WRITE_ALL_FILE and WRITE_JSON_FILE are what put received messages into /var/log/all.log and /var/log/all-json.log.

The image ships for amd64 only, so --platform linux/amd64 is what makes the identical command work on an Apple Silicon machine. It runs under emulation there and takes a few seconds longer to start.

Confirm it came up:

docker ps --filter name=syslog-collector --format '{{.Status}}'

Up 5 seconds

Step 3 — Declare the collector as a sink

A sink is one destination your Domain may deliver to. Declaring it is a Domain-level act: the Domain owns the destination, and Projects are granted the use of it separately.

plexctl sink create \
  --domain-id    "$DOMAIN_ID" \
  --slug         local-collector \
  --display-name "Local collector" \
  --type         syslog \
  --endpoint     172.18.0.250:6514 \
  --ca-pem-file  collector-tls/ca.pem

ID                                    DOMAIN_ID                             SLUG             DISPLAY_NAME     TYPE    ENDPOINT           BUILT_IN  CREATED_AT
01a02107-ee95-7cd2-8044-099bb488502a  01a02103-4c03-7bb1-8593-98d88e42b610  local-collector  Local collector  syslog  172.18.0.250:6514  false     2026-08-20T21:16:03Z

--ca-pem-file reads the bundle on your machine and sends the certificate bytes with the request; the path is local, and the platform stores what it read. A syslog sink takes no credential: the client-side convention a collector speaks is mutual TLS, which needs a certificate rather than the token the credential contract carries, so a syslog sink that named one would be left without an exporter entirely.

Capture the id the server minted — the next two steps both need it:

SINK_ID=$(plexctl sink list --domain-id "$DOMAIN_ID" --output json \
  | jq -r '.items[] | select(.slug == "local-collector") | .id')
echo "$SINK_ID"

01a02107-ee95-7cd2-8044-099bb488502a

Step 4 — Grant the sink to your Project

A Project may only deliver to a Domain sink it holds an approved grant for. The sink's owner places that grant in one step:

plexctl sink enablement grant \
  --sink-id    "$SINK_ID" \
  --project-id "$PROJECT_ID"

ID                                    PROJECT_ID                            SINK_ID                               STATE     REQUESTED_BY                          DECIDED_AT            CREATED_AT
01a02108-2b4b-7d98-ae88-1caaaba02a79  01a02106-1d58-7b87-82c2-53eaa3938eb1  01a02107-ee95-7cd2-8044-099bb488502a  approved  01a02103-4c14-7c87-936a-3a6efa31b24b  2026-08-20T21:16:18Z  2026-08-20T21:16:18Z

The grant lands approved immediately, and the single step is deliberate. The other direction — a Project requesting a sink it does not own — is decided by a second person, and requesting then approving your own request is refused by the four-eyes guard. Without this one-step path a sink's owner could not place their own sink at all.

Step 5 — Route the Project's logs to it

A route governs one signal for one Project, and it replaces the platform's default for that signal. That is the part worth slowing down for: a route naming only your collector would send the Project's logs to your collector and stop sending them to Loki.

So name both. Read the built-in Loki sink's id first:

LOKI_SINK_ID=$(plexctl sink list --built-in --output json \
  | jq -r '.items[] | select(.slug == "loki") | .id')
echo "$LOKI_SINK_ID"

01a02103-683c-734e-b3c5-66f110a0bfc8

plexctl route create \
  --project-id "$PROJECT_ID" \
  --signal     logs \
  --sink-id    "$SINK_ID" \
  --sink-id    "$LOKI_SINK_ID"

ID                                    PROJECT_ID                            SIGNAL  SEVERITY_FLOOR  NAME_PREFIX  SINK_IDS                                                                   CREATED_AT
01a02108-4864-767a-b29f-f3297779c6ca  01a02106-1d58-7b87-82c2-53eaa3938eb1  logs                                 01a02107-ee95-7cd2-8044-099bb488502a,01a02103-683c-734e-b3c5-66f110a0bfc8  2026-08-20T21:16:26Z

A built-in sink needs no grant — it belongs to the platform, not to a Domain — so it passes the usability check by construction.

Step 6 — Watch a line leave the platform

Two thirty-second windows stand between the route and the first delivery: the engine reconciles which destinations need a consumer every thirty seconds, and it caches a Project's routes for thirty seconds. Wait a minute, then write a line into the file your agent watches:

sleep 70
echo "the collector is mine and this line proves it" >> agent-logs/app.log

The agent collects new lines every ten seconds and reports every thirty, so give it another moment, then read your collector's file:

docker exec syslog-collector tail -1 /var/log/all.log

2026-08-20T21:17:51.468979Z 8521c6977d64 /var/log/agent/app.log the collector is mine and this line proves it

There it is. That line went from your shell into a file, from the file into the agent, from the agent into the platform's ingest buffer, out through the routing engine, over TLS to a container you started — and the platform never had to be told anything about that container except its address and its certificate authority.

The fields are the syslog projection of the record: the timestamp, the agent container's hostname, the source file as the application name, and your text as the message.

Loki has it too, because the route named both destinations:

FROM=$(date -u -v-15M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '-15 min' +%Y-%m-%dT%H:%M:%SZ)

plexctl logs query --domain "$DOMAIN_ID" \
  --query '{project="'"$PROJECT_ID"'"}' \
  --from "$FROM" --to "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output json | jq -r '.body | fromjson | .data.result[].values[][1]' \
  | grep "collector is mine"

{"severity":"info","unit":"/var/log/agent/app.log","hostname":"8521c6977d64","message":"the collector is mine and this line proves it","timestamp":"2026-08-20T21:17:51.468979466Z"}

One record, two destinations, because you asked for both.

The platform counts the same thing from its own side. The dev stack serves its metrics unauthenticated:

curl -s http://localhost:8080/metrics \
  | grep "observability_routing_batches_total" | grep -v '^#'

plexsphere_observability_routing_batches_total{outcome="exported",signal="logs",sink="01a02107-ee95-7cd2-8044-099bb488502a"} 1
plexsphere_observability_routing_batches_total{outcome="exported",signal="logs",sink="loki"} 1

Two rows, one per destination, both exported. A tenant sink is labelled by its id rather than its slug, because a slug is unique only within a Domain and two Domains both naming a destination local-collector would otherwise share one series.

Step 7 — Take it down again

Removal runs in reference order: the route first, then the grant, then the sink. A sink a route still targets cannot be deleted, and that refusal is what stops a route from quietly shrinking to no destination at all.

ROUTE_ID=$(plexctl route list --project-id "$PROJECT_ID" --output json \
  | jq -r '.items[] | select(.signal == "logs") | .id')

plexctl route delete "$ROUTE_ID" --yes
plexctl sink enablement revoke \
  "$(plexctl sink enablement list --project-id "$PROJECT_ID" --all --output json \
     | jq -r '.items[] | select(.sink_id == "'"$SINK_ID"'") | .id')" \
  --yes --reason "Lesson finished"
plexctl sink delete "$SINK_ID" --yes

docker rm --force syslog-collector

With the route gone, the Project's logs fall back to the platform's default for that signal, which is the Loki you were reading from all along.

Troubleshooting

422 sink_invalid when you declare the sink

The refusal names Sink.Endpoint. Your stack does not carry the internal-destination opt-in — it predates it, or it was started from a checkout without it. Confirm what the running deployment states:

kubectl get configmap plexsphere-config \
  -o jsonpath='{.data.PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS}'

true

An empty answer means the key is missing; re-run make dev from an up-to-date checkout.

Nothing arrives, and the collector's file stays empty

Check the collector is still up and its address has not moved:

docker inspect syslog-collector \
  --format '{{ (index .NetworkSettings.Networks "kind").IPAddress }}'

172.18.0.250

An address other than 172.18.0.250 means the container was recreated without --ip, so the sink's endpoint and the certificate now name a host nobody is listening on. Remove it and repeat Step 2.

Every transport failure here is retried rather than dropped, because syslog over TLS has no status channel to tell a refusal from an outage. So a wrong address, an expired certificate, or a certificate that does not name the endpoint's host all look the same from the outside: the batch waits, and the collector's file stays empty.

The line never reaches the platform at all

Then the problem is upstream of this lesson. Confirm the agent is still running and still reporting:

docker logs plexd-agent 2>&1 | grep 'logs reported' | tail -1

time=2026-08-20T21:15:01.477Z level=DEBUG msg="logs reported to platform" component=logfwd accepted_at=2026-08-20T21:15:01.474Z

No receipts means the agent is not forwarding — the troubleshooting section of Run your first agent covers that side.

The collector container will not start

On an Apple Silicon machine it runs under emulation and can take longer than you expect. docker logs syslog-collector ends with a line saying rsyslogd ... start once it is ready; a complaint about a certificate file means the collector-tls/ volume did not mount where the TLS_* variables point.

What you learned

• A destination is a first-class thing your Domain owns. A sink is declared once at the Domain and granted into Projects, so the same collector serves many Projects without being restated, and revoking the grant stops delivery without deleting anything.
• A route replaces, it does not add. Stating a route for a signal supersedes the platform's default for that signal, which is why this lesson's route names Loki alongside the collector. A route that names one destination is a decision to stop sending to the others.
• The platform refuses to dial its own network, on purpose. The rule you waived here is what stops a tenant-stated destination from becoming a request forger inside the cluster, and it is checked twice: once when the sink is declared, and again on every connection, against the address that was actually resolved.
• Egress is TLS-verified against material you provide. The platform checked your collector against a certificate authority you minted and a host you named, and it proved nothing about itself in return.

Where to go next

You have now followed telemetry in both directions: into the platform in the previous lesson, and back out in this one. That closes the mesh track.

• Keep learning by doing — Read the backup catalog opens the platform track: step into the platform operator's chair and read what the platform backs up and in what order it would be restored.

Or pick the quadrant that matches what you need now:

• You want to do this for real — Deliver logs to a syslog collector is the same flow against a collector on a routable address, with the certificate rotation and the failure modes an operator needs.
• You want the wire format your collector received — the syslog mapping covers the framing, the facility split, and the structured-data element that travelled with your line.
• You want the rules behind the refusals — the telemetry sinks context lists every invariant a sink is held to, and telemetry routes covers the replace-per-signal rule this lesson leaned on.
• You want the whole surface — the Telemetry Sinks API reference documents the fifteen operations these commands drove.