plexsphere

Appearance

🛡️ adminadmin@example.com · Domain admin in Acme Corp

Chapter 15 — Logs leave the platform ​

Every chapter so far put data in ​

Chapter 14 followed a log line into the platform: you appended it to a file, the agent forwarded it, and the platform's own Loki gave it back.

This chapter turns it around. The room is about to watch a line leave — to a collector on this machine, that the platform has never heard of until we tell it.

The rule we are about to bend, and why it exists ​

A tenant states where its telemetry goes. The platform dials that address from inside its own network.

So it refuses destinations that name its own network: loopback, the private ranges, the mesh. Without that, anyone who can declare a sink can point the delivery leg at the metadata service and read the answer off the delivery outcome.

Every collector on this laptop sits in exactly those ranges. The dev stack — only the dev stack — waives the rule.

Give the collector a certificate ​

The platform speaks syslog over TLS and nothing else. Self-signed is fine; it is its own authority.

bash
mkdir -p collector-tls
openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout collector-tls/collector.key -out collector-tls/collector.pem \
  -subj "/CN=syslog-collector" \
  -addext "subjectAltName=IP:172.18.0.250" \
  2>/dev/null
cp collector-tls/collector.pem collector-tls/ca.pem

The subject alternative name is the load-bearing part: the platform verifies the certificate against the exact host the endpoint names.

Start it on the cluster's own network ​

kind runs its nodes as containers on a Docker network. Put the collector there, at an address we choose.

bash
docker run --detach --name syslog-collector \
  --network kind --ip 172.18.0.250 --platform linux/amd64 \
  --volume "$PWD/collector-tls:/etc/rsyslog-tls:ro" \
  --env ENABLE_TLS=on \
  --env ENABLE_TCP=off --env ENABLE_UDP=off --env ENABLE_RELP=off \
  --env WRITE_ALL_FILE=on --env WRITE_JSON_FILE=on \
  --env TLS_AUTH_MODE=anon \
  --env TLS_CA_FILE=/etc/rsyslog-tls/ca.pem \
  --env TLS_CERT_FILE=/etc/rsyslog-tls/collector.pem \
  --env TLS_KEY_FILE=/etc/rsyslog-tls/collector.key \
  rsyslog/rsyslog-collector:2026-04

No config file: ENABLE_TLS=on is an RFC 5425 input on 6514. --platform linux/amd64 is what makes it run on Apple Silicon.

Declare it as a sink ​

A sink is one destination the Domain owns. Note what it takes: an address, and the authority to verify against.

bash
plexctl sink create \
  --domain-id    "$DOMAIN_ID" \
  --slug         local-collector \
  --display-name "Local collector" \
  --type         syslog \
  --endpoint     172.18.0.250:6514 \
  --ca-pem-file  collector-tls/ca.pem

No credential. A syslog collector authenticates clients with a certificate, not a token — so a syslog sink that named one would be left without an exporter at all.

Grant it into the Project ​

The Domain owns the destination. A Project may only deliver to one it holds an approved grant for.

bash
plexctl sink enablement grant \
  --sink-id    "$SINK_ID" \
  --project-id "$PROJECT_ID"

One step, and approved immediately — because the other direction is four-eyes, and requesting then approving your own request is refused. Without this path a sink's owner could not place their own sink.

Route the logs — to both destinations ​

This is the slide to slow down on. A route replaces the platform's default for its signal.

bash
plexctl route create \
  --project-id "$PROJECT_ID" \
  --signal     logs \
  --sink-id    "$SINK_ID" \
  --sink-id    "$LOKI_SINK_ID"

Name only the collector and the Project's logs stop reaching Loki. A route naming one destination is a decision to stop sending to the others.

Write the line ​

Two thirty-second windows: the engine reconciles consumers every thirty seconds and caches routes for thirty.

bash
sleep 70
echo "the collector is mine and this line proves it" >> agent-logs/app.log

Watch it arrive ​

bash
docker exec syslog-collector tail -1 /var/log/all.log
text
2026-08-20T21:17:51.468979Z 8521c6977d64 /var/log/agent/app.log the collector is mine and this line proves it

Shell → file → agent → ingest buffer → routing engine → TLS → a container this room watched start ninety seconds ago.

And Loki still has it ​

Because the route named both.

text
{"severity":"info","unit":"/var/log/agent/app.log","hostname":"8521c6977d64","message":"the collector is mine and this line proves it","timestamp":"2026-08-20T21:17:51.468979466Z"}

One record, two destinations, because we asked for both.

The platform counts it the same way ​

bash
kubectl get --raw "/api/v1/namespaces/default/services/http:plexsphere:http/proxy/metrics" \
  | grep "observability_routing_batches_total" | grep -v '^#'
text
plexsphere_observability_routing_batches_total{outcome="exported",signal="logs",sink="01a02107-ee95-7cd2-8044-099bb488502a"} 1
plexsphere_observability_routing_batches_total{outcome="exported",signal="logs",sink="loki"} 1

Two rows, one per destination, both exported. The tenant sink is labelled by id, not slug — two Domains naming a collector local-collector would otherwise share one series.

What the room just saw ​

  • A destination is a thing the Domain owns — declared once, granted into Projects, revoked without deleting.
  • A route replaces, it does not add — which is why Loki was named alongside the collector.
  • The platform refuses to dial its own network — checked when the sink is declared, and again on every connection, against the address actually resolved.
  • Egress is verified against material the tenant provides — our authority, our host, and the platform proving nothing in return.

That closes the mesh track, and the deck.