plexsphere

Appearance

Watch your Domain ​

The earlier lessons built things: a Project, a Group, a Label, a Resource. This one does not change your tenant at all — it teaches you to watch it. Once a Domain is carrying real work you live in four read-mostly surfaces: how close the Domain is to its capacity ceilings, what its metrics say right now, what its logs said when something broke, and which alert rules are standing guard. plexsphere exposes all four through plexctl, each one mediated, per-Domain, and gated by the same authorization you have met throughout.

By the end you will have read a capacity snapshot, run a metrics query and a logs query through the control plane's read-only observability proxy, and created — then removed — an alert rule. You will finish knowing where to look first when a Domain misbehaves.

This lesson takes about twenty minutes.

A note on the lean dev stack. make dev runs the real metrics backend (Grafana Mimir) and logs backend (Grafana Loki), and the queries below go through the real, audited proxy. What the lean stack does not run is the Node-side telemetry pipeline that fills those backends: there is no fleet of plexd agents streaming node_cpu_seconds_total and application logs into them. So you will query live backends that start out nearly empty, and the lesson shows you how to put a signal where a real Node's pipeline would land it and read it straight back. Against a production cluster the same commands return your fleet's real series and log lines.

Before you start ​

You need a running, logged-in stack from Set up your local plexsphere: a plexsphere kind cluster and a plexctl that is built, on $PATH, and logged in as the seeded admin@example.com. You also need jq on your $PATH — the query surfaces return their backend's response envelope verbatim, and jq is how you read the part you care about out of it.

Recreate the shell environment from that lesson so the commands below work in a fresh terminal:

bash
export PATH="$PWD/bin:$PATH"
export PLEXSPHERE_URL=http://localhost:8080

eval "$(make -s dev-ids)"

$DOMAIN_ID now holds the UUID of the Acme Corp demo Domain. You act as admin@example.com throughout — a Domain admin holds both the read relation the query surfaces gate on and the manage relation the alert surface gates on, so you need no extra grants. Every command below is scoped to Acme Corp with --domain "$DOMAIN_ID"; that is the tenancy boundary asserting itself, exactly as in the earlier lessons.

Step 1 — Run a metrics query ​

Start at the metrics backend. metrics query evaluates a PromQL expression and returns the backend's response envelope unchanged — the control plane proxies your query to the Domain's Mimir backend, stamping the Domain as the upstream tenant server-side so you can only ever read your own Domain's metrics.

Run an instant query. Use vector(1) — a synthetic PromQL expression the query engine evaluates on its own, without needing any ingested series — so you see a real, non-empty round trip on the lean stack:

bash
plexctl metrics query \
  --domain "$DOMAIN_ID" \
  --query  'vector(1)' \
  --time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

In text mode the result is two columns — the upstream status code and the verbatim response body in a single cell:

text
STATUS_CODE  BODY
200          {"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1782051335,"1"]}]}}

The CLI deliberately does not reshape that body. When you want the structured series, pass --output json and let jq walk it — the typed shape carries the upstream body as a string under body, so you decode it once:

bash
plexctl metrics query \
  --domain "$DOMAIN_ID" \
  --query  'vector(1)' \
  --time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output json | jq -r '.body | fromjson | .data.result[0].value'
text
[
  1782051443,
  "1"
]

A range query swaps --time for a --start / --end window and a --step resolution, and returns a sample per step:

bash
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
START=$(date -u -v-10M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '-10 min' +%Y-%m-%dT%H:%M:%SZ)

plexctl metrics query \
  --domain "$DOMAIN_ID" \
  --query  'vector(1)' \
  --start  "$START" \
  --end    "$NOW" \
  --step   2m \
  --output json | jq -r '.body | fromjson | .data.result[0].values'
text
[
  [1782050843, "1"],
  [1782050963, "1"],
  [1782051083, "1"],
  [1782051203, "1"],
  [1782051443, "1"]
]

That is the full query path working end to end: your expression reached the backend, evaluated, and came back. On a production cluster you would swap vector(1) for the series your fleet actually emits — up, rate(node_cpu_seconds_total[5m]), node_filesystem_avail_bytes — and scope a single Node with --node-id. Those return rows here only once the Node telemetry pipeline is feeding Mimir; the query mechanism you just exercised is identical either way.

Step 2 — Read the capacity snapshot ​

The same metrics family carries a second, very different read: metrics capacity, a per-Domain headroom snapshot. It does not touch Mimir at all — the control plane samples it from its own datastore — and it answers a question PromQL cannot: how close is this Domain to the ceilings the platform enforces on it?

bash
plexctl metrics capacity --domain "$DOMAIN_ID"

Capacity measures load, and a freshly-seeded Acme Corp has none — no enrolled Nodes, no sessions, no traffic — so the collector has nothing to sample for it yet and tells you so:

text
plexctl: Service Unavailable (capacity_snapshot_unavailable): the capacity collector has not produced a snapshot for this Domain yet; retry after the next sample

That is the honest answer for an idle Domain, not a fault in your setup: the collector only snapshots dimensions that carry a measurable level, so a Domain with nothing to measure has no snapshot. The moment the Domain carries load — provision a Node (see Reach your Resource, which leaves one running for exactly this lesson), issue a session, drive traffic — a snapshot appears. With load it projects each catalogued dimension onto five columns — what is used, the target ceiling, the ratio between them, and the unit:

text
DIMENSION             USED  TARGET       RATIO   UNIT
nodes                 2     10000        0.0002  count
sse_fanout            0     1000         0       events_per_second
secret_reads          0     10000        0       reads_per_second
mediated_sessions     0     500          0       count
observability_ingest  0     5.24288e+06  0       bytes_per_second
action_executions     0     1000         0       count

That is the answer to "how much headroom does this Domain have?" without writing a single PromQL expression — here, two enrolled Nodes against a ceiling of ten thousand. The control plane records an audit row the moment any dimension crosses 80% of its target, so capacity is both a snapshot you can read and a tripwire that fires on its own. With --output json the typed DomainCapacitySnapshot carries the sampled_at timestamp alongside the dimensions.

Step 3 — Run a logs query ​

Logs work the same way, with LogQL instead of PromQL and an explicit --from / --to window instead of a relative duration, so the query bounds are never ambiguous on the wire. Ask for the last hour of any plexd logs. Every hit comes back as a [nanosecond-timestamp, line] pair, so the jq tail turns the first ten digits — the seconds — into a UTC date and prefixes each line with it:

bash
FROM=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ)
TO=$(date -u +%Y-%m-%dT%H:%M:%SZ)

plexctl logs query \
  --domain "$DOMAIN_ID" \
  --query  '{app="plexd"}' \
  --from   "$FROM" \
  --to     "$TO" \
  --output json \
  | jq -r '.body | fromjson | .data.result[].values[] | "\(.[0][:10] | tonumber | todate)  \(.[1])"'

On the lean stack this prints nothing: the query succeeded (a 200 with an empty streams envelope), but no Node is streaming logs into Loki, so there is nothing to match. That empty-but-valid answer is itself the lesson — the proxy, the per-Domain tenant scoping, and the LogQL evaluation all ran; only the payload is absent.

To watch a real line come back, put one where a Node's routing pipeline would land it. Push a single line shaped like a plexd log entry — the probe runs inside the cluster where Loki is already reachable by service name, so there is no port forward to babysit. --rm deletes the Pod afterwards and --quiet keeps its lifecycle chatter off; the $(date +%s) stamp expands on your machine before the Pod starts, so the line lands at "now":

bash
kubectl run loki-push --rm -i --quiet --restart=Never \
  --image=curlimages/curl:8.10.1 -- -sS -XPOST http://loki:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"app\":\"plexd\",\"level\":\"error\"},\"values\":[[\"$(date +%s)000000000\",\"demo enrolment failed: bootstrap token expired\"]]}]}"

Now re-run the same logs query, widening --to to now, and the line comes back through the mediated proxy:

bash
plexctl logs query \
  --domain "$DOMAIN_ID" \
  --query  '{app="plexd"}' \
  --from   "$FROM" \
  --to     "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output json \
  | jq -r '.body | fromjson | .data.result[].values[] | "\(.[0][:10] | tonumber | todate)  \(.[1])"'
text
2026-07-31T15:12:47Z  demo enrolment failed: bootstrap token expired

Refine the scan with --limit, flip the order with --direction forward, or filter the stream with a richer LogQL expression such as {app="plexd"} |= "token". As with metrics, the value you take away is the path: a Domain-scoped, audited query against a backend you can only read through the control plane.

Step 4 — Store an alert rule ​

Reading signals by hand is Day-2 ops in the small. To watch a signal continuously you store an alert rule on the Domain. A rule names a signal, a comparator, a threshold, and a severity — managed configuration the platform records and serves. Create one that watches CPU:

bash
ALERT_ID=$(plexctl alert create \
  --domain     "$DOMAIN_ID" \
  --name       high-cpu \
  --signal     'avg(rate(node_cpu_seconds_total[5m]))' \
  --comparator gt \
  --threshold  0.9 \
  --severity   warning \
  --output json | jq -r '.id')
echo "$ALERT_ID"

--comparator and --severity are validated against their enums before the request leaves your machine, so a typo such as --severity urgent fails locally rather than on the wire. List the Domain's rules and the new one is there:

bash
plexctl alert list --domain "$DOMAIN_ID"
text
NAME      SIGNAL                                 COMPARATOR  THRESHOLD  SEVERITY  ENABLED  ID
high-cpu  avg(rate(node_cpu_seconds_total[5m]))  gt          0.9        warning   true     019eea8b-6dd9-7e4a-bd06-7a988ea37218

A PATCH-shaped update sends only the fields you name — raise just the threshold and the rest of the rule is untouched:

bash
plexctl alert update --domain "$DOMAIN_ID" --id "$ALERT_ID" --threshold 0.95
text
NAME      SIGNAL                                 COMPARATOR  THRESHOLD  SEVERITY  ENABLED  ID
high-cpu  avg(rate(node_cpu_seconds_total[5m]))  gt          0.95       warning   true     019eea8b-6dd9-7e4a-bd06-7a988ea37218

A stored rule is configuration, not an evaluator: in this phase the platform records and serves rules but does not fire them on its own, so nothing pages you when CPU climbs — the rule is the durable statement of what to watch, ready for the evaluator that consumes it. Now clean up. delete is destructive, so it refuses to act without --yes:

bash
plexctl alert delete --domain "$DOMAIN_ID" --id "$ALERT_ID" --yes

The command prints nothing and exits 0; a final plexctl alert list --domain "$DOMAIN_ID" shows only the header row again. You created a rule, changed it, and removed it — the same CRUD shape you have used for Projects and Groups, now on the alerting surface.

What you learned ​

  • Watching a Domain is four read-mostly surfaces. Capacity (headroom against ceilings), metrics (PromQL, right now), logs (LogQL, over a window), and alert rules (standing configuration) are the day-to-day observability surfaces, all reached through plexctl.
  • The query surfaces are a mediated proxy, not direct backend access. Your PromQL and LogQL travel through the control plane, which stamps your Domain as the upstream tenant server-side — you can only ever read your own Domain's metrics and logs, and every query is gated by the read relation.
  • The CLI returns the backend envelope verbatim. text mode shows the status code and raw body; --output json | jq is how you read the structured series or streams out of it.
  • An alert rule is stored configuration. It is recorded and served, validated against its enums at flag-parse time, and managed with the same create / list / get / update / delete shape as the rest of the platform.

Where to go next ​

  • Keep learning by doing — Enrol your first Node opens the mesh track: mint a bootstrap token, redeem it from a fresh Node to obtain a mesh identity, then rotate the Node's mesh key end to end — the manual counterpart to the broker-driven enrolment you watched in Reach your Resource.

Or pick the quadrant that matches what you need now: