Run your first agent

Enrol your first Node made you speak the agent contract by hand: mint a token, curl POST /v1/register, read the mesh, rotate a key. This lesson hands that same contract to the software written to speak it — the released plexd agent, started as one Docker container on your own host, outside the kind cluster. You stay in the operator's chair the whole time and watch what a real agent does with the seam you drove manually — including the two things the previous lesson could only describe: the WireGuard data plane and the telemetry pipelines.

You will:

• start the released plexd agent with a single docker run, granting it exactly the one capability its data plane needs,
• watch it redeem a bootstrap token, register itself, and bring up a real WireGuard interface with the whole mesh programmed onto it,
• read the now-five-row mesh as the operator,
• feed the agent a log file and follow one line from your shell all the way into the platform's logs backend and back out of plexctl logs query,
• and read the agent's reports bucket and its ingest receipts.

By the end you will understand what the previous lesson's curl calls look like when a real agent runs them on a timer, and how much of a Node's story you can read without ever logging into the Node.

This lesson takes about twenty minutes.

Before you start

This lesson continues the mesh track's first lesson, Enrol your first Node. You need a running, logged-in stack from Set up your local plexsphere and that lesson finished — its edge-fleet Project and the two hand-enrolled Nodes edge-router-01 and edge-router-02 are part of the state this lesson reads.

It adds two command-line tools beyond the base lesson's set:

Tool	Purpose
Docker	run the agent as a container on your own host
jq	read fields out of the JSON responses

There is no wg, no xxd, and no raw curl this time. The agent does the key handling the previous lesson made you do by hand, so your side of the work is plain plexctl, one config file, and one docker command.

Recreate the shell environment and read the Acme Corp Domain id straight off the stack — plexctl login needs the Domain UUID before any authenticated call could look it up:

export PATH="$PWD/bin:$PATH"
export PLEXSPHERE_URL=http://localhost:8080

eval "$(make -s dev-ids)"
echo "$DOMAIN_ID"

019fc90c-51f8-7ec8-a084-db0569d487d2

Device-code tokens are short-lived, so sign in fresh, completing the browser prompt as admin@example.com with the password password:

plexctl login --domain-id "$DOMAIN_ID"

You also need the edge-fleet Project id, and plexctl project get cannot help: it takes a Project UUID, which is the value you are trying to recover. Read it out of the Project list by slug instead:

PROJECT_ID=$(plexctl project list --domain "$DOMAIN_ID" --output json \
  | jq -r '.items[] | select(.slug == "edge-fleet") | .id')
echo "$PROJECT_ID"

019fc90e-f6f0-7944-b328-5eef9ae0680c

Step 1 — Mint a bootstrap token

The agent needs the same single-use credential the previous lesson taught: the plaintext is shown exactly once, the server keeps only a hash, and the first redemption burns it. Mint one and hold the plaintext in a variable:

TOKEN=$(plexctl bootstrap-token issue \
  --project "$PROJECT_ID" --kind node --env-prefix dev --ttl 1h \
  --output json | jq -r '.token')

The command prints nothing, because jq pulls one field out of the record. The record itself — the one the agent below consumed — looks like this:

{
  "expires_at": "2026-08-03T20:19:18.754337422Z",
  "issued_at": "2026-08-03T19:19:18.754337422Z",
  "token": "psb_dev_agp4sdxw6b4ujmzil3xzvydibq_node_5ohap2iqxixbel65h4yf5nsjoa",
  "token_id": "019fc910-f063-7436-a37f-35d9f05ebc32"
}

Every value on this page comes from one real run, that plaintext included: it was redeemed seconds later on a throwaway local stack and has been dead ever since. A token from your stack is a live credential until something redeems it.

Step 2 — Give the agent a config file and start it

The agent takes its whole configuration from environment variables and an optional config file. Two of this lesson's legs live in that file, so write it first, next to a directory the log-forwarding leg will watch:

mkdir -p agent-logs
cat > plexd-config.yaml <<'EOF'
actions:
  enabled: false

log_fwd:
  file_patterns:
    - /var/log/agent/*.log
EOF

The file does two jobs. log_fwd.file_patterns gives the agent a log source: every line appended to a matching file inside the container becomes a candidate for the logs ingest pipeline — that is the seam Step 6 walks through. And actions.enabled: false keeps the remote-execution kill switch off: with a config file present, an unset actions.enabled means on, and control-plane-driven command execution is a different lesson's story.

Now one command starts the agent:

docker run --detach --name plexd-agent \
  --add-host=host.docker.internal:host-gateway \
  --user 0:0 \
  --cap-add NET_ADMIN \
  --volume "$PWD/plexd-config.yaml:/etc/plexd/config.yaml:ro" \
  --volume "$PWD/agent-logs:/var/log/agent" \
  --env PLEXD_API=http://host.docker.internal:8080 \
  --env PLEXD_PROJECT_ID="$PROJECT_ID" \
  --env PLEXD_RESOURCE_HANDLE=edge-agent-01 \
  --env PLEXD_REQUESTED_RESOURCE_ID=edge-agent-01 \
  --env PLEXD_BOOTSTRAP_TOKEN="$TOKEN" \
  --env PLEXD_POLICY_ENABLED=false \
  --env PLEXD_LOG_LEVEL=debug \
  ghcr.io/plexsphere/plexd:latest up

38f770594dc1599c1627a96cb4caca1e5ec3f8f15a2e7011ef0079cf154abc9b

docker run --detach prints the container id and returns; the agent is already working. The image's entrypoint is the plexd binary, so the trailing up selects the resident daemon mode — register once, then heartbeat, reconcile, and report on timers until stopped. It is the same mode the dev stack's in-cluster plexd fixtures run.

--user 0:0 --cap-add NET_ADMIN is what makes the data plane real. The image runs as user 65534 by default, and a capability added to the container only reaches a non-root process if the binary carries file capabilities — which the distroless plexd binary does not. So the two flags travel together: run as root, grant CAP_NET_ADMIN, and the agent can create its kernel WireGuard interface. Nothing else is needed — no /dev/net/tun, no --privileged.

PLEXD_RESOURCE_HANDLE and PLEXD_REQUESTED_RESOURCE_ID take the adoption path the previous lesson explained. Nothing is pre-created here: the agent's register call creates the Resource edge-agent-01 and its Node together.

PLEXD_POLICY_ENABLED=false keeps network-policy enforcement off. With CAP_NET_ADMIN granted the agent could install its deny-by-default nftables baseline inside the container's network namespace, but policy enforcement is its own leg with its own story — this lesson is about the control plane, the mesh, and the telemetry pipelines.

PLEXD_LOG_LEVEL=debug is set from the start because the ingest receipts you read in Step 6 are DEBUG lines. Raising the level later means restarting the agent, which costs another bootstrap token.

Why PLEXD_API is not http://plexsphere:8080. The in-cluster fixtures reach the API by Kubernetes Service name. Your container is a genuinely external process on your host's Docker network and has no such name. The kind cluster publishes the API on host port 8080 on all interfaces, so host.docker.internal reaches it from inside the container, and --add-host=host.docker.internal:host-gateway makes that name resolve on Linux Docker Engine. Docker Desktop provides the name already and ignores the flag, so one command works on both.

Step 3 — Watch it register

docker logs plexd-agent 2>&1 | head -10

time=2026-08-03T19:19:19.010Z level=INFO msg="starting plexd" version=v0.6.0 mode=node actions_enabled=false
time=2026-08-03T19:19:19.014Z level=INFO msg="health listener started" component=health listen=127.0.0.1:9101
time=2026-08-03T19:19:19.663Z level=INFO msg="registration successful" component=registration node_id=019fc910-f565-7d55-9258-61595da2c8ab mesh_ip=10.50.0.5
time=2026-08-03T19:19:19.663Z level=INFO msg=registered node_id=019fc910-f565-7d55-9258-61595da2c8ab mesh_ip=10.50.0.5
time=2026-08-03T19:19:19.663Z level=DEBUG msg="netlink interface created" component=wireguard interface=plexd0
time=2026-08-03T19:19:19.664Z level=INFO msg="wireguard interface created" component=wireguard interface=plexd0 listen_port=51820
time=2026-08-03T19:19:19.664Z level=DEBUG msg="address configured" component=wireguard interface=plexd0 address=10.50.0.5/24
time=2026-08-03T19:19:19.664Z level=DEBUG msg="interface brought up" component=wireguard interface=plexd0
time=2026-08-03T19:19:19.664Z level=INFO msg="wireguard interface configured" component=wireguard interface=plexd0 listen_port=51820 mesh_ip=10.50.0.5
time=2026-08-03T19:19:19.666Z level=INFO msg="generated new host key" path=/var/lib/plexd/ssh_host_ed25519_key component=tunnel

Under a second after start, registration successful carries a node_id and a mesh_ip. The allocator handed this agent 10.50.0.5 — the next free address after the dev stack's two in-cluster plexd fixtures on .1 and .2 and your two hand-enrolled Nodes on .3 and .4. That exact value holds when the previous lesson's two Nodes are the only hand enrolments on the stack; on a stack you have enrolled into more, expect a higher last octet.

One millisecond later the data plane comes up: the agent creates the kernel WireGuard interface plexd0, puts its mesh address on it, and opens UDP listen port 51820. That is CAP_NET_ADMIN at work — the same registration on a capability-less container logs a warning and carries on without a data plane.

One line you will not see repeated is a journald complaint. The image carries no journald, the agent notices exactly once, and the file source you configured in Step 2 is this lesson's log source:

time=2026-08-03T19:19:19.687Z level=INFO msg="journald not available, journald log source disabled"

Everything the previous lesson's Step 3 made you do by hand just happened on its own. Generating a keypair, assembling the register body, posting the bootstrap token, keeping the node secret key it got back, programming the WireGuard interface — the agent did all of it in about a second, and it will keep using that credential for as long as it runs.

Capture the Node id the agent logged; the next steps address the Node by it:

NODE_ID=$(docker logs plexd-agent 2>&1 | grep -m1 'msg=registered' \
  | grep -o 'node_id=[^ ]*' | cut -d= -f2)
echo "$NODE_ID"

019fc910-f565-7d55-9258-61595da2c8ab

Step 4 — Read the mesh as the operator

plexctl peer list --domain "$DOMAIN_ID"

NODE_ID                               MESH_IP    REACHABILITY
019fc90c-89a7-7530-b993-aa941d12c480  10.50.0.1  healthy
019fc90c-fcab-7b79-854b-cd39aa624d6e  10.50.0.2  healthy
019fc90f-e597-76bc-b11f-8f0508dc944f  10.50.0.3  never_reported
019fc90f-e9e5-7b45-acb7-28f5f0e30918  10.50.0.4  never_reported
019fc910-f565-7d55-9258-61595da2c8ab  10.50.0.5  healthy

Five rows where the previous lesson left four, and only one of the new rows is yours.

The 10.50.0.1 and 10.50.0.2 rows are the dev stack's own in-cluster plexd fixture Pods. They enrolled the moment the stack came up, before you ran a single command, which is why they hold the first addresses and your hand-enrolled Nodes start at .3. Every shared environment has neighbours like these: agents nobody on the lesson started, already in the inventory. You will not touch them.

Now the part worth slowing down for. Three rows read healthy and two read never_reported, and the split is the whole point of the column. Your agent's row earns healthy: it heartbeats every 30 seconds and the platform admits each one, and the two in-cluster fixtures do the same. The two Nodes you enrolled by hand in the previous lesson stopped at registration and no agent ever spoke for them, so they read never_reported, the verdict for a Node the platform has never heard from.

Carry that distinction into an incident, because the two shapes ask for different work. never_reported says the agent was never started, so start it. stale and unreachable say a Node that was working went quiet, so find out what changed on it. Acme Corp's Domain configuration expects a heartbeat every 30s, calls a Node stale after 1m30s of silence and unreachable after 5m, and a Node reaches either verdict only after it has reported at least once. What this table shows you is heartbeat history, not a liveness probe. The mesh reachability context has the state machine and the per-Domain thresholds in full.

Step 5 — Read the agent's own words

The mesh view is the platform's opinion of the Node. Node state is the Node's own, so read that next:

plexctl state get --node "$NODE_ID"

METADATA
  (none)

DATA
  (none)

REPORTS
KEY                  VALUE                                                                                                                                                                                                                    WORKLOAD_TAG
status.bridge        {"enabled":false,"access_interface":"","active_routes":0,"relay_enabled":false,"active_relay_sessions":0,"ingress_enabled":false,"active_ingress_rules":0,"site_to_site_enabled":false,"active_site_to_site_tunnels":0}
status.ingress       {"enabled":false,"rule_count":0,"connection_count":0,"acme_enabled":false}
status.mesh          {"interface":"plexd0","peer_count":4,"listen_port":51820}
status.site-to-site  {"enabled":false,"tunnel_count":0}
status.user-access   {"enabled":false,"interface_name":"","peer_count":0,"listen_port":0}

The three buckets are the node-state read surface. Metadata and data are empty because they are the operator's side and you have written nothing there. Reports is the agent's outbound channel: every status.* key was authored and pushed by the agent itself, and no operator wrote a byte of it.

status.mesh is the data plane reporting for duty: interface plexd0, listen port 51820, and peer_count: 4 — the four other mesh members from Step 4, each programmed onto the interface with its public key and its mesh address as the allowed range. The previous lesson's curl enrolments put those keys into the inventory; the agent's reconcile loop pulled them back out and wrote them into the kernel.

Programmed peers are not yet live tunnels, and it is worth being precise about the difference. A WireGuard handshake additionally needs a reachable endpoint for the peer, and this single-host topology offers none: the in-cluster fixtures publish no UDP path out of the kind cluster, and your hand-enrolled Nodes were only ever curl calls. What you are reading is everything the control plane can program from inventory alone — key material, addressing, and peer topology, delivered to a kernel interface within a second of registration.

Right after registration the reports bucket can still render (none). The agent fills it on its own cycle: wait one heartbeat interval, 30 seconds, and re-run the command.

Step 6 — Follow a log line into the platform

The reports bucket is state; telemetry is the other direction, and this is the leg you configured in Step 2. The agent watches /var/log/agent/*.log inside the container, and that directory is your agent-logs/ on the host. Write one line into it:

echo "line one from the operator's shell" >> agent-logs/app.log

The agent collects new lines every 10 seconds and reports batches every 30, and it logs a receipt only after the platform accepted the batch:

docker logs plexd-agent 2>&1 | grep 'logs reported'

time=2026-08-03T19:19:49.705Z level=DEBUG msg="logs reported to platform" component=logfwd accepted_at=2026-08-03T19:19:49.704Z

Accepted is not yet readable — ingest lands in a buffer, and a routing consumer drains it onward to the stack's logs backend. Close the loop from the operator's chair: query the Domain's logs for your Node's stream. The window is computed from your own clock — the last fifteen minutes up to now — so the command runs unchanged whenever you reach this step:

FROM=$(date -u -v-15M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '-15 min' +%Y-%m-%dT%H:%M:%SZ)

plexctl logs query --domain "$DOMAIN_ID" \
  --query '{node="'"$NODE_ID"'"}' \
  --from "$FROM" --to "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output json | jq -r '.body | fromjson | .data.result[].values[][1]'

{"severity":"info","unit":"/var/log/agent/app.log","hostname":"38f770594dc1","message":"line one from the operator's shell","timestamp":"2026-08-03T19:19:49.691926298Z"}

There it is — the line you typed a minute ago, back out of the platform. The query language is LogQL; {node="…"} selects the stream by the Node label the routing layer stamps on every record, alongside the Domain and Project. The record itself is the agent's verbatim shipment: the source file as unit, the container id as hostname, and your text as message. Give the pipeline up to a minute end to end before you conclude anything; the observability ingest context has the full path.

The other two pipelines each have a story too. Metrics posts on its own 60-second timer, and its receipts are the same accepted-only kind — the first lands about a minute after registration:

docker logs plexd-agent 2>&1 | grep 'metrics reported'

time=2026-08-03T19:20:19.897Z level=DEBUG msg="metrics reported to platform" component=metrics accepted_at=2026-08-03T19:20:19.896Z
time=2026-08-03T19:21:19.855Z level=DEBUG msg="metrics reported to platform" component=metrics accepted_at=2026-08-03T19:21:19.855Z

The audit pipeline has one candidate and drops it. The audit ingest contract admits a closed set of sources, auditd and k8s, and the agent's own process-start entry belongs to neither, so the agent discards it client-side instead of posting a batch the platform would refuse:

time=2026-08-03T19:19:34.688Z level=WARN msg="auditfwd: skipping audit entry with unknown source" component=auditfwd source=process
time=2026-08-03T19:19:34.689Z level=WARN msg="dropping audit records" component=auditfwd reason="source outside the audit ingest contract" dropped=1 dropped_total=1

On this container the receipt stream is therefore metrics plus logs. The same agent on a host running auditd lights up the third leg.

Troubleshooting

Consumed or expired token

Re-running the docker run with the same $TOKEN fails registration and the container exits. docker logs plexd-agent shows:

Error: plexd up: registration: registration: register: api: HTTP 403 (token_consumed): POST /v1/register: the BootstrapToken has already been redeemed. (correlation_id=d8c9861f-fd2b-450e-ad82-09e7ff5fbd94)

An expired token fails the same way, with token_expired. Mint a fresh token (Step 1), remove the dead container with docker rm --force plexd-agent, and re-run Step 2. A bootstrap token is single-use by design; nothing replays one.

The API is unreachable

An agent that cannot reach PLEXD_API does not exit. It retries registration with growing backoff, so the container stays up and quiet while nothing happens. With PLEXD_API pointed at a closed port:

time=2026-08-03T19:26:49.059Z level=WARN msg="registration attempt failed, retrying" component=registration attempt=1 error="Post \"http://host.docker.internal:9999/v1/register\": dial tcp 192.168.65.254:9999: connect: connection refused" delay=904.345877ms
time=2026-08-03T19:26:49.968Z level=WARN msg="registration attempt failed, retrying" component=registration attempt=2 error="Post \"http://host.docker.internal:9999/v1/register\": dial tcp 192.168.65.254:9999: connect: connection refused" delay=1.790315587s

Confirm the stack answers on http://localhost:8080/readyz from your host. On Linux Docker Engine also confirm the --add-host=host.docker.internal:host-gateway flag is present; without it the name does not resolve there at all.

A stale local image

docker run does not re-pull a latest tag that is already on your host, and older releases fail in two distinct ways. An agent older than v0.4.0 registers fine and then loops an unauthorized heartbeat roughly every 30 seconds:

time=2026-07-29T06:14:21.520Z level=ERROR msg="agent: heartbeat: unauthorized" component=heartbeat

And on Apple Silicon, the v0.4.0 and v0.5.0 images carried an x86-64 binary in their arm64 variant. It runs under emulation, but the WireGuard setup fails even with CAP_NET_ADMIN granted:

time=2026-08-03T15:35:47.284Z level=WARN msg="wireguard setup failed, continuing without WireGuard" error="wireguard: setup: wireguard: create interface: open wgctrl: socket: protocol not supported"

Both shapes have the same cure. Pull the current image, drop the container, and start over from Step 1 with a fresh token:

docker pull ghcr.io/plexsphere/plexd:latest
docker rm --force plexd-agent

Re-running the lesson with the same handle

A Resource keeps the Node that first adopted it, so a second registration against the handle edge-agent-01 on the same stack cannot succeed — even with a fresh token. The agent retries a server error that does not name the conflict yet:

time=2026-08-03T19:26:56.941Z level=WARN msg="registration attempt failed, retrying" component=registration attempt=1 error="api: HTTP 500: POST /v1/register: an unexpected error occurred. (correlation_id=9fbdb410-f282-485a-8d90-f067541734cd)" delay=1.213427398s
time=2026-08-03T19:26:59.225Z level=WARN msg="registration attempt failed, retrying" component=registration attempt=2 error="api: HTTP 500: POST /v1/register: an unexpected error occurred. (correlation_id=275ca348-e247-4deb-846d-57e23098d77d)" delay=1.519827759s

Remove the container and re-run Step 2 with a different handle — for example edge-agent-02 — or reset the stack for a clean slate. The first run's Node row and its Resource stay in the inventory either way.

No ingest receipts appear

If registration and heartbeats work but no metrics reported to platform or logs reported to platform line ever shows, your dev stack predates the observability ingest wiring. If the receipts show but the plexctl logs query in Step 6 stays empty, the stack predates the logs routing sink instead. Both cures are the same: re-apply the dev manifests so the stack carries PLEXSPHERE_OBS_NATS_URL and PLEXSPHERE_OBS_LOKI_URL; the reset recipe is in the dev stack runbook.

What you learned

• A real agent is the previous lesson's curl steps running forever. It registers once, then heartbeats, reconciles, and reports on timers — the same contract, driven by a clock instead of your keyboard.
• The data plane costs exactly one capability. With CAP_NET_ADMIN (and root, because the image is non-root by default) the agent programs interface, address, listen port, and all four peers within a second of registering — and without it, the same agent simply carries on with no data plane.
• Reachability is driven by heartbeats. Your agent holds healthy because the platform keeps admitting its heartbeats, and the hand-enrolled Nodes read never_reported because no agent has ever sent one for them. A Node that was never started and a Node that went quiet get different verdicts, and they ask for different work.
• The reports bucket is the agent's outbound channel. Every status.* key under REPORTS was authored and pushed by the agent; the operator reads it, and writes to the metadata and data buckets instead.
• Ingest is receipt-confirmed and round-trips. The agent logs a receipt only for a batch the platform accepted, and a log line you write on your own host comes back out of plexctl logs query with the Node's stream labels on it.

Where to go next

The agent runs until you stop it. docker rm --force plexd-agent removes it and the Node row stays in the inventory: the container only ever held the credential. Running the lesson again needs a freshly minted token and a fresh resource handle, because the one you used stays bound to this run's Node — the troubleshooting section above has the details.

• Keep learning by doing — Read the backup catalog opens the platform track: step into the platform operator's chair and read what the platform backs up and in what order it would be restored.

Or pick the quadrant that matches what you need now:

• You want the exact contract the agent just spoke — the plexd agent contract reference lists every operation, its credential, and the gaps that are still open.
• You want to understand why reachability is shaped the way Step 4 showed — the mesh reachability context covers the heartbeat seam, the four-state machine, and the per-Domain thresholds.
• You want the telemetry path Step 6 walked — the observability ingest context covers the node-facing front door, and its routing and query companions cover the trip onward to the backends and back out.
• You want to know the stack you ran it against — the dev stack runbook documents what make dev brings up, including the in-cluster plexd fixtures that hold 10.50.0.1 and 10.50.0.2.