internal/platform/worker

internal/platform/worker supervises long-running background loops so a worker that crashes cannot stay silently dead for the process lifetime. A loop that crashes is restarted with backoff, and a loop that keeps crashing turns /readyz red. The API composition root runs every background loop under one Supervisor.

This document is the authoritative reference for:

• Worker and Options
• Supervision behaviour
• Readiness probe
• Bounded shutdown join
• Where the composition root wires it

See also:

• docs/reference/platform/db.md — the shared pool the same composition root constructs once and threads through every surface.
• docs/operations/failure-modes.md — the readiness model and the Background workers runbook a permanently dead worker feeds into.

Worker and Options

A Worker is a single supervised background loop:

Field	Type	Purpose
Name	string	Identifies the worker in logs and the probe error. Must be non-empty and unique within a Supervisor.
Run	func(ctx context.Context) error	The loop. It MUST return promptly once ctx is cancelled. A non-context error, a nil return while ctx is still live, or a panic is treated as a crash.

Options tune a Supervisor; the zero value is usable because New fills every field with its documented default.

Field	Type	Default	Purpose
Logger	*slog.Logger	slog.Default()	Receives restart and death events.
MaxRestarts	int	5	Restart budget within a single unhealthy streak before permanent death.
BaseBackoff	time.Duration	1s	First restart delay; doubles each restart.
MaxBackoff	time.Duration	30s	Caps the exponential restart delay.
HealthyAfter	time.Duration	1m	Uptime beyond which a worker's restart budget resets.
JoinTimeout	time.Duration	10s	Bounds Wait's post-cancel join.

Supervision behaviour

Register adds workers before Start; it rejects an empty name, a nil Run, a duplicate name, or registration after Start with a matching sentinel (ErrEmptyName, ErrNilRun, ErrDuplicateName, ErrAlreadyStarted).

Start(ctx) launches each worker in its own supervised goroutine and returns immediately. For each worker the supervisor:

• restarts the loop with exponential backoff plus equal jitter when it exits while ctx is still live, logging every restart at WARN;
• resets the restart budget once the loop has run longer than HealthyAfter, so a rare transient crash does not accumulate toward permanent death;
• marks the worker permanently dead after the restart budget is exhausted, logging at ERROR and flipping the readiness probe red;
• recovers panics — logging the value and stack — and treats them as crashes so one worker cannot abort the process.

sup := worker.New(worker.Options{Logger: logger})
if err := sup.Register(worker.Worker{Name: "endpoint-sweeper", Run: sweep}); err != nil {
    return err
}
if err := sup.Start(ctx); err != nil {
    return err
}

Readiness probe

Supervisor.HealthProbe matches internal/platform/health's ProbeFunc signature without importing the health package. It returns nil while every worker is running or restarting, and a non-nil error naming each permanently-dead worker once one exhausts its budget — so /readyz flips red instead of a dead sweeper degrading the replica silently.

registry.Register("workers", sup.HealthProbe)

Bounded shutdown join

Wait(ctx) blocks until every supervised goroutine has returned or the join deadline fires, whichever comes first. The composition root cancels the workers' context and then calls Wait, giving a worker mid-iteration JoinTimeout to unwind before shutdown moves on without it (see Shutdown order). It returns ErrJoinTimeout when the deadline fires with goroutines still running, or the caller's context error when that context is cancelled first, so shutdown neither abandons a sweep mid-flight nor hangs forever on a worker that ignores cancellation.

Where the composition root wires it

Run in ../../../cmd/plexsphere/app.go builds one Supervisor right after bootstrap and registers each background loop on it under a fixed name. A loop whose wiring the boot did not build, for example because PLEXSPHERE_DSN or the NATS URL is unset, is not registered. A leader-gated loop runs as a worker on every replica and does its work only on the background leader, see Which loops run on the leader.

Worker	Leader-gated	Loop
background-leader	no	Takes part in the background leader election.
audit-archiver	yes	Drains the audit chains to the object store.
capacity-collector	no	Samples capacity and tracks its hysteresis.
invitations-expire	yes	Expires the elapsed pending invitations.
approvals-expire	yes	Expires the proposals past their deadline.
actions-timeout	yes	Times out the expired live executions.
access-session-sweep	yes	Revokes the expired and idle sessions.
auth-state-sweep	yes	Drains the elapsed sign-in states, browser sessions and device authorizations.
rate-budget-sweep	yes	Reclaims the rate-budget token buckets idle for longer than an hour.
sse-relay	no	Drains the signed event bus outbox onto JetStream.
sse-signing-key-rotation-watcher	no	Drops the cached signing keys of every Domain that rotated.
sse-nonce-sweep	yes	Deletes the signed-event-bus nonces last seen before the freshness window.
credentials-sweeper	yes	Expires the credentials past their TTL.
cloud-credentials-sweeper	yes	Expires the cloud credentials past their TTL.
management-fleet-reconcile	yes	Places Projects and converges cluster packages.
provisioning-broker-reconcile	yes	Reconciles the declared provisioned resources.
observability-routing-consume	no	Runs the observability routing consume loops and reconciles their durables.
artifacts-refresh	yes	Re-indexes the artifact catalogue.
blueprint-catalogs-tracking	yes	Re-imports the track-tag catalog sources.
peers-endpoint-sweeper	yes	Tombstones the endpoint observations past TTL.
peers-relay-assigner	no	Re-flips the relay assigner's readiness flag on every tick.
reachability-evaluator	yes	Runs the stale and the recovery pass over Nodes.
authz-outbox-consumer	yes	Mirrors the outbox rows into SpiceDB.

Start and readiness

Just before it mounts the router, Run registers sup.HealthProbe on /readyz under the probe name workers and calls sup.Start. The leader election starts there as well; boot-time passes are not gated on leadership.

The workers probe is not boot-latched. It passes while every worker runs or restarts and fails once one worker is permanently dead. /readyz then answers 503 unhealthy with this entry, and health_probe_up{probe="workers"} reads 0:

{"name":"workers","ok":false,"error":"worker: 1 worker(s) permanently dead: <name>"}

A readiness failure takes the replica out of its Service but does not restart the container, so recovery is a Pod restart. The failure-modes runbook covers detection and recovery.

Test seam

Config.WorkerOptions passes Options to that Supervisor. It is a test seam: the zero value means the package defaults, main.go never sets it, and a nil Logger is replaced with the boot logger. No environment variable tunes the supervisor.

Shutdown order

SIGTERM cancels Run's context. The workers' context is detached from it, so the loops keep running while the HTTP server drains. Shutdown then runs in four phases:

1. Serve returns. health.Registry.Drain() has flipped /readyz to 503 draining, and http.Server.Shutdown has run under the grace period. Run logs server stopped.
2. Run cancels the workers' context.
3. Run joins the workers through Wait for at most JoinTimeout and logs background workers stopped. When the deadline fires first, it logs background workers did not stop within the join timeout at WARN instead. The warning does not change the exit code, so server.ErrForceClose from a missed grace deadline stays the only path to exit code 2.
4. The shared resources close newest first. A failed close is logged at WARN as shared resource close failed, with the resource name in the resource attribute, and the remaining resources still close. Run then logs shared resources closed and returns.

The shared resources close in this order. A resource the boot did not open is not on the stack.

Order	resource	Closes
1	secrets-openbao-client	The OpenBao client the secrets surface dialled.
2	cloud-credentials-openbao-client	The OpenBao client the cloud credentials surface built.
3	observability-routing-clients	The routing pipeline's NATS client and its OpenBao credential-store client.
4	observability-ingest-nats-client	The ingest front door's NATS client.
5	sse-nats-client	The signed event bus NATS client.
6	jetstream-probe-client	The NATS client behind the jetstream readiness probe.
7	db-primary-probe-pool	The Postgres pool behind the db-primary readiness probe.
8	db-pools	The shared interactive, worker and bulk Postgres pools.
9	spicedb-client	The SpiceDB client.

A Run that fails during boot takes the same teardown and closes the resources it opened; Wait returns at once when Start never ran.

With the default 25 s grace period, phases 1 and 3 take at most 35 s together. The API Pod's terminationGracePeriodSeconds is 60 s. Each of the four NATS clients gets at most 3 s to drain and is then closed without finishing the drain: the workers are joined by then, so a client has nothing left to flush, and the bound keeps a NATS server that stopped answering from pushing the close phase past the grace period. After a join timeout a worker can still hold a pooled Postgres connection, and closing a pool waits for its acquired connections to return, so the db-pools close can run until Kubernetes kills the Pod.

Cross-references

• ../../contributing/layout.md — the bounded-context map locating this package in the codebase.
• ../../../internal/platform/worker/ — the package source.
• ../index.md — the Reference quadrant index.