Failure modes and degradation

This runbook describes how a plexsphere API replica behaves when one of its backing dependencies is briefly unavailable, and how to detect, interpret, and recover from each case. The governing principle is "plexsphere down ≠ mesh down": a transient SpiceDB, Postgres, NATS, OpenBao, or Signing-Service outage must degrade a single surface, not pull every replica out of the Kubernetes Service and convert a partial outage into a total control-plane outage.

Two design rules make that hold:

1. Boot-gate-then-degrade readiness. A dependency probe gates a fresh replica out of rotation until that dependency is first reachable, but once it has succeeded once it never fails the replica again — a later outage is reported as informational and the replica keeps serving every surface that does not depend on the failed dependency.
2. Structured degradation, never a generic 5xx. Every dependency outage surfaces as an RFC 9457 problem document with a stable, machine-readable code, so a client can branch on the code (back off, fail over, page) instead of string-matching opaque 500s. The underlying error never reaches the response body — the cause is in the server's structured log record only, and the problem detail is a fixed, generic, retry-oriented string.

The readiness model

GET /readyz aggregates the registered probes into one of three top-level states:

status	HTTP	Meaning
ready	200	Every probe passes.
degraded	200	Every non-latched probe passes; one or more boot-latched probes failed after their first success. The replica stays in the Service.
unhealthy	503	A non-latched probe failed, or a boot-latched probe has not yet succeeded once. The replica is pulled from the Service.

A boot-latched probe is registered with the platform health registry's boot-latch option. Before its first success it gates hard (/readyz returns 503 unhealthy); after its first success a later failure is reported with "ok": false, "informational": true in that probe's entry, and the top-level status drops to degraded (still HTTP 200) rather than unhealthy.

The boot-latched dependency probes on a freshly booted dev stack are:

Probe name	Dependency
db-primary	Postgres (primary datastore)
jetstream	NATS JetStream (signed event bus)
spicedb-readiness	SpiceDB (ReBAC engine)
secrets-reconcile	OpenBao (secret backend)

No probe is registered for the Signing Service. The signer is dialed only on the issuance hot path; its health is alertable from the issuance error counters and the signing_unavailable problem code rather than from a boot probe — see Signing Service below.

Each probe's pass/fail state is also published as a Prometheus gauge, health_probe_up{probe="<name>"}, which reads 1 while the probe passes and 0 while it fails (including the post-latch informational case). Alert on health_probe_up == 0 for any probe to surface a degraded dependency, and join it with the per-dependency counters below to scope the blast radius.

curl http://<replica>:8080/readyz on a healthy replica lists entries for db-primary, jetstream, spicedb-readiness, and secrets-reconcile (among the non-dependency probes); the absence of one of those four is itself a wiring regression worth investigating.

The seven background sweep probes follow a different contract. invitations-expire, approvals-expire, actions-timeout, access-session-sweeper, management-fleet-reconcile, provisioning-broker-reconcile, and artifacts-refresh each report the outcome of the last pass the sweep loop ran on this replica. The boot pass runs on every replica and gates readiness before the API serves its routes. After boot the loops tick only on the elected background leader, so a standby runs no pass, has no outcome of its own, and its probe passes. None of the seven is boot-latched, so a failing pass takes the replica that ran it out of the Service until a later pass succeeds. A probe evaluation never runs a sweep itself, which is why a three-second kubelet poll does not turn into sweep work. See Background leader.

The workers probe covers the background loops themselves. It is not boot-latched: it passes while every worker runs or restarts after a crash, and it fails once one worker exhausts its restart budget (five restarts within one unhealthy streak; a worker that ran for a minute before it crashed starts a fresh budget). A permanently dead worker therefore takes its replica out of the Service. See Background workers.

GET /metrics gathers at most four scrapes at once and answers 503 to one that arrives while four are still gathering, with a 10-second budget for each gather. Some collectors read the database, so a scrape costs a query rather than a memory read, and the endpoint takes no credentials — hence the bound. Four clears the usual scraper population (a highly-available Prometheus pair, a federating scraper, an operator's curl), so a 503 in a scraper's logs means more than four scrapes overlapped on one replica, not that the replica is unhealthy. Persistent 503s point at a scrape interval shorter than a gather takes, or at more scrapers than the bound admits.

Boot

An API replica binds its address at the start of its boot, once its logger and health registry exist, and logs server starting with the bound addr. From that moment until it serves its routes:

• GET /livez answers 200, so the liveness probe does not restart a replica that is still booting.
• GET /readyz answers 503 unhealthy. The boot probe fails with the API is still booting, and every probe registered so far is evaluated as well, so the body names what the boot waits for. health_probe_up{probe="boot"} reads 0 until the API serves its routes.
• Every other request, /metrics included, answers 503 with an application/problem+json body and Retry-After: 5. The body carries the request path as instance and the X-Correlation-Id echo as correlation_id, but no code, because no surface serves yet. Prometheus therefore sees the target down during boot.

Once the replica serves its routes, the boot probe passes and the replica logs boot complete with the same addr.

Waiting for SpiceDB. On a fresh install SpiceDB starts beside the API. The boot applies the authorization schema before the API serves its routes, and while SpiceDB does not answer (gRPC Unavailable or DeadlineExceeded, or an attempt that ran into its 10-second deadline) it waits and tries again. The pause starts at 0.5 s and doubles up to 5 s. Before each pause the replica logs at INFO waiting for SpiceDB before applying the authorization schema with the attempt number, the backoff it pauses for and the attempt's error, and /readyz lists spicedb-readiness failing next to boot. The SpiceDB client retries its connection at least every six seconds, so a waiting replica applies the schema, and becomes ready, within about eleven seconds of SpiceDB answering, however long it waited. The replica keeps a restart count of 0 throughout.

What still ends the boot. A replica exits with code 1 when SpiceDB rejects the API instead of not answering: a rejected credential (Unauthenticated or PermissionDenied; SpiceDB answers a wrong SPICEDB_PRESHARED_KEY with PermissionDenied invalid preshared key) or a schema SpiceDB refuses (ErrSchemaInvalid). Waiting cannot fix either. A TLS handshake failure, such as a SPICEDB_TLS_CA_PATH that does not hold the CA of SpiceDB's certificate, reaches the API as Unavailable, so the boot waits on it: every waiting line and the spicedb-readiness entry on /readyz carry the x509 error.

A hung boot. A boot step that never returns leaves the replica live and not ready, with boot failing; the liveness probe does not restart it. Read the replica's log for the last boot step it reached. A SIGTERM during boot stops the replica with exit code 1.

SpiceDB (authorization)

SpiceDB is the ReBAC engine every gated operation consults.

Detection. health_probe_up{probe="spicedb-readiness"} drops to 0 and that probe reports informational on /readyz. Any ReBAC-gated operation that cannot be served from the stale cache returns HTTP 503 with problem code authz_unavailable (never a generic 500 — no handler fails open). The stale-serve counter authz_stale_check_served_total{outcome="allow|deny"} and the gauge authz_stale_check_cache_size show how much load the last-known-good cache is absorbing during the outage.

Degraded behaviour. A read-mostly Check whose (subject, relation, object) tuple is in the last-known-good cache, was cached within SPICEDB_CHECK_STALE_TTL, and carries an empty caveat context is served from cache and audited with the distinct reason stale_cache. Three guards keep this safe:

• Caveated checks (a non-empty caveat context — e.g. a time- or CIDR-sensitive grant) are never served stale, because a stale decision could silently become a fail-open on a condition that has since changed.
• Write, Delete, LookupResources, and LookupSubjects never consult the cache — they return authz_unavailable immediately, so a mutation never proceeds against a stale view.
• The cache is TTL-bounded and capped at SPICEDB_CHECK_STALE_MAX_ENTRIES. Once full, warming a new triple evicts the oldest cached decision (a least-recently-inserted drop), so the cache size stays bounded and authz_stale_check_cache_size holds at the cap. Evicting the oldest keeps the freshest decisions available during an outage; either way an evicted triple simply misses and surfaces authz_unavailable, never a fail-open.

Tuning. SPICEDB_CHECK_STALE_TTL (default 30s; 0 disables the stale cache entirely) bounds how old a served decision may be. SPICEDB_CHECK_STALE_MAX_ENTRIES (default 10000) bounds the cache size. Shorten the TTL to tighten the staleness window at the cost of more authz_unavailable responses during an outage; lengthen it to ride out longer SpiceDB blips at the cost of a longer stale window.

Recovery. When SpiceDB returns, the next fresh Check repopulates the cache and authz_stale_check_served_total stops incrementing; health_probe_up{probe="spicedb-readiness"} returns to 1 and /readyz returns to ready.

Postgres (datastore / node state)

Postgres is the primary datastore. The read-mostly node-state path survives a brief outage via a per-node stale-snapshot cache; write funnels do not.

Detection. health_probe_up{probe="db-primary"} drops to 0 and that probe reports informational on /readyz (which stays HTTP 200 degraded). GET /v1/nodes/{id}/state for a Node with no fresh-enough cached snapshot returns HTTP 503 with problem codedatastore_unavailable. The counter state_stale_snapshot_served_total and the gauge state_stale_snapshot_cache_size show how many reads are being served from cache.

Degraded behaviour. A Node whose last successful snapshot is cached and fresher than PLEXSPHERE_STATE_STALE_TTL keeps getting HTTP 200 with that snapshot, so an already-enrolled node keeps its desired state instead of seeing a generic 500. A cold or expired cache entry returns 503 datastore_unavailable rather than a stale serve. Writes keep returning 5xx during a Postgres outage by design — the contract for write funnels is only that they fail loudly, not that they degrade to a structured outage code.

Tuning. PLEXSPHERE_STATE_STALE_TTL (default 5m; 0 disables the stale cache) bounds how old a served snapshot may be. PLEXSPHERE_STATE_STALE_MAX_ENTRIES (default 10000) caps the cache (evicts the oldest cached snapshot when full, same as the authz cache).

Rate-budgeted requests. Secret reads, session issuances and observability ingest pushes admit against a rate budget held in Postgres, so during an outage the admit gate cannot answer. Each fails closed with the generic HTTP 500 a write funnel answers, never a 429, and the cause is logged. The distinction is deliberate: a 429 would tell every plexd agent in the fleet to back off from a cap none of them reached, and the rate-limited counters would show a fleet-wide breach that never happened. plexsphere_secret_fetch_rate_limited_total and the ingest rejects_total throttle reasons therefore stay flat through an outage. No operator action is needed on recovery: the next request admits against the bucket it finds, and a bucket nobody could reach has been refilling the whole time. Full reference: internal/platform/db/ratebudget.

Signed-in operators. Every cookie-authenticated request resolves its session against Postgres, so during an outage /v1/* answers HTTP 503 with problem detail session store unavailable rather than 401. The distinction is deliberate: a 401 would tell every console user their session had ended, their browser would drop the cookie, and the recovery would be a fleet-wide re-authentication instead of a retry. No response on this path clears the session cookie, so when Postgres returns the next request succeeds with the session the operator already had. Sign-in and the device grant are unavailable for the duration — there is nowhere to record their state — and answer 500.

Signed node event bus. The envelope replay window is a Postgres table, so during an outage the bus cannot tell a fresh envelope from a replay. It refuses the envelope rather than admitting it and closes the open GET /v1/nodes/{id}/events streams; a node agent reconnects with its Last-Event-ID and the envelope is delivered once the store answers again. The relay and the signing-key rotation watcher back off on their own loops and resume where they stopped, so no outbox row is skipped. The reconciliation pull keeps serving its cached snapshots throughout, which is the channel a node converges on either way.

Recovery. When Postgres returns, the next successful snapshot read refreshes the cache, state_stale_snapshot_served_total stops incrementing, and fresh (non-cached) snapshots resume. Signed-in operators resume on their next request without signing in again. Node streams reopen on the agents' own reconnect.

Connection budget. The server holds Postgres connections in three shared pools, not one per surface: interactive (HTTP paths, default MaxConns=20), worker (sweepers / reconcilers / consumers / seeds, default 10), and bulk (backup / migration / one-shot, default 5). Each pool opens lazily and at most once, so a replica's ceiling is interactive + worker + bulk (default 35) rather than the sum of every surface, and the idle floor is a handful of connections. Size the tiers with PLEXSPHERE_POOL_{INTERACTIVE,WORKER,BULK}_{MIN,MAX}_CONNS — an empty knob keeps the default, MIN above MAX refuses the boot. Keep (interactive_max + worker_max + bulk_max) × replicas plus one db-primary probe connection per replica comfortably under the server's max_connections. Full reference: internal/platform/db § Composition-root connection pools.

NATS JetStream (signed event bus)

JetStream backs the Server-Sent-Events stream that pushes signed state deltas to node agents. The state pull path does not depend on it.

Detection. health_probe_up{probe="jetstream"} drops to 0 and that probe reports informational on /readyz (HTTP 200 degraded). A fresh GET /v1/nodes/{id}/events subscribe returns HTTP 503 with problem code event_stream_unavailable and a fixed generic detail — the underlying subscribe error is in the server log record only, never in the response body.

Degraded behaviour. While JetStream is down, the SSE subscribe fails fast with event_stream_unavailable, but GET /v1/nodes/{id}/state keeps returning HTTP 200 — the state path reads the snapshot and never touches JetStream, so a node agent that polls /state keeps its desired configuration. "plexsphere down ≠ mesh down" holds: existing peer connections and already-pushed deltas are unaffected.

Recovery. When NATS returns, the messaging client reconnects, a fresh subscribe succeeds, and the relay drains any outbox rows that accumulated during the outage so replay resumes.

OpenBao (secret backend)

OpenBao stores Cloud Credentials and Project secrets. A sealed or unreachable backend degrades the secret-read edge and stalls the credential sweepers.

Detection. A node-secret read against a sealed or unreachable OpenBao returns HTTP 503 with problem code openbao_unavailable. The credential-broker sweep ticks that are blocked by the outage increment secretstore_unavailable_errors_total{consumer="<broker>"} and each logs a WARN naming the affected subsystem and the operator action. After the secrets-reconcile probe's first success, an OpenBao outage leaves /readyz at HTTP 200 (the probe reports informational), so a sealed backend does not pull the replica out of rotation.

Degraded behaviour. New secret reads fail with openbao_unavailable, so Cloud Credential materialisation into Kubernetes Secrets and Project secret reads from nodes stall and new provisioning cannot start. Already-delivered, NSK-wrapped secrets on nodes remain usable — the ciphertext is local to plexd — and already-issued sessions are unaffected.

Divergence note. The project README's degradation table uses the illustrative string secret_backend_unavailable for this case. The shipped wire code is openbao_unavailable, and the wire code is authoritative: renaming it would be a breaking change for zero behavioural gain, so the divergence is documented here and in the OpenAPI Problem.code description rather than reconciled. Operators and clients branch on openbao_unavailable.

Recovery. Unseal (or restore reachability to) OpenBao; the next secret read succeeds, secretstore_unavailable_errors_total stops incrementing, and the credential sweepers resume on their next tick. See the secrets context for the secret-read contract.

Signing Service

The Signing Service mints session tokens and signs SSE event frames. Unlike the four dependencies above it has no boot probe — it is dialed only on the issuance and publish hot paths.

Detection. POST /v1/projects/{project_id}/sessions during a signer outage returns HTTP 503 with problem code signing_unavailable; no Session is persisted and no token is issued, so a retry after recovery is safe (idempotent). On the publish side, the SSE relay increments two counters that are the load-bearing alerting signals for a signer outage:

• sse_publisher_sign_failures_total{reason} — a publish attempt could not sign an event frame.
• sse_relay_skip_total{reason} — the relay loop skipped an outbox row (e.g. reason="no_signing_key") and left it unacked for retry.

Alert on a sustained increase in either counter together with the signing_unavailable 503 rate, rather than on a probe. The signer outage deliberately does not raise a tenancy integrity_alert outbox event — that aggregate is node-agent-scoped (binary/hook checksums, SSH host keys), and operator paging for a control-plane signer outage maps to these metrics and the structured 503 instead.

Degraded behaviour. New session issuance and new signed-event publishes fail; already-issued sessions keep working to their expiry, and the relay cursor does not advance past an unsigned row, so no event is lost.

Recovery. When the signer returns, issuance succeeds on retry and the relay drains the rows it skipped. See the SSE context for the publish/relay contract and the observability ingest context for the back-pressure path that shares the same structured-degradation philosophy.

Background leader

One replica of a deployment runs the background sweep loops. It is the replica whose elector holds a PostgreSQL session advisory lock. The background leader reference covers the lock key, the election cadence, and which loops are gated.

Detection. plexsphere_background_leader reads 1 on the leader and 0 on every other replica, so sum(plexsphere_background_leader) over a deployment is 1 in steady state. A sum of 0 means no replica holds the lock and no sweep is running. A sum of 2 is the handover overlap window and clears within a few election intervals. Alert on a sum that stays off 1 for longer than that.

Name the holder from the database:

SELECT pid, granted
FROM pg_locks
WHERE locktype = 'advisory'
  AND classid = 1886156915
  AND objid = 1
  AND objsubid = 2
  AND database = (SELECT oid FROM pg_database WHERE datname = current_database());

pg_stat_activity maps the returned pid to a client address, which identifies the leading Pod. No row means no replica leads at the moment.

Degraded behaviour. A Postgres outage stops the loops. The lock lives on one database session, so the leader loses it when that connection breaks, demotes itself at its next check, and every standby's acquire attempt fails while the server is unreachable. Rows that fall due during the outage stay due: invitations and proposals are not expired, executions are not timed out, sessions are not revoked. The seven sweep probes keep reporting their last pass, so they do not turn red for the passes that never ran. db-primary is the probe that reports the outage, and plexsphere_background_leader sums to 0 for its duration.

A leader whose process vanishes without closing its connection — a node that loses power, a partitioned kubelet — leaves a database session that still holds the lock. That session is bounded: it carries an idle_session_timeout of a few check intervals, so the server ends it, and releases the lock with it, seconds after the pings stop rather than when TCP eventually gives up. plexsphere_background_leader sums to 0 for that window.

Recovery. A standby takes the lock within one check interval of it becoming free and starts ticking the loops. During a handover the old leader's sweep probes clear on their first standby tick, because a replica that runs no pass has no outcome to report, and the new leader's probes report its first pass. A sweep probe that stays red after a handover names a loop whose pass keeps failing on whichever replica leads, so read it together with plexsphere_background_leader to find the replica actually running the pass.

Background workers

The API runs its background loops as supervised workers, the background leader election among them. The supervisor restarts a worker that returns or panics while the process runs, after a jittered backoff that starts at 1 s and doubles up to 30 s. A worker that crashes again after five restarts within one unhealthy streak is declared permanently dead and is not restarted again. The worker reference lists every worker name.

Detection. health_probe_up{probe="workers"} drops to 0, and /readyz answers 503 unhealthy with this entry:

{"name":"workers","ok":false,"error":"worker: 1 worker(s) permanently dead: <name>"}

The replica logs the death at ERROR as worker permanently dead after exhausting restarts, with the dead worker's name in the worker attribute. Earlier lines for the same worker show how it got there: one WARN line worker exited, restarting after backoff per restart, carrying the error the worker returned, and one ERROR line worker panicked with a stack attribute per panic.

Degraded behaviour. The named loop has stopped on that replica, and the replica has left its Service. The HTTP surface and the other workers keep running, but the replica takes no traffic. A loop that runs on every replica keeps running on the other replicas. A leader-gated loop runs on no replica while the dead worker's replica leads, because leaving the Service does not release the leader lock; another replica takes the loop over only when leadership moves.

A leader-gated loop that ticks does not use up its budget over a bad pass: a pass that fails or panics is recorded and logged, and the next tick tries again, see Readiness. The two self-paced leader-gated loops stop for good, and the loss reaches beyond their replica. While authz-outbox-consumer is dead on the leader, no replica mirrors the authz outbox into SpiceDB, so the grants and revocations queued there do not take effect. While audit-archiver is dead on the leader, no replica drains the audit chains to the object store.

Recovery. Restart the Pod, for example with kubectl -n <namespace> delete pod <pod> or a kubectl -n <namespace> rollout restart of the API Deployment. A dead worker does not restart itself, and a readiness failure does not restart the container, so the replica stays out of the Service until its Pod is replaced. A restart clears the symptom only: read the error attribute of the death line, and the stack attribute of any worker panicked line, and fix the cause before the new Pod exhausts its budget too. Replacing the leader's Pod also moves leadership, so another replica takes a dead leader-gated loop over; when the cause lies in the data, that replica's worker dies the same way.

At shutdown the API cancels the workers after the HTTP drain and waits at most 10 s for them to return. A WARN line background workers did not stop within the join timeout means at least one worker had not returned when that wait ended, because it was still inside a pass or ignores cancellation. The exit code does not change. Such a worker can still hold a Postgres connection, and the pool close that follows waits for it, so the Pod can run until Kubernetes kills it at the end of its 60-second grace period.

Quick reference

Dependency	Probe	health_probe_up{probe=…}	Problem code	Degradation metric(s)	Tuning knob(s)
API boot	boot	boot	—	(probe gauge)	—
SpiceDB	spicedb-readiness	spicedb-readiness	authz_unavailable	authz_stale_check_served_total, authz_stale_check_cache_size	SPICEDB_CHECK_STALE_TTL, SPICEDB_CHECK_STALE_MAX_ENTRIES
Postgres	db-primary	db-primary	datastore_unavailable	state_stale_snapshot_served_total, state_stale_snapshot_cache_size	PLEXSPHERE_STATE_STALE_TTL, PLEXSPHERE_STATE_STALE_MAX_ENTRIES
NATS JetStream	jetstream	jetstream	event_stream_unavailable	(probe gauge)	—
OpenBao	secrets-reconcile	secrets-reconcile	openbao_unavailable	secretstore_unavailable_errors_total	—
Signing Service	(none)	—	signing_unavailable	sse_publisher_sign_failures_total, sse_relay_skip_total	—
Background workers	workers	workers	—	(probe gauge)	—

See also

• Disaster recovery — restoring the control plane after a full disaster (a different, larger-blast-radius event than a single dependency blip).
• Availability — the replica count of each workload and what an upgrade, a drain and a node failure cost, the planned-disruption counterpart to the dependency outages on this page.
• Secrets context — the node-secret read contract behind openbao_unavailable.
• SSE context — the signed event bus, the relay loop, and the signer alerting signals.
• Observability ingest context — the ingest back-pressure path that shares the structured-degradation contract.