Disaster recovery

This runbook covers backing up and restoring a plexsphere control plane after a disaster, and the change-freeze invariant that bounds the blast radius. The mesh data plane outlives a control-plane disaster: existing plexds keep running on their last-reconciled state, so a DR event is a change-freeze window, not a mesh outage. Peers stay connected, already-issued sessions keep working to their expiry, and traffic between nodes is unaffected — what pauses is the ability to make new changes through the control plane.

Every durable artefact is backed up, and every plane crossing (Postgres → OpenBao → Management Fleet) has a defined restore order so a restore never leaves a dangling pointer or writes stale state. The baseline below is a starting point; it may be hardened or relaxed per operator SLA.

Coverage

Store	Method	Cadence	Retention	Priority
PostgreSQL	Nightly logical dump (pg_dumpall) uploaded to an S3-compatible object store by the reference backup job. Continuous WAL archiving and point-in-time recovery (PITR) are an operator requirement on the operator's own Postgres; the release ships no archiver.	Dump every 24 h; WAL continuous once the operator archives it.	≥ 35 days (compliance) + 7 years cold for audit partitions.	P0
OpenBao	bao operator raft snapshot to an encrypted, access-controlled bucket independent of the primary blob store.	Every 4 h + on rotation events.	≥ 35 days. Older snapshots cascade to cold storage.	P0
Management-fleet clusters	etcd snapshots per cluster; Crossplane XR / Composition manifests also re-derivable from the plexsphere DB via the Provisioning Broker.	etcd every 15 min, retained 7 days.	30 days of etcd snapshots.	P1
Blueprint catalogue	External to plexsphere — lives in the source Git host (GitHub / GitLab / self-hosted). plexsphere stores the pinned revision per Resource.	As per the Git host's own backup.	Indefinite.	P2
Object store	Operator-run S3-compatible store replicated across failure domains (AWS S3 cross-region replication, or SeaweedFS cross-datacenter replication). plexsphere keeps no second copy of what it writes there. See Availability.	Continuous.	Per-Domain configured (default 90 days for action outputs, 1 year for diagnostics, 7 years for audit archive).	P0 — P2 for artefacts, P0 for the audit archive
Grafana Mimir	Mimir's object-store backend (AWS S3 / SeaweedFS via S3 API) is the durability layer; Mimir replicas are stateless and rebuildable from it.	Continuous (tenant object-store writes).	Per-Domain (default 13 months).	P3
Grafana Loki	Loki's object-store backend (AWS S3 / SeaweedFS via S3 API) — same as Mimir.	Continuous.	Per-Domain (logs default 90 days; node-side audit default 7 years).	P0 — P2 for logs, P0 for node-side audit
NATS JetStream	Operator-run JetStream cluster: three servers, file storage on persistent volumes, stream replicas raised to 3 (the API creates streams with 1 replica unless PLEXSPHERE_SSE_STREAM_REPLICAS and PLEXSPHERE_OBS_STREAM_REPLICAS say otherwise). No external archival: the SSE replay window is bounded and plexd's reconcile-pull covers any loss. See Availability.	In-cluster continuous.	≥ 24 h replay window.	P2
Configuration-as-code	External to plexsphere — Helm values, kustomize overlays, and deployment manifests live in the operator's Git host and are re-applied on restore.	As per the Git host's own backup.	Indefinite.	P2

The priority column grades restore order, from P0 (restore first, nothing else starts healthily without it) down to P3 (gaps self-heal after restore). The two compound cells — object store and Loki — carry a single dominant priority for ordering because their audit-archive obligation is the most critical content they hold.

Recovery objectives

Plane / Function	RPO	RTO
Data plane (mesh traffic between plexds)	n/a — not affected by plexsphere DR	n/a
Control-plane reads	≤ 5 min with operator WAL archiving; ≤ 24 h from the nightly dump alone	≤ 15 min
Control-plane writes	≤ 5 min with operator WAL archiving; ≤ 24 h from the nightly dump alone; secret material ≤ 4 h (OpenBao snapshot cadence)	≤ 30 min
Session-plane issuance	Already-issued sessions continue to their JWT expiry; new issuance depends on Signing Service restore.	≤ 30 min
Observability ingest	≤ 15 min (plexd local buffering backfills short gaps)	≤ 1 h
Audit archival	≤ 1 h	≤ 4 h (regulatory reads)

These targets are per region in a SaaS deployment; cross-region failover is a separate procedure with its own, weaker RTO.

Point-in-time recovery is an operator requirement

The release ships a nightly pg_dumpall and a four-hourly OpenBao raft snapshot, and nothing else. A recovery point under 24 hours needs continuous WAL archiving on the operator's own Postgres, which plexsphere neither configures nor runs.

These settings reach the 5-minute bound:

• archive_mode = on on the primary.
• An archive_command from the operator's archiver: pgBackRest, WAL-G, Barman, or the managed service's built-in PITR. The mechanism is described in the PostgreSQL continuous-archiving documentation.
• archive_timeout of at most 300 seconds. This is what bounds the recovery point at 5 minutes on a quiet database: a segment that never fills is otherwise never shipped.
• Base backups at least every 24 hours, so replay starts from a recent base rather than walking a week of WAL.
• Retention of at least 35 days, matching the retention the coverage table records for PostgreSQL.

Three bounds follow from that split:

• Postgres is at most 5 minutes behind with operator WAL archiving, because archive_timeout forces a segment switch every 300 seconds.
• Postgres is at most 24 hours behind from the nightly dump alone: that dump is the only Postgres artefact the release produces.
• Secret material is at most 4 hours behind, the cadence of the OpenBao snapshot.

The rewind trade. Step 2 of the restore checklist rewinds Postgres to the OpenBao snapshot's wall-clock time. Restoring OpenBao from a snapshot up to 4 hours old and rewinding Postgres to it discards up to 4 hours of Postgres writes; not rewinding keeps those writes but leaves rows that reference secret paths the snapshot does not hold. The rewind itself needs WAL archiving, because a nightly dump cannot be replayed to an arbitrary wall-clock time.

Restore checklist

Run these steps in order. Running them out of order leaves dangling pointers (Postgres rows that reference OpenBao paths that are not yet restored) or writes stale state.

1. OpenBao. Restore the most recent Raft snapshot into a fresh OpenBao cluster and unseal via the configured auto-unseal root. Verify: seals are healthy and known paths read back (platform-level wrap keys, a sample Cloud Credential).
2. PostgreSQL. Restore to the latest recovery point the deployment keeps: with operator WAL archiving, the latest base backup plus WAL up to the chosen target time; without it, the newest nightly dump. A nightly dump replays only into a freshly initialized, empty cluster, and psql must stop on the first error — see Replaying a nightly dump for the precondition and the command. The dump carries both logical databases, but pg_dumpall snapshots each one separately, so core and spicedb are mutually consistent only to within the dump's runtime; step 6 reconciles them. Verify: core and spicedb databases are present and their row counts match the source deployment's last known counts, rather than only the empty tables the migrations create; if OpenBao was restored to an earlier snapshot, Postgres is rewound to that snapshot's wall-clock time, which needs WAL archiving.
3. Signing Service. Attach the HSM / KMS slot holding the relevant signing keys; if the original HSM is lost, rotate via the transition-window mechanism. Verify: the signing slot is reachable and serves the active key.
4. NATS JetStream. Bring the NATS cluster back; replay data older than the retention window is lost but handled by reconcile-pull. Verify: the cluster is up; no client-side action is required.
5. Management-fleet clusters. Restore etcd from the most recent snapshot per cluster, or rebuild clusters fresh; Crossplane + ESO reconcile the namespaces for every assigned Project. Verify: ProviderConfigs, ExternalSecrets, and XRs are re-emitted by the Provisioning Broker.
6. SpiceDB. Start SpiceDB replicas against the restored spicedb datastore, then arm the authz outbox for replay over the window the dump spans — see Re-deriving the SpiceDB tuples. Arming is all this step does. The dual-write consumer that performs the replay is a leader-elected worker inside the core binary, so no row is drained and no tuple is written until step 7 starts it — do not wait here for an outbox that cannot move yet. Verify: schema read shows the schema and a sanity check against a known tuple passes; the UPDATE reports the number of rows it armed, and SELECT count(*) FROM plexsphere.outbox_events WHERE applied_at IS NULL AND occurred_at >= timestamptz '<dump start, UTC>' matches it; SpiceDB is green before the core binary passes its readiness probe.
7. Core plexsphere binary + Signing Service replicas. Start with the readiness probe gating on the above dependencies. Starting it starts the outbox consumer, which drains the rows step 6 armed; run the mirrored-relation reconcile sweep once that drain has finished, and before you lift the change freeze this runbook opens with. Verify: the drain finished — SELECT count(*) FROM plexsphere.outbox_events WHERE sync_failed_at IS NOT NULL AND occurred_at >= timestamptz '<dump start, UTC>' is zero and authz_sync_outbox_surrendered_total is flat, so no armed row was dropped again instead of mirrored; a permission check on an object written inside the dump's runtime resolves; plexds reconnect the SSE stream and run reconcile-pull; peer deltas, policy, and secret deliveries catch up automatically.
8. Observability backends. Bring up Grafana Mimir and Grafana Loki last. Verify: metrics and logs ingest resume; a plexsphere that serves writes without metrics is operational.

Replaying a nightly dump

pg_dumpall writes a plain SQL script of CREATE ROLE, CREATE DATABASE, \connect and COPY. It carries no DROP and no transaction wrapper, so it restores only into a freshly initialized, empty cluster. A standby brought up from the deployment manifests is not empty: it already holds the plexsphere role and the core and spicedb databases with the migrations applied. Replaying the dump into it skips every CREATE as "already exists" and collides on the primary key of every COPY, and psql still exits 0 unless it is told to stop.

The other half of that precondition is the databases: the target must hold none of the databases the dump creates. An image that pre-creates one is not empty either. The postgres image creates the database named in POSTGRES_DB, and with POSTGRES_DB unset but POSTGRES_USER set to something other than postgres its entrypoint creates a database named after the user. Either way the replay stops at ERROR: database "plexsphere" already exists, which is the refusal this runbook wants.

Fetch the artefact, then replay it into the empty cluster. The object is gzip: the dump job pipes pg_dumpall through gzip and stores the result as postgres/<job-name>.sql.gz, and get copies the object body byte for byte without decompressing it. Decompress on the way in rather than handing psql a -f it cannot parse. The block runs under bash rather than sh: set -o pipefail is a bash option, and /bin/sh on Debian and Ubuntu is dash, which answers Illegal option -o pipefail and aborts the block on its first line. It reaches that bash on stdin rather than by being pasted into an interactive one, because set -e exits the shell it runs in and an interactive bash is no exception: a paste that trips the first line would otherwise take the session — and the PGHOST, PGUSER and PGPASSWORD exported into it — with it.

bash <<'EOF'
set -euo pipefail
plexsphere-backup get --key postgres/<job-name>.sql.gz --file dump.sql.gz
gunzip -c dump.sql.gz \
  | awk -v r="$PGUSER" '$0 != "CREATE ROLE " r ";" && $0 != "CREATE ROLE \"" r "\";"' \
  | psql -v ON_ERROR_STOP=1 -h "$PGHOST" -U "$PGUSER" -d postgres
EOF

set -e is what stops a failed fetch from replaying a stale artefact. get deletes the destination when a download fails part-way, but it returns before creating it at all when the arguments are wrong or the object-store credentials are unset — pasting the block with the <job-name> placeholder still in it is enough, because the bash the block feeds reads the < and > as redirections and the binary never runs. Without -e the block falls through to the next line, decompresses whatever dump.sql.gz a previous drill left in the working directory, and replays it: valid SQL, exit 0, the wrong cluster contents. set -u fails the same way on an unexported PGHOST or PGUSER rather than connecting somewhere else — the heredoc is quoted, so the inner shell sees only what the outer one exported.

The awk covers the one line an empty cluster still collides on. pg_dumpall always writes a CREATE ROLE for the role it connected as, on a line of its own, and a fresh target already holds that role when the restoring superuser carries the same name, so the replay fails with ERROR: role "postgres" already exists and exits 3 under ON_ERROR_STOP=1. That is the postgres-to-postgres restore, which is the common case. What keeps the filter off every other role is the role name it compares against: every role's CREATE ROLE line has this same shape, and the attributes arrive on a separate ALTER ROLE line after it. That ALTER ROLE still applies, so the role keeps the attributes and the password hash the dump carries. The second comparison covers the quoted spelling, because pg_dumpall quotes any identifier outside [a-z0-9_]: a superuser named plexsphere-app appears in the dump as CREATE ROLE "plexsphere-app";, and a filter that knew only the bare form would leave the collision in place.

Both comparisons are whole-line string equality rather than a pattern, which is what keeps the role name out of the matching language. The same class of name that forces the quoted spelling is the class that carries regular-expression syntax: a superuser named ops|dr would make an alternation, svc.dr a wildcard, and one carrying (, { or [ an address the matcher refuses outright. awk compares them literally. The one character -v still transforms is a backslash, which it reads as opening an escape sequence in the assigned value; a role name carrying one needs the comparison built from ENVIRON["PGUSER"] instead.

Every other role in the dump is replayed verbatim and must not already exist in the target. Each one arrives as its own CREATE ROLE, and pg_dumpall emits every role before the first CREATE DATABASE, so a collision aborts under ON_ERROR_STOP=1 while the target still holds no restored database. So restore as the role the dump connected as, into a cluster that holds no role the dump creates. A target where a provisioning step already created the application role is not empty in this sense. psql runs without --single-transaction, so a replay that aborts in the roles section has already committed every role it emitted before the colliding one, and the target is no longer the clean one the block started against: re-initialise the cluster, or drop the colliding role together with every role the aborted run created before it. Dropping only the one the error names and re-running surfaces the next collision instead — one per attempt, each naming a role the operator never provisioned.

ON_ERROR_STOP=1 is what turns a wrong target into a non-zero exit instead of a restore that reports success over zero restored rows. Step 2's verify is row counts for the same reason: "the databases are present" passes on a cluster the migrations created and the dump never filled. The block runs under set -o pipefail for the same reason the dump job does: a truncated artefact fails in gunzip, and the shell would otherwise report only psql's status over the prefix that did decompress.

Re-deriving the SpiceDB tuples

pg_dumpall runs one pg_dump per database, each on its own connection and its own MVCC snapshot. PostgreSQL has no cross-database transaction and exported snapshots cannot be shared between databases, so core and spicedb are captured at different instants — as far apart as the dump takes to run. A write landing in that gap reaches one database's snapshot and not the other's, and the restored deployment then answers permission checks from tuples that disagree with the rows they describe.

Core is the source of truth for every tuple the dual-write consumer mirrors, so re-derive those tuples from it. The consumer polls WHERE applied_at IS NULL AND sync_failed_at IS NULL, and it writes tuples with SpiceDB's TOUCH semantics, so a replay is idempotent. Clear all three sync columns, not just applied_at:

UPDATE plexsphere.outbox_events
   SET applied_at = NULL,
       sync_failed_at = NULL,
       attempts = 0
 WHERE occurred_at >= timestamptz '<dump start, UTC>';

sync_failed_at is the surrender stamp: the consumer sets it once a row has failed attempts full poll cycles, and a row carrying it is filtered out of the poll for good. Re-arming such a row on applied_at alone puts it in a state the consumer never selects — no tuple, no retry, and no log line saying so. attempts needs the reset for the same reason: a successful apply stamps applied_at and leaves the counter where it was, so a row that once took four cycles is one failing cycle away from the threshold, and the first cycle against a freshly-started SpiceDB is exactly when a failure is likely.

Widen the window rather than narrow it: replaying a row that was already applied costs one TOUCH, while missing one leaves a member denied access to a Project they own, with nothing in any log to explain it.

The UPDATE arms the rows; it replays nothing. The consumer that drains them is a leader-elected worker inside the core binary, and authz_sync_outbox_surrendered_total is registered in that same process. So run the UPDATE in step 6, and expect the armed rows to sit untouched — and the metric to export no series at all — until step 7 starts the core binary. The drain and the surrender counter are step 7's verify, not step 6's; a count taken before the core binary starts is the row count you just armed, which is what step 6 checks instead.

Bound that count to the same window as the UPDATE. outbox_events is append-only with no prune path, so it still holds every row the consumer surrendered before the dump window — restored with sync_failed_at set, and deliberately left alone by a windowed re-arm. An unbounded WHERE sync_failed_at IS NOT NULL therefore counts those too and never reaches zero, which turns the gate into one an operator learns to step over rather than one that catches a row armed and surrendered again.

The replay only corrects one direction. It re-derives tuples core has rows for. It cannot remove a tuple core has no row for, and that is the skew with the worse failure mode.

pg_dumpall dumps the databases one after another, so one of the two snapshots is the newer, and which one is pg_dumpall's own ordering rather than anything the job chooses. Plan for the case where it is spicedb: a write that lands in the gap is then in SpiceDB and not in core — including its outbox row, because the outbox lives in core. Nothing replays it, and nothing deletes it.

Whether that matters depends on what was written:

• A new object and its tuples. A Project created in the gap leaves tuples whose object has no row. Those grant access to nothing: the restored control plane has no Project to check them against, and it re-creates neither the object nor the tuples.
• A relation between two objects that both predate the gap. A group membership added in the gap writes group:<gid>#member@user:<uid> while both the group and the user are already in the core snapshot. The restored core has no membership row, so no surface lists it and no revoke path reaches it — but SpiceDB still holds the tuple, and permission checks still pass on it. The grant survives the restore as an authorization the control plane can neither see nor withdraw.

The second case is the one to act on, and the exposure window is exactly the dump's runtime. Two ways to close it, in preference order:

• Make the window empty. Hold control-plane writes for the duration of the dump. The same change-freeze this runbook opens with, applied nightly for the minutes the dump takes, means no write can land between the two snapshots and the restored databases agree by construction.
• Reconcile the mirrored relations after the restore, in step 7 once the consumer has drained the rows step 6 armed. For each relation the dual-write consumer mirrors, compare SpiceDB's relationships against the restored core's rows and delete the ones with no backing row. Sweeping before the drain compares against a SpiceDB the replay has not finished correcting, so the answer is stale by the next tuple the consumer writes.

Do not widen that second option into "delete every tuple core has no row for". The rule is the narrow one: the sweep covers the relations the dual-write consumer mirrors, and nothing else. That set is exactly the set the sweep can decide, because the consumer's mapping is what tells it which core row backs a given tuple. Every tuple outside it either has no core row at all or none the sweep knows how to look up, so a blanket sweep reads all of them as skew and revokes grants that are not skew at all.

The writers that put tuples in SpiceDB outside the outbox — each out of scope for the sweep by construction, not by exception:

• The platform-operator grant seed. The core binary's boot seeds platform:plexsphere#admin and managementfleet:fleet#admin for the configured operator subject straight through the authorizer. Both objects are anchor strings with no core table behind them at all. Sweeping them revokes every surface those grants gate — Cloud create, catalog register/import, platform IdP bindings, platform Labels, the backup catalog, register-a-cluster, terminate-a-Project-assignment — in the middle of a disaster recovery, with no row left to re-derive them from. The operator locks themselves out of the control plane they are restoring.
• The bootstrap seeds. plexsphere-bootstrap writes its domain-admin grants and the structural parent edge behind them through that same direct path. They emit no outbox row — the adoption path deliberately does not re-emit the event the JIT sign-in path does — so the consumer's mapping does not reach them either.
• The assignment uses tuples. Cloud- and credential-assignment grant and approve write cloud:<id>#uses@project:<id> and cloudcredential:<id>#uses@project:<id> synchronously, outside the outbox, so a follow-up check resolves without waiting for a drain. An approved assignment row does back them, but not through the consumer's mapping, so the sweep cannot find it.
• The sink-enablement uses tuples. Audit-sink enablement writes sink:<id>#uses@project:<id> on that same synchronous path.
• POST /v1/authz/relation-tuples. The endpoint writes tuples straight to SpiceDB with no core row and no outbox row by design.

Reading any single one of these as "the exception" is how the blanket sweep gets run: an operator who has never enabled the relation-tuples endpoint concludes the warning does not apply to them. It applies to all five.

Restore drill

The checklist above states a recovery point and an RTO. The drill is what measures both against the artefacts the reference jobs actually produce, so the figures this runbook carries come from a rehearsal rather than from an assumption.

1. Scope and cadence. The drill restores OpenBao and Postgres from the newest artefacts in the bucket into an isolated environment, never into production. Isolated means a separate cluster, or at the very least a namespace with its own RBAC, a default-deny NetworkPolicy, and no service account shared with anything else. What lands there is production data: the restored OpenBao serves the whole production secret set over its API and trusts the source cluster's tokens, and the Postgres replay carries pg_dumpall's globals, which include every role's password hash. The drill environment is production-trust for its entire lifetime and is subject to the same access controls as production until step 8 destroys it. Unsealing it is a break-glass operation, so it follows the Shamir path in Seal-key custody — that is the point of rehearsing it. Run the drill before go-live, then at least quarterly and after any change to the backup jobs or to the Postgres or OpenBao major version.

2. Write a marker, then trigger both jobs by hand. Write a marker row into core and a marker secret into OpenBao, then start one Job from each CronJob:

   kubectl create job --from=cronjob/dr-postgres-backup drill-postgres
   kubectl create job --from=cronjob/dr-openbao-snapshot drill-openbao

   Record the wall-clock time as the backup start. The object keys carry the name of the Job, not the name of the CronJob, so these two runs land postgres/drill-postgres.sql.gz and openbao/drill-openbao.snap instead of the time-sorted keys a scheduled run produces.

3. Write a second marker once both Jobs have completed. This one must not survive the restore. Its absence afterwards is what proves the restore landed the artefact rather than leaving the source data in place.

4. Record the wall-clock time as the simulated loss and start the restore clock.

5. Restore OpenBao first. Fetch the snapshot, then force it into the drill cluster:

   plexsphere-backup get --key openbao/drill-openbao.snap \
     --file openbao.snap
   plexsphere-backup openbao-restore --force --file openbao.snap

   Unseal with the source cluster's key, reconstructed from three of the five Shamir shares. A forced restore adopts the source keyring, so the drill cluster's own key leaves it sealed. Do not reach for an auto-unseal root instead: that grants the drill workload decrypt on the production KMS seal key, which turns anyone who can schedule a Pod in that namespace into a single operator who can unseal any production snapshot in the bucket — exactly what the 3-of-5 split exists to prevent. If a deployment accepts that trade anyway, it needs the same sign-off as any other grant on the seal key.

6. Restore Postgres into an empty cluster with the command block from Replaying a nightly dump, reading the key postgres/drill-postgres.sql.gz. Follow the link rather than copying the block here, so the two cannot drift.

7. Pass criteria. The first marker row and the first marker secret read back. The second marker is absent from both stores. Row counts match the source for the core and spicedb databases step 2 of the checklist names. The recovery point (loss time minus backup start) is at most 24 hours for Postgres and at most 4 hours for OpenBao on a scheduled artefact. The elapsed restore time fits the RTO in Recovery objectives. Record those two measured figures with the date of the drill.

8. Destroy the environment. Immediately after the figures in step 7 are recorded, delete the drill namespace (or tear down the drill cluster) and the openbao.snap and dump.sql.gz files fetched onto the operator's machine. A drill that is merely left idle keeps a live, unmonitored copy of every production secret and every role's password hash until the next one. What the drill retains is the two measured figures and its date — not the environment, and not the artefacts.

The regional-failover suite under tests/e2e/dr/regional-failover/ runs this procedure nightly against the shipped manifests.

Seal-key custody

OpenBao's seal key is the single point that, if lost, renders every plexsphere secret unreadable. Treat its custody as the most sensitive part of the DR plan.

• Steady-state auto-unseal. In normal operation OpenBao auto-unseals via a cloud KMS (AWS KMS / GCP Cloud KMS / Azure Key Vault), so no human handles the seal key during a routine restart.
• Shamir break-glass. The recovery key is split with Shamir's scheme: five shares held by distinct operators, three required to reconstruct, each on physically segregated media. No single operator may unseal alone.
• Drills. Run a restore drill every quarter into an isolated environment using the Shamir path, so the break-glass procedure is exercised before it is needed in anger. Restore drill is the step-by-step procedure, and it follows this bullet.

Snapshot-force caveat. A bao operator raft snapshot restore -force writes the source cluster's encryption keyring into the target cluster. The restored cluster must therefore be unsealed with the source snapshot's seal/unseal key, not the target cluster's own key. Unsealing with the wrong key leaves the restored cluster permanently sealed — custody of the source key must travel with the snapshot.

Regional failover

Failover is a promotion, not a live migration. Domains and Projects are region-pinned at creation, so each region runs its own core replicas, Signing Service, pub/sub, Postgres primary, OpenBao cluster, and management-fleet cluster. Bringing a region online means promoting its standby, not streaming live traffic across region boundaries.

• Warm standby. Postgres and OpenBao replicate to the standby region asynchronously, giving a cross-region RPO of roughly ≤ 5 min for Postgres and ≤ 15 min for OpenBao snapshots. The Postgres figure holds only when the operator replicates or archives WAL to the standby region; without that, the standby is as fresh as the last nightly dump shipped to it.
• Promotion. The standby region brings up an empty management cluster and re-applies plexsphere state for the promoted Projects; Crossplane and ESO then reconcile their namespaces.
• Residency opt-out. Residency-restricted Domains can opt out of cross-region standby and run in-region DR only, keeping their data within a single region's boundary.

For the deployment model behind region pinning and per-region ingress, see the multi-region runbook. The per-region overlays live under deploy/local/overlays/region-reference/.

What plexsphere does not back up

Some state is deliberately not backed up because it is reconstructible or short-lived. Each item below names its recovery path.

1. Plexd agent state on nodes — plexd is deterministic given plexsphere's source of truth; after a control-plane restore it reconciles back to consistency via SSE replay and reconcile-pull.
2. Customer-side secret stores — customer Vault / AWS Secrets Manager used for Adopted-K8s bootstrap-token delivery belong to the customer; subject to the customer's own DR plan.
3. WireGuard keys on plexd nodes — mesh keys are rotated on reconnect when needed rather than restored; re-established via rotate_keys events on reconnect.
4. Bootstrap-token plaintext — bootstrap tokens are short-TTL and one-time-use; only their hashes are persisted server-side; mint a fresh bootstrap token for any pending enrolment.
5. Transient SSE state — the signed SSE replay window is bounded and intentionally not archived; plexd reconcile-pull reconstructs current state after the window is lost.

Automation reference

The reference overlay under deploy/local/overlays/dr-reference/ carries the two scheduled jobs that produce the artefacts this runbook restores from: a nightly Postgres dump and a four-hourly OpenBao raft snapshot. It is not applied by make dev, because the dev OpenBao runs server -dev with an in-memory storage backend and the snapshot endpoint rejects the call with raft storage is not in use. Against a Raft-backed cluster the render is a production shape: an operator supplies the four Secrets described below, pins the release by digest, and sizes the staging volumes for this deployment's own dump.

Both jobs run ghcr.io/plexsphere/plexsphere-backup, one of the images a release publishes. The overlay's images: block ships the in-tree placeholder tag v0.0.0-dev, which resolves against nothing. Replace it with a digest, not a tag, before applying the overlay.

A tag is mutable. Anyone who can move vX.Y.Z in the ghcr.io namespace — a leaked publish token, a compromised release workflow, a registry-side compromise — decides what these CronJobs run, and they run on a schedule with a BYPASSRLS Postgres credential and an OpenBao snapshot credential in the pod. A digest names one manifest and cannot be moved. Resolve it, verify the signature the release lane produced (the identity and issuer are in Verify what you downloaded), and pin the result:

crane below is from go-containerregistry; docker buildx imagetools inspect prints the same digest.

digest="$(crane digest ghcr.io/plexsphere/plexsphere-backup:vX.Y.Z)"

cosign verify "ghcr.io/plexsphere/plexsphere-backup@$digest" \
  --certificate-identity https://github.com/plexsphere/plexsphere/.github/workflows/release.yaml@refs/tags/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

kustomize edit set image \
  "plexsphere-backup=ghcr.io/plexsphere/plexsphere-backup@$digest"

Re-run the three commands to move to a new release; nothing else in the overlay changes.

The repository ships further automation around the two jobs:

• CLI. The plexsphere-backup CLI exposes the catalog, sequence, put, get, openbao-snapshot, openbao-restore, nsk-rewrap, and verify subcommands for inspecting the coverage matrix and driving snapshot/restore round-trips.
• Read-only inspection surfaces. The same coverage matrix and ordered restore plan are also exposed read-only through the platform-scoped GET /v1/platform/backup/catalog and GET /v1/platform/backup/restore-plan operations, and over the plexctl backup catalog and plexctl restore plan command surfaces — so an operator can read the catalogue and the restore sequence without the plexsphere-backup binary on the box. These surfaces are inspection-only: they project the plan, they do not stream artefacts or drive a restore.
• Tests. The DR integration suite (tests/integration/dr_*) and the regional-failover e2e suite (tests/e2e/dr/) prove the sequence and integrity against real Postgres and OpenBao round-trips; the e2e suite restores from the reference jobs' own artefacts, with the commands the Restore drill documents.

What each job writes

dr-postgres-backup runs nightly at 02:00. Its pg-dump init container streams pg_dumpall into /backup/dump.sql.gz on an emptyDir; the upload container then puts that file in the object store under the key postgres/<job-name>.sql.gz.

dr-openbao-snapshot runs every four hours. Its raft-snapshot init container writes /snapshot/openbao.snap; the upload container puts that file under the key openbao/<job-name>.snap.

Both keys carry the name of the Job rather than the name of the CronJob. A CronJob names each Job <cronjob>-<scheduled minute since epoch>, so the keys sort by time and nothing has to compute a timestamp.

The dump and the snapshot run as init containers. A non-zero exit stops the pod before upload starts, so a failed dump is never uploaded: every key in the bucket belongs to a run that finished.

Size the staging volumes before you apply

Both artefacts stage on an emptyDir before upload streams them out — plexsphere-backup put takes --file, not stdin. The overlay ships sizeLimit: 1Gi on each, which is a value for a small deployment and not a default to leave alone.

sizeLimit is a hard ceiling. Since Kubernetes 1.25 the kubelet's ephemeral-storage eviction manager evicts a pod whose emptyDir grows past it. The eviction lands mid-write, so the CronJob logs show a truncated dump message and nothing about capacity; backoffLimit retries into the same wall; the Job ends Failed and the night produces no artefact. Nothing about the CronJob looks wrong afterwards, and the same thing happens every night from the first night the threshold is crossed — the newest recoverable point simply stops advancing.

Measure the compressed pg_dumpall output for this deployment, size sizeLimit above it with headroom, and re-check as the database grows — audit partitions retained for years are what push it. Swap the emptyDir for a PersistentVolumeClaim once the dump outgrows node ephemeral storage.

Above roughly 5 GiB, raising sizeLimit is not enough on its own: plexsphere-backup put issues a single-part PutObject, so the upload hits S3's single-PUT limit. Split the dump per database, or add a multipart path to put, before going there.

The four Secrets

There is one Secret per container, not per job, and the operator creates all four. The overlay renders none of them, so no credential material lives in the repository.

The split is the blast radius. The dump role carries BYPASSRLS and the snapshot role reads sys/storage/raft/snapshot — between them, every tenant's rows and the entire secret store. The upload containers need neither: they speak S3 to an operator-supplied endpoint and parse its responses. Handing one Secret to both containers of a job would put each credential in the environment of a container that has no use for it.

dr-postgres-backup-db reaches only the pg-dump init container:

Key	Required	Meaning
PGHOST	yes	Host pg_dumpall connects to.
PGUSER	yes	The dump role, which carries BYPASSRLS (see below).
PGPASSWORD	yes	Password of that role.
PGPORT	no	libpq's own default (5432) applies when unset.

dr-postgres-backup-store reaches only the upload container:

Key	Required	Meaning
PLEXSPHERE_BACKUP_BUCKET	yes	Bucket the dump is uploaded to.
PLEXSPHERE_BACKUP_S3_REGION	yes	Object-store region.
PLEXSPHERE_BACKUP_S3_ACCESS_KEY	yes	Object-store access key.
PLEXSPHERE_BACKUP_S3_SECRET_KEY	yes	Object-store secret key.
PLEXSPHERE_BACKUP_S3_ENDPOINT	no	S3-compatible endpoint; empty means AWS S3.
PLEXSPHERE_BACKUP_S3_USE_PATH_STYLE	no	true for a store that addresses buckets by path.
PLEXSPHERE_BACKUP_S3_ALLOW_INSECURE_ENDPOINT	no	true to allow a plaintext endpoint.

dr-openbao-snapshot-auth reaches only the raft-snapshot init container:

Key	Required	Meaning
PLEXSPHERE_BACKUP_OPENBAO_ADDRESS	yes	OpenBao API address.
PLEXSPHERE_BACKUP_OPENBAO_K8S_ROLE	auth choice	OpenBao Kubernetes role the pod logs in as.
PLEXSPHERE_BACKUP_OPENBAO_APPROLE_ROLE_ID	auth choice	AppRole role id.
PLEXSPHERE_BACKUP_OPENBAO_APPROLE_SECRET_ID	auth choice	AppRole secret id; belongs with the role id.
PLEXSPHERE_BACKUP_OPENBAO_NAMESPACE	no	OpenBao namespace the call is scoped to.

The three auth choice rows are one decision: either the Kubernetes role, or the AppRole pair. The next subsection covers both.

dr-openbao-snapshot-store reaches only the upload container. Point it at a bucket and credentials independent of the primary blob store, so losing the primary store's credentials does not also lose the snapshots that recover it:

Key	Required	Meaning
PLEXSPHERE_BACKUP_BUCKET	yes	Bucket the snapshot is uploaded to.
PLEXSPHERE_BACKUP_S3_REGION	yes	Object-store region.
PLEXSPHERE_BACKUP_S3_ACCESS_KEY	yes	Object-store access key.
PLEXSPHERE_BACKUP_S3_SECRET_KEY	yes	Object-store secret key.
PLEXSPHERE_BACKUP_S3_ENDPOINT	no	S3-compatible endpoint; empty means AWS S3.
PLEXSPHERE_BACKUP_S3_USE_PATH_STYLE	no	true for a store that addresses buckets by path.
PLEXSPHERE_BACKUP_S3_ALLOW_INSECURE_ENDPOINT	no	true to allow a plaintext endpoint.

Authenticating the snapshot job

PLEXSPHERE_BACKUP_OPENBAO_K8S_ROLE selects Kubernetes auth: the CLI reads the pod's ServiceAccount token from /var/run/secrets/kubernetes.io/serviceaccount/token and logs in with it. The pair PLEXSPHERE_BACKUP_OPENBAO_APPROLE_ROLE_ID and PLEXSPHERE_BACKUP_OPENBAO_APPROLE_SECRET_ID selects AppRole. The CLI prefers AppRole whenever a role id is set and reaches Kubernetes auth only when it is empty, so a Secret carrying both logs in with AppRole.

Both jobs set automountServiceAccountToken: false. The dr-postgres-backup pod needs no Kubernetes identity at all. The dr-openbao-snapshot pod does, but only in raft-snapshot, so it projects its own ServiceAccount token into that container alone, at the default path. Automounting would put the same token in upload — a container that speaks only S3 would then hold a login for the role that reads the whole secret store. Move the mount if you prefer another path and set PLEXSPHERE_BACKUP_OPENBAO_K8S_JWT_PATH to match; the CLI reads the token from wherever that variable points.

Bind the role to a policy scoped to the snapshot endpoint rather than to a root policy, so a leaked snapshot credential cannot read or mutate arbitrary secrets:

path "sys/storage/raft/snapshot" { capabilities = ["read", "sudo"] }

The policy grants a read of the raft snapshot endpoint, and sudo alongside it because that path is root-protected. It grants nothing else: writing a snapshot back needs create or update on the same path or on sys/storage/raft/snapshot-force, so the credential the CronJob holds cannot overwrite the cluster it backs up.

A private certificate authority. secretstore.Config carries an address, an auth strategy, a namespace, and a TLS-verification switch, and has no field for a CA bundle. Trust a private CA at the process level instead: mount the bundle into the pod and set SSL_CERT_FILE to its path. Go's crypto/x509 reads that variable on Linux when it builds the system root pool, so the login and the snapshot call validate the cluster's certificate against that root.

Snapshotting on a rotation event

The coverage table records the OpenBao cadence as every four hours plus on rotation events. The CronJob covers the cadence; the rotation half is an operator action. After rotating seal or wrap-key material, take a snapshot at once instead of waiting up to four hours for the next scheduled run:

kubectl create job --from=cronjob/dr-openbao-snapshot \
  openbao-snapshot-after-rotation

The Job inherits the CronJob's template, so it reads the same two Secrets and uploads to the same openbao/ prefix, under the Job name given on the command line.

Checking one artefact by hand

plexsphere-backup verify reads one named object's metadata and compares its age with a window. The bucket and the object-store credentials come from the same PLEXSPHERE_BACKUP_* variables the matching -store Secret carries:

plexsphere-backup verify --key postgres/<job-name>.sql.gz --max-age 26h
plexsphere-backup verify --key openbao/<job-name>.snap --max-age 5h

Exit code 0 means the artefact exists and is younger than --max-age; 1 means it is missing or older than the window. Each window is the cadence plus room for one late run (26 h for the nightly dump, 5 h for the four-hourly snapshot).

<job-name> is the name of the Job, not the CronJob — a CronJob names each Job <cronjob>-<scheduled minute since epoch>. Read the name you want off the cluster:

kubectl -n <namespace> get jobs \
  -l app.kubernetes.io/name=dr-postgres-backup \
  --sort-by=.status.startTime

This makes verify a post-run and drill check: it confirms the artefact a run you can name actually landed in the bucket. It is not a scheduled freshness probe. There is no prefix or newest-object mode, and a probe on a timer cannot reconstruct the key — it would have to recompute the CronJob controller's own scheduled-minute arithmetic for the run it expects.

Alert on the jobs, not on the bucket

Nothing in this release pages when the backups stop, and they stop for ordinary reasons: a rotated PGUSER password, an expired object-store access key, a staging volume the dump outgrew, a pg_dumpall version skew after a Postgres upgrade. The Job goes Failed, failedJobsHistoryLimit: 3 rotates the evidence away within a day, and the next signal is the restore.

Wire an alert to the CronJob's own status, which the operator's monitoring stack already scrapes through kube-state-metrics. The last-success timestamp is the freshness signal, with the same windows verify uses:

time() - kube_cronjob_status_last_successful_time{cronjob="dr-postgres-backup"} > 26 * 3600
  or absent_over_time(kube_cronjob_status_last_successful_time{cronjob="dr-postgres-backup"}[1h])
time() - kube_cronjob_status_last_successful_time{cronjob="dr-openbao-snapshot"} > 5 * 3600
  or absent_over_time(kube_cronjob_status_last_successful_time{cronjob="dr-openbao-snapshot"}[1h])

The absent-series arm is not belt-and-braces. kube-state-metrics emits kube_cronjob_status_last_successful_time only once the CronJob's .status.lastSuccessfulTime is non-nil, so a CronJob that has never produced a successful run exports no series at all and the subtraction evaluates to an empty vector — which never fires. That is the state after a first apply with the wrong PGUSER, the wrong bucket, or an object-store key that was never granted PutObject: exactly the day-one misconfigurations this section exists to catch. Without that arm the operator wires the alert, watches it stay green forever, and finds the empty bucket at the restore. The same hole opens if the CronJob is deleted or renamed.

Write that arm as absent_over_time(...[1h]), not as the instant absent(). absent() is evaluated per instant, so a kube-state-metrics rollout, a node drain, or one missed scrape empties the selector for a single evaluation and the rule pages "backups are stale" while the backups are fine. Over a 1h range it fires only when the series is missing for the whole hour — which still covers every case above, since a CronJob that has never succeeded, or was deleted or renamed, exports nothing for as long as that lasts — without firing on routine control-plane churn. A rule that cries wolf on a kube-state-metrics restart trains the operator to ignore the one backup signal this section exists to make trustworthy.

Alert on failures as well as on staleness — kube_job_failed{job_name=~"dr-(postgres-backup|openbao-snapshot)-.*"} > 0 fires on the first bad run rather than a cadence later, and while the Job is still in history to read logs from.

The Postgres dump needs its own role

Dump Postgres as a SUPERUSER or BYPASSRLS role — not as the application role.

Three tables (secret_metadata, node_secret_key, domain_nsk_wrap_key) carry FORCE ROW LEVEL SECURITY for tenant isolation, and pg_dump / pg_dumpall set row_security = off and then fail with query would be affected by row-level security policy on any table whose policies apply to the connected role. Because the application role is documented to run NOSUPERUSER NOBYPASSRLS (see the row-level-security reference), dumping as it aborts the backup — every night, silently, until someone reads the CronJob logs.

A backup is cross-tenant by definition, so granting the dump role BYPASSRLS is the right answer rather than a concession:

CREATE ROLE plexsphere_backup LOGIN PASSWORD '…' BYPASSRLS;
GRANT pg_read_all_data TO plexsphere_backup;

The job connects as the PGUSER of the dr-postgres-backup-db Secret, which names that BYPASSRLS role.

DECISION: the repo's automation proves the restore sequence and integrity via logical pg_dumpall + raft-snapshot round-trips, and deliberately does not encode production-grade point-in-time recovery (continuous WAL archiving). The reference jobs upload the dump and the snapshot to an S3-compatible object store with plexsphere-backup put, so each round-trip ends in a durable artefact rather than pod-local scratch space. Rejected alternative: ship a pinned WAL-archiver deployment (a configured archive command + base-backup + recovery-target tooling) in this reference overlay. It was rejected because continuous WAL archiving is deployment-specific — the archive command, retention policy, and recovery-target tooling are operator choices — so a pinned archiver would either hard-code one operator's substrate (misleading every other deployment) or carry so many placeholders it stops being runnable. The portable, verifiable contract the automation can prove is the restore ordering and integrity; PITR wiring stays documented as an operator responsibility above.