plexsphere

Appearance

Rotate the NSK wrap key ​

Every Node Secret Key (NSK) is stored wrapped at rest. The software wrap-key provider seals each NSK with AES-256-GCM under a single static key the binary reads from PLEXSPHERE_NSK_WRAP_KEY_B64. The server holds exactly one wrap key at a time, so it can only unwrap rows that were sealed under that same key.

Rotating the wrap key therefore has two halves that must happen in order: re-wrap every stored row from the old key to the new key, then flip the server's PLEXSPHERE_NSK_WRAP_KEY_B64 to the new key. The re-wrap is driven by the nsk-rewrap subcommand of the plexsphere-backup operator CLI implemented in ../../../cmd/plexsphere-backup/nsk_rewrap.go.

A KMS- or OpenBao-backed wrap-key provider (external custody, no env key at all) is a planned follow-up. Until it ships, the wrap key is an operator-held env value and this runbook is the sanctioned way to rotate it.

Prerequisites ​

  • Operator access to the plexsphere core Postgres and a plexsphere-backup binary that can reach it.

  • The current wrap key (the value of PLEXSPHERE_NSK_WRAP_KEY_B64 on the running deployment).

  • A new 32-byte wrap key. Generate one with:

    shell
    head -c 32 /dev/urandom | base64

  • A maintenance window on the NSK-serving path. Between the re-wrap and the env flip, rows are sealed under the new key while the server still holds the old one, so heartbeat and secret-fetch unwraps fail until the flip completes. Drain or scale down the API before step 3.

Steps ​

1. Dry-run against the live database ​

The --dry-run pass unwraps every stored row under the old key and re-seals it in memory, but writes nothing and commits nothing. Run it first to prove the old key is correct and see how many rows a real run would touch:

shell
export PLEXSPHERE_DSN='postgres://…'
export PLEXSPHERE_NSK_WRAP_KEY_B64_OLD='<current key>'
export PLEXSPHERE_NSK_WRAP_KEY_B64_NEW='<new key>'

plexsphere-backup nsk-rewrap --dry-run
# level=INFO msg="nsk-rewrap complete" dry_run=true scanned=42 rewrapped=42 already_new=0

A scanned count that does not match your Node fleet, or an error naming a node whose row unwraps under neither key, means the old key is wrong — stop and re-check before touching production data. The subcommand exits 0 on success, 2 for a configuration error (missing DSN, a key that is not 32 bytes, or identical old and new keys), and 1 for an operation error (database failure, or a row that unwraps under neither key).

2. Drain the NSK-serving path ​

Scale the API deployment to zero, or otherwise stop it serving heartbeat and secret-fetch requests, so no unwrap runs against half-rotated data.

3. Re-wrap every stored key ​

Run the same command without --dry-run. The whole sweep runs inside a single transaction, so a mid-run failure rolls back and every row stays on the old key. It also holds a SHARE ROW EXCLUSIVE lock on plexsphere.node_secret_key for its duration: reads are unaffected, but a Node registering while the sweep runs waits for it. That is deliberate — a registration that slipped between the sweep's snapshot and your key flip would store its key under the old wrap key, and nothing would hold that key afterwards:

shell
plexsphere-backup nsk-rewrap
# level=INFO msg="nsk-rewrap complete" dry_run=false scanned=42 rewrapped=42 already_new=0

The command is safe to re-run: a row already sealed under the new key is counted in already_new and left untouched, so a retried run after a transient failure converges rather than double-wrapping. Rows for retired NSKs are re-wrapped too, so historical secret deliveries stay forensically recoverable under the new key.

4. Flip the server wrap key ​

Set PLEXSPHERE_NSK_WRAP_KEY_B64 to the new key on the API deployment (and the signer, if it shares the wrap key) and restart.

5. Bring the NSK-serving path back ​

Scale the API back up. Fresh heartbeats and secret fetches now unwrap against the new key.

Flags ​

Each flag falls back to the named environment variable.

FlagEnvironment variableMeaning
--dsnPLEXSPHERE_DSNPostgreSQL connection string.
--old-key-b64PLEXSPHERE_NSK_WRAP_KEY_B64_OLDCurrent base64 32-byte wrap key.
--new-key-b64PLEXSPHERE_NSK_WRAP_KEY_B64_NEWReplacement base64 32-byte wrap key.
--dry-run—Verify only; write and commit nothing.

Verification ​

  • The nsk-rewrap summary line reports rewrapped equal to the number of rows that were still on the old key and already_new equal to the rows a re-run would skip; a second immediate run should report rewrapped=0.
  • After the env flip and restart, a Node heartbeat succeeds — the server unwraps the Node's freshly re-wrapped NSK under the new key. A row that still failed to re-wrap would surface as a 401 on the Node's next heartbeat.

See also ​