Availability

This runbook covers how many replicas each workload runs and what changes that, what a rolling upgrade, a node drain and the loss of a node cost each workload, and what the stores you provide have to do. The charts run the API, the console, the bundled SpiceDB and the signer. Postgres, NATS JetStream, OpenBao and the object store are yours; this page states what each must provide and links the vendor's own high-availability documentation.

What the charts run

Workload	Default replicas	Value that changes it	Above one replica
API	1	`api.replicas`	Budget and spread
Console	1	`console.replicas`	Budget and spread
SpiceDB (bundled)	1	`spicedb.replicas`	Budget and spread
Signer	1	none	Nothing
Migrate Job	one Pod per install and upgrade	none	Nothing

Budget and spread means two documents the chart renders only above one replica: a PodDisruptionBudget with minAvailable: 1 and unhealthyPodEvictionPolicy: AlwaysAllow, and a topology spread constraint with maxSkew: 1 over kubernetes.io/hostname and whenUnsatisfiable: ScheduleAnyway. At one replica the chart renders neither. AlwaysAllow lets a drain evict a Pod that is not ready without waiting for the budget, so replicas that all fail readiness do not block the drain that fixes the outage.

Raising spicedb.replicas buys availability, not throughput. The bundled SpiceDB is a ClusterIP Service with one virtual IP, and each API process holds a single long-lived gRPC connection that kube-proxy pins to one SpiceDB Pod for the connection's lifetime. A second SpiceDB Pod survives a drain or a node failure, and it answers no checks for an API Pod already pinned elsewhere, so check latency does not fall with the count. Size the count against disruptions, not against load.

SpiceDB loads --grpc-tls-cert-path through a certificate watcher, so it serves a renewed leaf without a restart. A changed CA is the case that costs one: the API reads its trust root once per process, so roll it with kubectl rollout restart deployment/plexsphere once the new root is in the Secret. Nothing does it for you while the Secret keeps its name — the Pod template names that Secret and no more, so it does not change and helm upgrade replaces no Pod. A rotation and a swap of the issuer behind the same name both count, the latter including the move off a hand-issued certificate. The CA the spicedb chart issues for itself lives ten years, so a rotation of that one is a rare event.

What an upgrade, a drain and a node failure cost

At one replica

An upgrade surges a new Pod before it stops the old one: the Deployment's default rolling update (25 % / 25 %) rounds to maxUnavailable: 0 and maxSurge: 1. A drain, or the loss of the node, is an outage of that workload until the Pod is rescheduled. After a node failure that wait is the Kubernetes default 300-second not-ready toleration plus the Pod's own start time.

At two or more replicas

The budget keeps one Pod through a drain. The spread is best effort (whenUnsatisfiable: ScheduleAnyway), so the scheduler co-locates Pods when that is all it can do: check where they landed with kubectl get pods -o wide. A cluster with fewer nodes than replicas co-locates Pods every time, and a single node failure then takes every replica of that workload.

An upgrade of the API also runs the pre-upgrade migrate hook before the first API Pod is replaced. Run more than one API replica describes that window and which migrations make it visible.

The one-time move onto TLS

The costs above are those of an ordinary upgrade. Moving a release installed before the bundled SpiceDB served TLS onto it is not one: both halves of that transport flip in the same revision, no staged cutover is supported, and the rollout window answers 503 on every gated operation the check cache cannot serve. Upgrading an existing release onto TLS has the window and how to keep it short.

What the loss of a workload costs

SpiceDB down: a gated operation that cannot be served from the last-known-good check cache fails with HTTP 503 and the problem codeauthz_unavailable. Mutations never consult that cache. Which checks the cache does serve, for how long, and the counters that show it, are under SpiceDB (authorization).

Signer down: new session issuance and new signed-event publishes fail, and already-issued sessions keep working. See Signing Service.

Console down: the web UI is unavailable. The API and plexctl keep working, so every operation stays reachable over HTTP and the CLI.

The signer runs one replica

The signer's Deployment pins spec.replicas: 1 and the chart exposes no value to raise it. What a second Pod does to a key rotation has not been analysed: every signer Pod reconciles the same scope's keys on its own readiness probe, and which of the reconcile and rotation paths stay correct with two of them running at once is an open question. The count stays at one until it is answered, whichever provider keyProvider names.

The chart renders no PodDisruptionBudget for the signer, so a node drain is never blocked by it: the Pod is evicted, and issuance fails until it runs again. The API registers no readiness probe for the signer, so alert on the two publish counters named under Signing Service together with the signing_unavailable 503 rate.

Postgres

One writer endpoint. The chart reads a single DSN from the Secret named in database.existingSecret, and every API Pod, the migrate Job and the signer connect through it. That endpoint has to follow a failover: after a promotion it must resolve to the new primary.
Synchronous replication to at least one standby, if the control-plane recovery point in Disaster recovery is to hold across a failover. Asynchronous replication loses whatever the old primary had not shipped when it failed.
A connection budget that fits. Each API Pod holds up to 46 connections with the default pool sizes, and a rolling upgrade runs up to twice api.replicas Pods at once, so budget 46 connections for each of those Pods, and add the signer's pool. Every plexsphere pool ceiling is in the connection pools reference, which covers those two and not SpiceDB. When the bundled SpiceDB shares the database, budget it separately: SpiceDB opens a read pool and a write pool per process and fills each to its minimum at start, 20 plus 10 connections with the release's defaults. That term multiplies by spicedb.replicas, and a SpiceDB rollout surges a Pod the same way an API rollout does, so budget 30 connections for twice spicedb.replicas Pods. Lower those pools through spicedb.extraArgs when the budget is tight:
yaml
```
spicedb:
  extraArgs:
    - --datastore-conn-pool-read-min-open=5
    - --datastore-conn-pool-read-max-open=5
    - --datastore-conn-pool-write-min-open=2
    - --datastore-conn-pool-write-max-open=2
```
Keep the total under max_connections less superuser_reserved_connections.
A session-pinned connection path. The background leader holds a PostgreSQL session advisory lock on one connection it hijacks out of the pool, so a transaction-mode pooler between the API and Postgres breaks leader election: every leader-gated loop stops, and nothing at install time says so. Run the pooler in session mode, or let the API reach Postgres directly. See Session connection requirement.

What a failover looks like from the API, which reads keep being served from cache and which paths answer a structured 503, is under Postgres (datastore / node state). PostgreSQL's own replication and failover options are in the PostgreSQL high availability documentation.

NATS JetStream

Run three JetStream-enabled servers with file storage on persistent volumes.

The API pins a 24-hour replay window on the SSE node-event stream and on every observability ingest stream, and no backup copies a stream anywhere. That bounds what a JetStream loss costs. With the stream replicas at 3 across three servers, losing one server costs nothing: the remaining two hold quorum and the streams stay writable. Losing the storage under all three costs the replay window and no more. plexd rebuilds its current state through reconcile-pull, and buffered observability data that no consumer had read yet is lost.

Memory storage, or a single stream replica, turns a routine server restart into that same loss.

The API creates its streams with one replica unless told otherwise. On a three-node cluster raise both stream replica counts through api.extraEnv before the first install:

yaml

api:
  extraEnv:
    - name: PLEXSPHERE_SSE_STREAM_REPLICAS
      value: "3"
    - name: PLEXSPHERE_OBS_STREAM_REPLICAS
      value: "3"

The API never edits a stream that already exists, so a stream created at one replica keeps that count no matter what the environment says later. Raise it out of band:

bash

nats stream edit PLEXSPHERE_NODE_EVENTS --replicas 3

nats stream ls lists the other streams to raise with it.

A replica count above the number of JetStream servers fails stream creation, and the API exits at boot with plexsphere: provision sse node-events stream: and the server's error. The server side is in the JetStream clustering documentation.

OpenBao

The dev stack runs server -dev (deploy/local/base/openbao/deployment.yaml), which keeps its storage in memory and loses every secret on restart. A production install needs integrated raft storage on persistent volumes with three or five voters, auto-unseal so a routine restart needs no human, and the custody rules in Seal-key custody.

A sealed or unreachable OpenBao costs what OpenBao (secret backend) describes. When the install selects the OpenBao audit pepper store (audit.pepper.mount), an API Pod that starts while that store is unreachable does not become ready: the probe is latched at boot, so it gates /readyz hard until its first success.

OpenBao documents both halves itself: integrated storage and seal and unseal.

Object store

plexsphere and its reference backup jobs write three buckets, and each needs the durability of your own S3-compatible service. plexsphere keeps no second copy of what it writes there, and what plexsphere does not back up lists the rest of the state a restore does not bring back.

The action-output bucket, named in actions.objectStoreBucket. The chart ships that value empty, so every install names its own bucket. The callback service mints presigned PUT URLs against it for action output over the inline ceiling.
The Postgres dump bucket, which the reference backup job uploads the nightly pg_dumpall to.
The OpenBao snapshot bucket, which takes the four-hourly raft snapshot.

What those buckets have to provide:

Replication across failure domains. The loss of one domain then leaves a copy standing.
Versioning, or object lock, on the two backup buckets. A leaked job credential then writes a new version instead of replacing the history behind it.
A lifecycle rule that keeps backups at least 35 days, the retention the coverage table records.
Server-side encryption. The Postgres dump holds every tenant's data in one file.
Credentials of its own on the snapshot bucket, independent of the primary store, so a credential that reaches the action-output bucket reaches no OpenBao snapshot.

Availability ​

What the charts run ​

What an upgrade, a drain and a node failure cost ​

At one replica ​

At two or more replicas ​

The one-time move onto TLS ​

What the loss of a workload costs ​

The signer runs one replica ​

Postgres ​

NATS JetStream ​

OpenBao ​

Object store ​

See also ​