Install plexsphere with Helm

The Helm charts under deploy/charts/ install the control plane and its signing service into a Kubernetes cluster:

• plexsphere: the API, the console, the migrations that run before every install and upgrade, the API's cluster RBAC, optional Gateway API routes, and SpiceDB as an optional dependency, enabled by default;
• plexsphere-signer: the service that holds the signing key;
• spicedb: the authorization engine, normally installed through the plexsphere chart.

Postgres, NATS, OpenBao, the OIDC issuer, the object store and the observability backends stay yours to run: the charts point at them and never install them. No chart renders a Secret. Every credential comes from a Secret you create, which the chart reads by name.

Production checklist

A production install runs the sections of this page in this order.

1. Prerequisites: the cluster, cert-manager, Postgres, NATS with JetStream, and images the cluster can pull.
2. Create the namespace and the Secrets: every credential the charts read, each one from a Secret you create.
3. Write the values: one values file per chart. Two values move the release off what the dev stack runs. The signing key custody is settled in Choose a key provider, part of this section; the audit.pepper keys point at the OpenBao mount that The audit pepper store prepares back in step 2.
4. Size the workloads: the requests and limits the charts apply, and the floor a namespace quota has to fit.
5. Install: the control plane first, then the signer.
6. Verify: /readyz on the API and on the signer.
7. Expose the API and the console: the Gateway, the two routes, and the listener that terminates TLS.
8. Install the blueprints you offer: the XRD and the Composition of every blueprint version a Project provisions from, on every management cluster.
9. Restrict network access: the default-deny policies and the peers you readmit.
10. Back up and rehearse a restore: the scheduled jobs, and the drill that proves they restore.

A release that has been through every step serves TLS on its public endpoint and on SpiceDB, reads every credential from a Secret an operator created, signs with a hardware or KMS key provider, carries the default-deny network policies, backs up on a schedule, and has passed one restore drill.

Prerequisites

• A Kubernetes cluster, kubectl, helm 3.19 or later, and OpenSSL 3.

• cert-manager 1.15 or later in the cluster. The bundled SpiceDB serves gRPC over TLS, and the spicedb chart renders the cert-manager Certificate that issues its serving certificate, so helm install fails on a cluster whose cert-manager CRDs are absent. Install it from the cert-manager documentation, or issue the certificate yourself and name its Secret in spicedb.tls.existingSecret, as The SpiceDB certificate describes.

• A Postgres database for the core schema. SpiceDB can share it, as the dev stack does, or use its own database; see Run database migrations for a dedicated spicedb database.

• NATS with JetStream enabled, reachable from the cluster.

• Images of the API, the migrations, the console and the signer that the cluster can pull. A release publishes them to ghcr.io/plexsphere/<name> as private packages that members of the plexsphere organisation pull after docker login ghcr.io, and each released chart defaults its images to the release's own. Without that access, or to install the working tree, build the images and push them to a registry you control:

  make docker-build console-image
  REGISTRY=registry.example.com/plexsphere
  TAG=my-build
  for image in plexsphere plexsphere-signer plexsphere-migrate; do
    docker tag "$image:ci" "$REGISTRY/$image:$TAG"
    docker push "$REGISTRY/$image:$TAG"
  done
  docker tag console:dev "$REGISTRY/console:$TAG"
  docker push "$REGISTRY/console:$TAG"

  A cluster whose nodes hold no credential for that registry needs a pull secret: create it as in The registry credential and name it in imagePullSecrets.

Availability states what Postgres, NATS JetStream, OpenBao and the object store must provide in production.

Create the namespace and the Secrets

Every Secret below lives in the release namespace under the name the charts read by default. The values reference names the value that points a chart at a different Secret. Keep a copy of every key in your secret manager: a reinstall that generates new keys cannot read what the old ones sealed.

NS=plexsphere
kubectl create namespace "$NS"

The kubectl create secret commands below put plaintext on the operator's shell. To deliver the same Secrets from a secret manager instead, the reference overlay at deploy/local/overlays/prod-reference/kustomization.yaml carries a commented SealedSecret and a commented ExternalSecret for plexsphere-signer-seed; both produce the Secret name the charts read, and the same shape delivers plexsphere-cursor-keys and the other Secrets in this section. The overlay is a reference for those templates, not a second install path.

The registry credential

Pods pull the API, migrate, console and signer images with the credential in a kubernetes.io/dockerconfigjson Secret. Create it before helm install: the migrate hook is the first Pod of the release and pulls before anything else exists, so a missing Secret stops the install there.

kubectl -n "$NS" create secret docker-registry plexsphere-registry \
  --docker-server=ghcr.io \
  --docker-username=<user> \
  --docker-password=<token>

--docker-server is the host the images come from: ghcr.io for a release's images, the host of your own registry otherwise, such as registry.example.com. For ghcr.io the password is a personal access token with the read:packages scope. Write the values names the Secret in imagePullSecrets. Skip this step when every node already holds a credential for the registry.

Database and SpiceDB

kubectl -n "$NS" create secret generic plexsphere-database \
  --from-literal=DATABASE_URL='postgres://plexsphere:<password>@postgres.example.com:5432/plexsphere?sslmode=require'
kubectl -n "$NS" create secret generic spicedb-datastore \
  --from-literal=SPICEDB_DATASTORE_CONN_URI='postgres://plexsphere:<password>@postgres.example.com:5432/plexsphere?sslmode=require'
kubectl -n "$NS" create secret generic spicedb-credentials \
  --from-literal=SPICEDB_GRPC_PRESHARED_KEY="$(openssl rand -hex 32)"

The signing seed and its public key

The signer signs with an Ed25519 key derived from a 32-byte seed. The API needs the matching public key. OpenSSL reads a raw seed only behind the PKCS#8 header the printf below supplies:

openssl rand -out seed 32
SIGNING_PUBLIC_KEY_B64="$({ printf '\x30\x2e\x02\x01\x00\x30\x05\x06\x03\x2b\x65\x70\x04\x22\x04\x20'; cat seed; } \
  | openssl pkey -inform DER -pubout -outform DER | tail -c 32 | base64)"
kubectl -n "$NS" create secret generic plexsphere-signer-seed --from-file=seed=seed

The API keys

Thirteen cursor HMAC keys, the NSK wrap key that seals node secret keys at rest, and the signing public key from the previous step:

args=()
for key in PROJECTS CLOUDS PROVIDER_BUNDLES CLOUD_CREDENTIALS DOMAINS IDENTITIES \
  INVITATIONS AUDIT AUTHZ NODES CAPABILITIES HOOKS INTEGRITY_VIOLATIONS; do
  args+=(--from-literal="${key}_CURSOR_HMAC_KEY=$(openssl rand -hex 32)")
done
kubectl -n "$NS" create secret generic plexsphere-cursor-keys "${args[@]}" \
  --from-literal=NSK_WRAP_KEY_B64="$(openssl rand -base64 32)" \
  --from-literal=SIGNING_PUBLIC_KEY_B64="$SIGNING_PUBLIC_KEY_B64"

The signer's mutual TLS

The API reaches the signer over mutual TLS. One private CA signs the signer's server certificate, which must name the host plexsphere-signer (the Service name the API dials), and the API's client certificate, whose SPIFFE ID is the one the signer admits:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout ca.key -out ca.crt -days 365 -subj "/CN=plexsphere-signer-ca" \
  -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout signer.key -out signer.csr -subj "/CN=plexsphere-signer"
openssl x509 -req -in signer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out signer.crt \
  -extfile <(printf 'subjectAltName=DNS:plexsphere-signer,DNS:plexsphere-signer.%s.svc,DNS:plexsphere-signer.%s.svc.cluster.local\nextendedKeyUsage=serverAuth\nkeyUsage=critical,digitalSignature\n' "$NS" "$NS")
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout client.key -out client.csr -subj "/CN=plexsphere-access-signer-client"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out client.crt \
  -extfile <(printf 'subjectAltName=URI:spiffe://plexsphere.example/core-binary\nextendedKeyUsage=clientAuth\nkeyUsage=critical,digitalSignature\n')

kubectl -n "$NS" create secret tls plexsphere-signer-tls --cert=signer.crt --key=signer.key
kubectl -n "$NS" create secret generic plexsphere-signer-client-ca --from-file=client-ca.crt=ca.crt
kubectl -n "$NS" create secret generic plexsphere-access-signer-client \
  --from-file=client.crt --from-file=client.key --from-file=ca.crt

The SpiceDB certificate

A stock install needs nothing from this section: the spicedb chart renders a Certificate for the Secret spicedb-tls, cert-manager writes it, SpiceDB serves the leaf and the API verifies it against the ca.crt of the same Secret. Two cases need more: a cluster without cert-manager, which is the recipe below, and certificates that come from an issuer of your own, which is the issuerRef at the end.

Without cert-manager, issue the certificate yourself. One private CA signs SpiceDB's server certificate, which names the host spicedb.$NS.svc the API dials and the shorter spicedb.$NS the in-cluster resolver also answers. The unqualified spicedb is deliberately not among them: a bare Service name belongs to no namespace, so a CA that signs it hands the same identity to every namespace of the cluster, and the API trusts that CA and nothing else. The API reads the CA from the same Secret, so all three files go in together, and kubectl create secret tls takes a certificate and a key and no third file. Create a generic Secret and set the kubernetes.io/tls type by hand:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout spicedb-ca.key -out spicedb-ca.crt -days 3650 -subj "/CN=spicedb-ca" \
  -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout tls.key -out spicedb.csr -subj "/CN=spicedb"
openssl x509 -req -in spicedb.csr -CA spicedb-ca.crt -CAkey spicedb-ca.key -CAcreateserial -days 365 -out tls.crt \
  -extfile <(printf 'subjectAltName=DNS:spicedb.%s,DNS:spicedb.%s.svc\nextendedKeyUsage=serverAuth\nkeyUsage=critical,digitalSignature\n' "$NS" "$NS")

kubectl -n "$NS" create secret generic spicedb-tls --type=kubernetes.io/tls \
  --from-file=tls.crt --from-file=tls.key --from-file=ca.crt=spicedb-ca.crt

Name that Secret in the values, or the chart renders its Certificate anyway and the install fails on the missing CRDs:

spicedb:
  tls:
    existingSecret: spicedb-tls

A fullnameOverride on the spicedb chart moves the host the API dials, so the certificate's names have to follow it.

With cert-manager installed and an issuer of your own, name that issuer instead of creating anything. The chart then renders the leaf Certificate against it and renders neither its self-signed Issuer nor the CA above it:

spicedb:
  tls:
    certManager:
      issuerRef:
        name: internal-ca
        kind: ClusterIssuer
        group: cert-manager.io

An issuer shared with other namespaces, a ClusterIssuer in particular, signs whatever name a Certificate asks it for, and the API trusts it as its only root. Put a name-constrained Issuer or cert-manager's approver-policy in front of such an issuer, or the identity of your SpiceDB is only as narrow as the set of principals that may create a Certificate against it.

spicedb.tls.existingSecret and spicedb.tls.certManager.issuerRef are two ways to source the same Secret, and setting both fails the render. The API reads its trust root from the ca.crt entry cert-manager writes beside the leaf, which an issuer publishes only when it has a CA to publish: a CA or self-signed issuer does, and a public ACME issuer does not. Point authz.spicedb.caSecret and authz.spicedb.caKey at a Secret that carries the root when your issuer leaves ca.crt out.

The audit pepper store

The API derives audit pseudonyms from a pepper per Domain, which it keeps in a KV v2 mount of your OpenBao. With BAO_ADDR and BAO_TOKEN set for a token that may enable mounts, write policies and configure auth methods, enable the mount plexsphere. To reuse a KV v2 mount you already run, skip this command and put that mount's path in place of plexsphere below:

bao secrets enable -path=plexsphere kv-v2

The API's policy grants create and read on the pepper paths. It withholds update, so the API's credential cannot rotate or overwrite a pepper once it exists:

bao policy write plexsphere-audit-pepper - <<'EOF'
path "plexsphere/data/audit/pepper/*" {
  capabilities = ["create", "read"]
}
EOF

Enable AppRole (skip the first command when it is already enabled), create a role bound to the policy, and store the role id and a new secret id in the Secret the API logs in with:

bao auth enable approle
bao write auth/approle/role/plexsphere-audit-pepper \
  token_policies=plexsphere-audit-pepper token_ttl=20m token_max_ttl=1h
kubectl -n "$NS" create secret generic plexsphere-audit-pepper-approle \
  --from-literal=ROLE_ID="$(bao read -field=role_id auth/approle/role/plexsphere-audit-pepper/role-id)" \
  --from-literal=SECRET_ID="$(bao write -f -field=secret_id auth/approle/role/plexsphere-audit-pepper/secret-id)"

On a cluster that logs in to OpenBao through its Kubernetes auth method, bind a Kubernetes auth role to the same policy and set audit.pepper.kubernetesRole instead of audit.pepper.appRoleSecret, plus audit.pepper.kubernetesJwtPath when the service-account token is not at the default path.

The API creates the platform pepper when it boots and each Domain's pepper on first use. It refuses to boot while the store is unreachable or sealed. It never replaces a pepper that is missing for an audit chain that already holds rows: it refuses to boot when that is the platform chain, and refuses that Domain's audit writes and erasure requests otherwise, until you restore the secret.

Dev clusters only: audit.allowInsecurePepper: true boots the API with the deterministic static pepper instead of an OpenBao store. That pepper makes audit pseudonyms reversible, and the value cannot be combined with audit.pepper.mount.

An install that ran on audit.allowInsecurePepper keeps audit rows whose pseudonyms derive from the static pepper, the SHA-256 of each chain's 16-byte id. Pointed at an empty audit.pepper store, the upgraded API finds those chains' peppers missing and refuses them as above: a new pepper would give each recorded subject a second pseudonym, and erasure would no longer find the rows written before. Before you switch, either install on an empty database, or carry the static pepper of every chain that holds rows into the mount. Carrying them over keeps those chains' pseudonyms as reversible as they are today.

Each pepper is the SHA-256 of the chain id's 16 raw bytes, so the derivation has to decode the id's hex digits to bytes rather than hash its text. Check the pipeline against the platform chain's known value first — a pepper that does not match the one the static resolver produced is adopted silently, and every pseudonym recorded under the static pepper stays unresolvable until you overwrite it:

printf '%s' 00000000-0000-0000-0000-706c6174666d | tr -d '-' | xxd -r -p |
  openssl dgst -sha256 -binary | base64

That must print FRkb35iH3qsFf8e260aXv46odgvo2hqsQrsqBNzBUGg=. Then stop the old API before you read the chain heads. The query below is a point-in-time snapshot: a chain that gains its first row while the old API keeps serving is missed, and the switched API refuses that Domain's audit writes, its erasure requests and the cursor paging on every list surface its callers open, until you write that pepper by hand.

kubectl -n "$NS" scale deployment/plexsphere --replicas=0
kubectl -n "$NS" wait --for=delete pod -l app.kubernetes.io/name=plexsphere --timeout=2m

With the API stopped and DSN set to the core database's connection string, carry every chain over:

set -euo pipefail
psql "$DSN" -At -c \
  "SELECT domain_id FROM plexsphere.audit_log_chain_head WHERE next_seq > 1" |
while read -r id; do
  if bao kv get -field=pepper "plexsphere/audit/pepper/${id}" >/dev/null 2>&1; then
    echo "already present ${id}"
    continue
  fi
  pepper="$(printf '%s' "$id" | tr -d '-' | xxd -r -p |
    openssl dgst -sha256 -binary | base64)"
  bao kv put -cas=0 "plexsphere/audit/pepper/${id}" pepper="${pepper}"
  echo "carried over ${id}"
done

set -euo pipefail stops the run on the first failure instead of leaving a half-populated mount behind, and the loop names every chain the query returned — the ones it wrote and the ones that already held a pepper. It skips the latter instead of failing on them, so re-running it after a failure resumes at the chain it stopped on. The -cas=0 is the backstop under that skip: it refuses to overwrite a pepper the loop did not read.

The cloud credential store

Skip this section unless the install provisions machines on a Cloud. The API stores every cloud credential it issues in a KV v2 mount of your OpenBao, and the External Secrets Operator (ESO) copies each one into the Project namespaces whose resources use it. Enable the mount cloud-credentials:

bao secrets enable -path=cloud-credentials kv-v2

The API only writes credentials, so its policy grants create and update and no read:

bao policy write plexsphere-cloud-credentials - <<'EOF'
path "cloud-credentials/data/clouds/*" {
  capabilities = ["create", "update"]
}
EOF

The policy keeps the API from reading credentials out of OpenBao only. The chart's ClusterRole reads Secrets in every namespace, so the API can read each credential ESO copies into a Project namespace.

Bind a Kubernetes auth role to the policy for the API's ServiceAccount plexsphere (Kubernetes auth enabled and configured for the cluster), or create an AppRole as for the audit pepper and name its Secret in cloudCredentials.appRoleSecret:

bao write auth/kubernetes/role/plexsphere-cloud-credentials \
  bound_service_account_names=plexsphere \
  bound_service_account_namespaces="$NS" \
  policies=plexsphere-cloud-credentials audience=vault ttl=20m max_ttl=1h

ESO reads the credentials with a policy of its own and a role bound to its ServiceAccount; adjust the name and namespace to your ESO install:

bao policy write eso-cloud-credentials - <<'EOF'
path "cloud-credentials/data/clouds/*" {
  capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/eso-cloud-credentials \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=eso-cloud-credentials audience=vault ttl=1h

Install one ClusterSecretStore that logs ESO in with that role. The conditions limit it to the namespaces the Management Fleet creates for Projects, which carry both labels. The store names no path: the broker puts the mount into every key it asks ESO for, so a credential issued under any mount resolves.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: plexsphere-cloud-credentials
spec:
  conditions:
    - namespaceSelector:
        matchLabels:
          app.kubernetes.io/managed-by: plexsphere
          app.kubernetes.io/component: project-namespace
  provider:
    vault:
      server: https://openbao.example:8200
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso-cloud-credentials
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
            audiences: [vault]

Then set the values. providerConfigApiGroups lists the ProviderConfig group of every provider your Clouds name; the chart grants the API providerconfigs in exactly those groups and externalsecrets in external-secrets.io:

cloudCredentials:
  kvMount: cloud-credentials
  openbaoAddress: https://openbao.example:8200
  kubernetesRole: plexsphere-cloud-credentials
  defaultTtl: 2160h
provisioning:
  cloudCredentialsStore: plexsphere-cloud-credentials
  providerConfigApiGroups:
    - openstack.m.crossplane.io

The API logs in at boot and logs cloud credential store ready with the mount, the endpoint, the auth method and the default lifetime. It refuses to boot while the store is unreachable. A credential issued without a lifetime of its own expires after defaultTtl, 24 hours when it is unset. While OpenBao is sealed or unreachable, issuing a credential answers 503 openbao_unavailable and stores nothing.

Write the values

values-plexsphere.yaml for the control plane:

externalUrl: https://plexsphere.example.com
image:
  tag: my-build
imagePullSecrets:
  - name: plexsphere-registry
api:
  image:
    repository: registry.example.com/plexsphere/plexsphere
migrate:
  image:
    repository: registry.example.com/plexsphere/plexsphere-migrate
console:
  image:
    repository: registry.example.com/plexsphere/console
signing:
  keyId: did:plexsphere:domain/example#k1
nats:
  url: nats://nats.example.com:4222
actions:
  objectStoreBucket: plexsphere-action-output
objectStore:
  endpoint: https://s3.eu-central-1.amazonaws.com
  region: eu-central-1
audit:
  pepper:
    endpoint: https://openbao.example.com:8200
    mount: plexsphere
    appRoleSecret: plexsphere-audit-pepper-approle

externalUrl is the origin operators and nodes reach the API at; the action, access and enrolment callbacks, the OIDC callback URL and the plexd download URL derive from it. signing.keyId is the identifier plexd records next to the signing public key. When the object store needs static credentials, put S3_ACCESS_KEY and S3_SECRET_KEY in a Secret and name it in objectStore.credentialsSecret.

audit.pepper names the OpenBao address, the KV v2 mount and the AppRole Secret from The audit pepper store. The values reference lists every audit.pepper value.

imagePullSecrets names the Secret from The registry credential for the API, migrate and console Pods; leave it out when the nodes pull without one. The bundled SpiceDB pulls the public authzed/spicedb image and reads its own list, so a SpiceDB image mirrored into a private registry takes the Secret under spicedb::

spicedb:
  image:
    repository: registry.example.com/mirror/spicedb
  imagePullSecrets:
    - name: plexsphere-registry

Those values name neither spicedb.tls nor authz.spicedb.caSecret, because the defaults already serve the bundled SpiceDB over TLS and point the API at the CA that signed its certificate.

values-signer.yaml for the signer:

image:
  repository: registry.example.com/plexsphere/plexsphere-signer
  tag: my-build
imagePullSecrets:
  - name: plexsphere-registry
allowSpiffeId: spiffe://plexsphere.example/core-binary

That file needs one more thing before it renders: the key provider.

Choose a key provider

keyProvider has no default. The signer chart refuses the render until the values name one of pkcs11, aws-kms, gcp-kms and software, with the settings that provider needs. No default is safe to inherit: software keeps the signing key in the container's memory, and the other three reach a key you provisioned, at an address only you know.

An upgrade of a release installed before this was a choice fails the same way, because the release's own values name no provider either. Such a release runs the software provider, so adding keyProvider: software beside the devMode.enabled: true it already carries keeps it where it is.

AWS KMS. The Pod reaches KMS through the IAM role the ServiceAccount annotation names. The EKS webhook injects a projected token for that role itself, so the chart's automountServiceAccountToken: false does not stand in the way; GKE reaches its own service account through the metadata server. On a cluster with neither, name a Secret holding the static credentials in extraEnvFrom instead:

keyProvider: aws-kms
awsKms:
  region: eu-central-1
  keyArn: arn:aws:kms:eu-central-1:111122223333:key/<key-id>
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/plexsphere-signer

Google Cloud KMS. keyVersion names the CryptoKeyVersion, version included: the signer signs with that version and no other.

keyProvider: gcp-kms
gcpKms:
  keyVersion: projects/my-project/locations/europe-west3/keyRings/plexsphere/cryptoKeys/signer/cryptoKeyVersions/1
serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: plexsphere-signer@my-project.iam.gserviceaccount.com

PKCS #11. The image ships no vendor module, so carry one in: an init container copies it out of the vendor's image into an emptyDir both containers mount, and modulePath points at it there. Address the token by tokenLabel or by slot, never by both. slot is a string, so the first slot of a token is "0". The PIN reaches the process as a mounted file alone, from the Secret pin.existingSecret names:

keyProvider: pkcs11
pkcs11:
  modulePath: /opt/vendor/libvendorpkcs11.so
  tokenLabel: plexsphere-signer
  pin:
    existingSecret: plexsphere-signer-hsm-pin
extraInitContainers:
  - name: copy-vendor-module
    image: registry.example.com/vendor/pkcs11:1.2.3
    command: ["cp", "/usr/lib/libvendorpkcs11.so", "/opt/vendor/"]
    volumeMounts:
      - name: vendor-module
        mountPath: /opt/vendor
extraVolumes:
  - name: vendor-module
    emptyDir:
      medium: Memory
      sizeLimit: 64Mi
extraVolumeMounts:
  - name: vendor-module
    mountPath: /opt/vendor
    readOnly: true

Create the PIN Secret with the key pkcs11.pin.key names, pin by default:

kubectl -n "$NS" create secret generic plexsphere-signer-hsm-pin \
  --from-literal=pin="$HSM_USER_PIN"

Software. It keeps the signing seed in the container's memory, derived from the Secret seed.existingSecret names, and the signer refuses it unless devMode.enabled sets PLEXSPHERE_ENV=dev. The chart refuses the pair in the other direction too: dev mode with any other provider, which would mark a release holding a real key as a dev deployment.

keyProvider: software
devMode:
  enabled: true

The values reference lists every key of every provider, and render-time failures lists the message each missing value produces.

Move an install off the software provider

Naming a production provider on a release that has been signing with software is not enough on its own. The signer's boot reconcile reads the scope's active signing_key row and asks the provider it was configured with for the public half of the handle that row stores. A software handle means nothing to an HSM or to a KMS, so the new provider answers "key not found" and the signer exits before it serves anything:

plexsphere-signer: register signing reconcile probe: bootstrap: signing
reconcile initial run: signing: ReconcileKeys: provider.PublicKey for
scope "platform" key "signing-platform-…": …: signing: key not found

The middle of that chain is whatever the new provider said about the handle it was handed — kms-aws: GetPublicKey handle "software:…" on AWS, pkcs11: object class=… on an HSM.

The provider that mints on the new back-end is the same boot reconcile, and it only mints for a scope that has no active key and no rotation in flight. So the move is: retire the software rows first, then restart on the new provider.

Close any rotation still in its overlap window before you start — plexctl signing close — otherwise the reconcile verifies the incoming half of that rotation and fails the same way. Then stop the signer, so nothing signs with the key you are about to retire:

kubectl -n "$NS" scale deployment/plexsphere-signer --replicas=0
kubectl -n "$NS" wait --for=delete pod \
  -l app.kubernetes.io/name=plexsphere-signer --timeout=2m

With DSN set to the core database's connection string, retire what the software provider minted for the scope this signer serves. The statement below is the platform scope; for a Domain scope replace the predicate with scope_kind = 'domain' AND scope_id = '<domain-uuid>':

psql "$DSN" -c \
  "UPDATE plexsphere.signing_key
      SET state = 'retired', updated_at = now()
    WHERE scope_kind = 'platform'
      AND scope_id IS NULL
      AND state <> 'retired'"

Now run the helm upgrade that sets keyProvider and its settings, and scale the signer back up. The boot reconcile finds no active key, mints one on the new provider — on a cloud KMS it resolves the key you provisioned rather than creating one — and writes the row that every later call resolves through.

The retired rows stay. Their cached public halves keep serving from the public-key bundle, so every signature the software key produced still verifies; nothing re-signs with it, because a retired key never signs again. Age them out later by stamping valid_until on them when you no longer need those signatures verifiable.

This is a one-way move. Going back means the same procedure in the other direction, and the seed that derived the old software key has to still exist for its signatures to keep verifying.

Size the workloads

Every container the charts run carries a request and a limit. These are the chart defaults, which apply to an install that sets no resources value:

Workload	Values key	CPU request	Memory request	CPU limit	Memory limit	Default replicas
API	api.resources	50m	128Mi	500m	512Mi	1, from api.replicas
migrate hook	migrate.resources	50m	64Mi	500m	256Mi	One Job per install and per upgrade
Console	console.resources	10m	16Mi	250m	64Mi	1, from console.replicas
SpiceDB	spicedb.resources, installed through the plexsphere chart	50m	64Mi	500m	256Mi	1, from spicedb.replicas
Signer	resources of the signer chart	25m	64Mi	500m	256Mi	1, fixed; the chart exposes no value to raise

At those replica counts the four long-running Pods request 135m of CPU and 272Mi of memory together, and their limits sum to 1750m and 1088Mi. That is the floor a namespace quota or a single node has to fit. The migrate hook adds 50m and 64Mi of requests, and 500m and 256Mi of limits, for the length of one Job. Its share is transient and still has to fit: the pre-upgrade hook runs while the API Pods are still serving.

These figures are chart defaults, not load-tested tiers. No tier measured against a real workload exists anywhere in this repository. Capacity and scale targets calls its own per-Domain numbers design-phase orientation figures, and it describes the load-test harness an operator drives against a provisioned deployment to measure their own.

When a default is too small

• A container shows OOMKilled in kubectl describe pod: raise that workload's memory limit.
• container_cpu_cfs_throttled_periods_total rises for a container: raise its CPU limit. Throttling caps the work the container does without killing it, so nothing restarts and latency grows instead.
• A capacity ratio in Capacity and scale targets crosses 80%: the Domain is approaching one of its capacity targets, which is a load signal rather than a container-limit one. Scale the API out with api.replicas, as Run more than one API replica describes.

What the charts do not size

Postgres, NATS JetStream, OpenBao and the object store are yours to run, and no value here sizes them; Availability states what each has to provide in production. The reference backup jobs stage their artefacts on volumes of their own, sized in Size the staging volumes before you apply.

Install

Install the control plane first. Its migrations run as a pre-install hook, before any workload starts, and the signer keeps its keys in tables those migrations create:

helm dependency update deploy/charts/plexsphere
helm install plexsphere deploy/charts/plexsphere --namespace "$NS" \
  --values values-plexsphere.yaml --wait --timeout 10m
helm install plexsphere-signer deploy/charts/plexsphere-signer --namespace "$NS" \
  --values values-signer.yaml --wait --timeout 5m

To install a published release instead of the working tree, log helm in to ghcr.io and install the charts by version. The released plexsphere chart carries SpiceDB already, so it needs no dependency update:

helm registry login ghcr.io --username <github-user>
helm install plexsphere oci://ghcr.io/plexsphere/charts/plexsphere --version X.Y.Z \
  --namespace "$NS" --values values-plexsphere.yaml --wait --timeout 10m
helm install plexsphere-signer oci://ghcr.io/plexsphere/charts/plexsphere-signer --version X.Y.Z \
  --namespace "$NS" --values values-signer.yaml --wait --timeout 5m

Leave the image.tag and image.repository values out when you install a release: each chart's images default to the release's own. The releases runbook shows how to verify a chart's signature before you install it.

Verify

The API answers /readyz and reports the version it was built from. Its NetworkPolicy admits only the peers networkPolicy.gatewayFrom and networkPolicy.metricsFrom name, so a port-forward is the check that works on any cluster:

kubectl -n "$NS" get deployments
kubectl -n "$NS" port-forward deployment/plexsphere 8080:8080 &
curl -fsS http://127.0.0.1:8080/readyz
curl -fsS http://127.0.0.1:8080/v1/version
kill %1

The signer's NetworkPolicy admits only the API on its gRPC port, so reach its /readyz through a port-forward:

kubectl -n "$NS" port-forward deployment/plexsphere-signer 8081:8081 &
curl -fsS http://127.0.0.1:8081/readyz
kill %1

Next, seed the tenancy Domains and set the platformIdp and platformOperator values so the first Platform Operator can sign in.

Run more than one API replica

Add replicas under the api key of values-plexsphere.yaml, next to image:

api:
  replicas: 2

Apply it with the control plane upgrade:

helm upgrade plexsphere deploy/charts/plexsphere --namespace "$NS" \
  --values values-plexsphere.yaml --wait --timeout 10m

Above one replica the chart also renders a PodDisruptionBudget named plexsphere, which keeps one API Pod available, and a topology spread constraint, which places the API Pods on different nodes where the scheduler can. On a single-node cluster it still schedules every replica on that node.

A node drain waits for the budget: it evicts a ready API Pod only when another one stays ready. When both Pods run on the drained node, it evicts the second one after the first one's replacement is ready on another node. An API Pod that is not ready is evicted without waiting, because it serves no traffic.

An upgrade rolls the API Pods with the Deployment's default rolling update, which starts new Pods before it stops old ones; at two or three replicas it stops an old Pod only once a new one is ready. The pre-upgrade hook migrates the schema before the first Pod is replaced, so until the last old Pod stops, old Pods serve against the new schema. When a release's migrations drop or rename a column or table the previous release reads, the old Pods answer the requests that touch it with errors for that window. A single replica has the same window.

Two rate limits are kept in memory in each API Pod rather than in the database: the per-IP limit on the auth surfaces, /v1/register and /v1/keys/rotate (see Rate limiting on the auth surfaces), and the per-Node limit on edge-PSK fetches. Every replica admits the full budget, so api.replicas Pods admit up to api.replicas times the configured rate from one client.

Each API Pod holds up to 46 Postgres connections with the default pool sizes: the interactive, worker and bulk pools (20, 10 and 5), the readiness probe's own pool (up to 10), and the background leader lock's session (1). While a rolling upgrade runs, old Pods keep their connections until they exit, which takes up to 30 seconds after they are told to stop, so up to twice api.replicas Pods hold connections at once. Keep 46 connections for each of those Pods, plus the signer's pool (up to 10, and twice that while the signer rolls) and SpiceDB's connections when it shares the database, under the server's max_connections less its superuser_reserved_connections, or lower the API's pool ceilings through api.extraEnv:

api:
  replicas: 2
  extraEnv:
    - name: PLEXSPHERE_POOL_INTERACTIVE_MAX_CONNS
      value: "12"
    - name: PLEXSPHERE_POOL_WORKER_MAX_CONNS
      value: "6"

Every pool knob is in the connection pools reference.

The console and the bundled SpiceDB take console.replicas and spicedb.replicas the same way the API takes api.replicas, and above one replica the chart renders the same budget and hostname spread for them. The signer runs one replica. What Postgres, NATS JetStream, OpenBao and the object store must provide is in Availability.

Every value the chart reads is in the Helm charts reference.

Upgrade

Upgrade the control plane first; its pre-upgrade hook migrates the schema before the API rolls. Then upgrade the signer:

helm upgrade plexsphere deploy/charts/plexsphere --namespace "$NS" \
  --values values-plexsphere.yaml --wait --timeout 10m
helm upgrade plexsphere-signer deploy/charts/plexsphere-signer --namespace "$NS" \
  --values values-signer.yaml --wait --timeout 5m

Upgrading onto the cloud credential login

Three changes reach a release that used cloudCredentials:

• cloudCredentials.tokenSecret is refused at render. A published image never read the token; log the API in as in The cloud credential store with cloudCredentials.kubernetesRole or cloudCredentials.appRoleSecret.
• A release with cloudCredentials.kvMount set and no login stops the render, and an API started with the mount and no login stops at boot.
• The broker renders each ProviderConfig into the Project namespace and without spec.endpoint. Its next Server-Side Apply drops spec.endpoint from a ProviderConfig it applied before, because its field manager no longer sends the field.

Upgrading an existing release onto TLS

A release installed before the bundled SpiceDB served TLS dials it in plaintext. This revision moves both halves of that transport at once — spicedb.tls.enabled on and authz.spicedb.insecure off — and that upgrade costs a one-time authorization outage. Plan it into a maintenance window.

The outage is the rollout itself. Helm applies both Deployments in one pass and orders neither, so the new SpiceDB Pod becomes Ready and the old one terminates while an old API Pod, still carrying SPICEDB_PRESHARED_KEY_INSECURE from the previous revision, is alive and reconnecting its dropped gRPC connection into a TLS listener in plaintext. For as long as that lasts, a gated operation the last-known-good check cache cannot serve answers 503 with the problem code authz_unavailable, and every mutation fails, because mutations never consult that cache. Two or more API replicas do not close it: each old Pod keeps the old environment until it is replaced.

There is no staged cutover to avoid it. The chart refuses a release whose two halves disagree about the transport, on purpose: such a release installs cleanly and then fails inside the gRPC handshake, where no Pod status reports it and every permission check dies with it.

What you control is the length of the window. Have the Secret both Deployments mount in place before the upgrade; otherwise both sets of new Pods block in ContainerCreating until cert-manager has issued, and that latency lands on top of the rollout with nothing waiting on it or reporting it. Issue the certificate as The SpiceDB certificate describes — by hand, or from an issuer of your own into a Secret you name — create it in the namespace, and name it before you upgrade:

spicedb:
  tls:
    existingSecret: spicedb-tls

Then run the helm upgrade above and watch the window close:

kubectl -n "$NS" rollout status deployment/spicedb
kubectl -n "$NS" rollout status deployment/plexsphere

Moving to the cert-manager chain the chart renders is a later, separate upgrade: clear spicedb.tls.existingSecret, and the transport does not move while the certificate underneath it is replaced. Wait for the new certificate, then restart the API, because the CA moves even though the transport does not:

kubectl -n "$NS" wait --for=condition=Ready certificate/spicedb-tls --timeout=5m
kubectl -n "$NS" rollout restart deployment/plexsphere
kubectl -n "$NS" rollout status deployment/plexsphere

The wait is the part helm upgrade --wait does not do for you: its readiness wait covers Pods, this upgrade replaces none, and so it returns while cert-manager is still working through the issuer chain the chart just rendered. A restart inside that gap boots the new API Pods on the root the Secret still holds, they reach a SpiceDB still serving the old leaf, their first readiness tick passes, and rollout status reports success on a rollout that moved nothing. The new root lands seconds later, and the release ends in the state the next paragraph describes with nothing left to end it. After the wait returns, rollout status can sit for up to a minute the other way round: the new API Pods hold the new root while SpiceDB still serves the old leaf from a file kubelet has yet to refresh in its Pod, so they fail readiness until that refresh and SpiceDB's certificate watcher catch up. That one is loud, and it ends on its own.

The chart's Certificate writes the Secret spicedb-tls, which is the name the recipe above created by hand, so cert-manager overwrites that Secret in place with a leaf from a CA of its own. Nothing in the API Pod template changes — it names the Secret, and the name is the same one — so helm upgrade replaces no Pod, while the API read its trust root once at boot and does not re-read the file. SpiceDB serves the new leaf without a restart, and the next reconnect of the gRPC connection fails on a root the running API does not hold. Nothing reports it: the SpiceDB readiness probe latches on its first success, so /readyz keeps answering 200 and no Pod is evicted, while gated reads fail as they fall off the check cache and every mutation answers 503 with the problem code authz_unavailable. The restart is what ends it.

That holds for any change of the CA behind a Secret whose name stays put, not for this migration alone; Availability says so beside the renewals that cost nothing. The same page has the costs of an ordinary upgrade, which this one is not.

Uninstall

helm uninstall plexsphere-signer --namespace "$NS"
helm uninstall plexsphere --namespace "$NS"

Uninstalling removes the workloads and the API's cluster RBAC. The Secrets, the namespace and the database stay.

Expose the API and the console

Set gateway.enabled to route one hostname through a Gateway: /v1 and /downloads to the API, everything else to the console. Behind a proxy that appends X-Forwarded-For, switch the rate limiter to the forwarded client address; see Run behind a reverse proxy. The API's NetworkPolicy denies the Gateway until networkPolicy.gatewayFrom names its Pods, and the render fails while the list is empty:

externalUrl: https://plexsphere.example.com
gateway:
  enabled: true
  parentRefs:
    - name: public
      namespace: gateway-system
      sectionName: https
  hostnames:
    - plexsphere.example.com
networkPolicy:
  gatewayFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: gateway-system
      podSelector:
        matchLabels:
          app.kubernetes.io/name: envoy
rateLimit:
  clientIpSource: proxy-headers

externalUrl starts with https://. The session cookie's origin, the CSRF allowlist, the OIDC callback URL, the action, access and enrolment callbacks and the plexd download URL all derive from it, so an http:// origin hands every one of them a plaintext scheme the edge does not serve.

/downloads is the plexd download path. A machine the provisioning broker boots fetches its agent from <externalUrl>/downloads/plexd, which the API serves from the plexd release this version pins. The API fetches each platform's binary from the plexd GitHub release on the first request for it, once per replica, and keeps it in memory, so the API Pods need egress to github.com and release-assets.githubusercontent.com on TCP 443 when the first machine is provisioned; the chart's NetworkPolicies restrict ingress only. An install without that path sets provisioning.plexdDownloadSourceUrl to a copy of the release assets that serves <base>/<version>/plexd-<platform> and <base>/<version>/plexd-<platform>.sigstore.json. An install that never provisions a machine never reaches either.

The OIDC callback URL is <externalUrl>/v1/auth/callback, here https://plexsphere.example.com/v1/auth/callback. Register that redirect URI at the IdP. The API sends it as the redirect_uri of every sign-in, because behind a Gateway that terminates TLS the API cannot derive the scheme from the request: the Gateway forwards plain HTTP, and the API does not read X-Forwarded-Proto for the callback. When the IdP has another redirect URI registered, set it in auth.callbackUrl, which takes precedence over externalUrl.

Two responses under /v1 stay open for as long as the client keeps them: a node's event stream (/v1/nodes/{id}/events, server-sent events) and the session attach WebSocket. A Gateway whose request timeout spans the whole response cuts both, and the node reconnects every time the timeout runs out. Envoy Gateway applies a route timeout of 15 seconds by default; turn it off for the API's route with a BackendTrafficPolicy in the release's namespace:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: plexsphere-api
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: plexsphere-api
  timeout:
    http:
      requestTimeout: 0s

Envoy reads a zero duration as no route timeout. The idle timeout of a stream stays at its default, because the API sends a keep-alive on the event stream every 25 seconds and a ping on the attach WebSocket every 30 seconds. A slow fetch from /downloads outlives the default timeout the same way. Other Gateway implementations name the setting differently; whatever its name, it must not cut a response that is still open.

An IdP whose issuer is served under a certificate from a private CA needs that CA in the API container, or every sign-in fails on the discovery request with certificate signed by unknown authority. Mount the CA certificate with api.extraVolumes and api.extraVolumeMounts and point SSL_CERT_DIR at the mount through api.extraEnv:

api:
  extraEnv:
    - name: SSL_CERT_DIR
      value: /etc/plexsphere/idp-ca
  extraVolumes:
    - name: idp-ca
      configMap:
        name: idp-ca
  extraVolumeMounts:
    - name: idp-ca
      mountPath: /etc/plexsphere/idp-ca
      readOnly: true

SSL_CERT_DIR replaces the directories Go scans for CA certificates and keeps the image's bundle file, so the API still verifies GitHub and the ACME directory. SSL_CERT_FILE would replace that bundle and drop the public roots the API also needs.

sectionName names the listener the routes attach to. A parentRef without it attaches the route to every listener of the Gateway, a plaintext port-80 listener included, which then serves /v1, /downloads and the console over HTTP beside the HTTPS listener you meant.

The namespaceSelector and the podSelector in that one entry are read together: the rule admits the Pods the pod selector matches inside the namespaces the namespace selector matches, not either set on its own. The labels above are an example of a Gateway's own Pods. Read the ones your Gateway carries with kubectl -n gateway-system get pods --show-labels.

The Gateway and the redirect

The Gateway is yours; no chart renders one. It terminates TLS on the https listener from a Secret in its own namespace, and its port-80 listener exists to redirect:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: gateway-system
spec:
  gatewayClassName: envoy
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: plexsphere.example.com
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      port: 443
      hostname: plexsphere.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: plexsphere-edge-tls
      allowedRoutes:
        namespaces:
          from: All

certificateRefs reads a kubernetes.io/tls Secret in the Gateway's namespace; a Secret elsewhere needs a ReferenceGrant in its own namespace. allowedRoutes has to admit the release's namespace, which is not the Gateway's in the example above.

The redirect from port 80 is yours too. The chart renders no redirect route: its two routes share the one gateway.parentRefs list, so a redirect would need a second parentRef naming the http listener, and putting that listener in gateway.parentRefs attaches /v1, /downloads and the console to it as well. Apply the route beside the Gateway instead:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: plexsphere-https-redirect
  namespace: gateway-system
spec:
  parentRefs:
    - name: public
      sectionName: http
  hostnames:
    - plexsphere.example.com
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

Provisioned machines and a private CA

A machine the provisioning broker boots fetches plexd from the control plane and registers against it with its system trust store. When the control plane serves a certificate from a private CA, the machine fails its first HTTPS call with certificate signed by unknown authority and never enrols. Put the CA certificates into a ConfigMap and name it in the values:

kubectl -n "$NS" create configmap provisioning-ca --from-file=ca.crt=org-ca.crt

provisioning:
  caBundle:
    configMap: provisioning-ca
    key: ca.crt

The broker then writes the CA into the cloud-init document of every machine it provisions, once for cloud-init's ca_certs module and once as the file /etc/plexd/ca.crt. The CA becomes trusted system-wide on each of those machines, for every program on it and not only for plexd, so name only a CA every provisioned machine may trust. The public roots stay trusted beside it.

The key holds PEM certificates and nothing else. The API refuses to start on a file with a private key, text around the certificates, no certificate, or more than 6144 bytes, and logs the SHA-256 fingerprint of each certificate it accepted. It reads the file once at boot, so a changed ConfigMap takes effect on the next API rollout, kubectl -n "$NS" rollout restart deployment/plexsphere, and only for machines provisioned after it. A ConfigMap or key that does not exist keeps the API Pod in ContainerCreating with a mount error until it exists.

This CA is separate from the CA of an IdP above: SSL_CERT_DIR makes the API trust the IdP, and provisioning.caBundle makes the provisioned machines trust the control plane.

Install the blueprints you offer

The control plane never applies a blueprint's XRD and Composition to a cluster. Importing a blueprint version records it in the catalog, and the API's ClusterRole grants no write on apiextensions.crossplane.io (the reasons are in On the management cluster). Apply both files, at exactly the version you imported, to every management cluster before a Project provisions from that version:

kubectl apply -f xrd.yaml -f composition.yaml

The official catalog's files live under catalog/<slug>/ of github.com/plexsphere/blueprints, at the tag the catalog source tracks. A new version of a blueprint means applying its pair again.

Until the XRD is on the cluster, a resource declared from the version stays in Pending, and the API logs this line on every sweep:

provisioning broker: substrate not ready; skipping resource until its XRD is registered

A composite whose Composition is missing never composes anything, because Crossplane finds no Composition that matches its compositeTypeRef.

Keep every XRD out of any GitOps prune. Deleting an XRD deletes its CRD, every composite of its kind and, through Crossplane, every cloud resource they composed. With Flux, annotate each XRD with kustomize.toolkit.fluxcd.io/prune: disabled.

Restrict network access

networkPolicy.enabled is on by default in all three charts. The plexsphere chart renders three ingress policies:

• plexsphere-default-deny selects every Pod of the release by app.kubernetes.io/instance and carries no ingress rule. Nothing reaches a Pod of the release unless another policy admits it.
• plexsphere admits the peers of networkPolicy.gatewayFrom and of networkPolicy.metricsFrom to the API on TCP 8080. The API serves /v1, /downloads, the health endpoints and /metrics on that one port.
• plexsphere-console admits the peers of networkPolicy.gatewayFrom to the console on TCP 8080, the container port behind the Service's port 80. It is absent when console.enabled is false.

The spicedb chart renders a policy named spicedb that admits the Pods spicedb.networkPolicy.clientPodLabels selects on TCP 50051, the API's labels by default. Set a fullnameOverride on the plexsphere chart and those labels have to follow it: that policy isolates the SpiceDB Pods on its own and is the only rule that readmits the API, so labels missing the API cut it off SpiceDB whether the plexsphere chart's own policies are on or off. As its parent, that chart refuses a render whose clientPodLabels name Pods its API does not carry.

The plexsphere-signer chart renders a policy named plexsphere-signer that admits the API on its gRPC port 8443, and a metrics scraper on TCP 8081 once its own networkPolicy.metricsFrom names one. Its networkPolicy.apiPodLabels has to follow a fullnameOverride of the plexsphere chart too, and nothing catches it if you forget: the signer is a separate release, so no chart can compare the two, and the API does not gate its readiness on the signer. Every Pod goes Ready, and access issuance answers 503 at runtime instead.

Every entry of networkPolicy.gatewayFrom and networkPolicy.metricsFrom is a Kubernetes NetworkPolicyPeer: it sets a podSelector, a namespaceSelector, an ipBlock, or a combination of them. The chart refuses an entry whose selectors name nothing, because that entry matches every source: podSelector: {} is every Pod of the release's namespace to Kubernetes and namespaceSelector: {} is every Pod of every namespace. An empty namespaceSelector beside a podSelector that names labels is a different thing — it widens those labels to every namespace — and renders.

Admit a Prometheus scraper in the monitoring namespace to the API and to the signer:

# values-plexsphere.yaml
networkPolicy:
  metricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus

# values-signer.yaml
networkPolicy:
  metricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus

The policies restrict ingress only. Egress from the release's Pods stays open, and restricting it is your job: the API's egress set is open-ended. It reaches your Postgres, NATS, OpenBao and object store, the IdP, an ACME directory, the Kubernetes API, and every tenant sink destination a Project adds at runtime, none of which the chart can see.

A NetworkPolicy is enforced by the cluster's CNI plugin. Where the plugin does not implement NetworkPolicy the objects are created and ignored, and every Pod stays reachable. Confirm your plugin enforces them before you count on the isolation.

An empty networkPolicy.gatewayFrom admits nobody, which is what a stock install renders. Nothing on the Pod network then reaches the API or the console — not a Gateway, not an ingress controller, not a service mesh — while every Pod goes Ready and a port-forward keeps working. The chart fails the render over it only when gateway.enabled is set, because that is the one edge it can see; for every other edge the notes Helm prints after helm install and helm upgrade carry the warning. Name the Pods of whatever fronts the release, as in Expose the API and the console.

Set networkPolicy.enabled: false on a chart to render none of its policies, for a cluster that isolates Pods by other means. The plexsphere chart's default deny selects the bundled SpiceDB's Pods too, and the only rule that readmits the API to them is the spicedb chart's own policy, so the two keys move together: leaving networkPolicy.enabled on while switching spicedb.networkPolicy.enabled off is refused at render time rather than stalling the rollout on an API that cannot reach SpiceDB.

Back up and rehearse a restore

No chart renders a backup job. Apply the two scheduled jobs from Automation reference, a nightly Postgres dump and a four-hourly OpenBao raft snapshot, and pin their image by digest before you do.

Then run the restore drill once before go-live and on a recurring schedule after that: at least quarterly, and after any change to the backup jobs or to the Postgres or OpenBao major version. The drill restores both stores into an isolated environment from the newest artefacts in the bucket, and it measures the recovery point and the restore time this deployment reaches. What it restores is production secret material — the whole OpenBao secret set and every Postgres role's password hash — so that environment carries production's access controls for as long as it exists, and the drill's last step destroys it.

Without WAL archiving on your own Postgres, the recovery point those two jobs keep is 24 hours for Postgres, the cadence of the dump, and 4 hours for secret material, the cadence of the OpenBao snapshot. Continuous archiving and point-in-time recovery are an operator requirement: the release ships no archiver. Recovery objectives states both bounds beside the RTO each plane targets.

Troubleshooting

Symptom	Cause	Fix
The plexsphere container exits with plexsphere: PLEXSPHERE_AUDIT_PEPPER_MOUNT must be set for production audit pepper resolution; set PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER=true to enable the deterministic static fallback (NOT for production load)	Neither audit.pepper.mount nor audit.allowInsecurePepper is set	Configure audit.pepper as in The audit pepper store.
The plexsphere container exits with plexsphere: PLEXSPHERE_AUDIT_PEPPER_MOUNT and PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER=true are mutually exclusive	api.extraEnv sets PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER beside audit.pepper.mount; the chart itself refuses the combination	Drop one of the two.
The plexsphere container exits with plexsphere: could not reach the OpenBao audit pepper mount: after 3 attempts: …	audit.pepper.endpoint is wrong, OpenBao is sealed or unreachable, or the AppRole credentials are wrong	Check audit.pepper.endpoint, unseal OpenBao, or mint a new secret id into plexsphere-audit-pepper-approle.
The plexsphere container exits with plexsphere: resolve platform audit pepper: audit pepper: pepper is missing for an audit chain that holds rows: …	The platform audit chain holds rows but audit.pepper.mount holds no platform pepper: the mount is wrong or empty, the secret was deleted, or the install ran on the static pepper before	Point audit.pepper.mount at the mount that holds the peppers, restore the secret, or carry the static peppers over as in The audit pepper store.
The plexsphere-signer container exits with --key-provider=software is refused when PLEXSPHERE_ENV != dev (got "")	devMode.enabled is off	Set it beside keyProvider: software as in Choose a key provider. The chart refuses that pair at render time, so a container that reaches this message was configured past it.
The plexsphere container exits with plexsphere: PLEXSPHERE_<NAME> must be set when PLEXSPHERE_DSN is set	A required value is empty	Find PLEXSPHERE_<NAME> in the values reference and set the value that drives it.
The plexsphere container exits with plexsphere: PLEXSPHERE_AUTH_CALLBACK_URL must be an absolute http(s) URI (RFC 6749 §3.1.2); got "<value>"	auth.callbackUrl, or externalUrl when the callback URL derives from it, does not start with http:// or https://	Set an absolute URL such as https://plexsphere.example.com/v1/auth/callback in auth.callbackUrl, or give externalUrl its scheme.
The plexsphere container exits with rate limiting requires an explicit client-IP source: …	api.extraEnv overrides the client-IP source the chart sets	Drop the override and choose with rateLimit.clientIpSource.
A provisioned machine's cloud-init log shows the plexd install failing, and curl -s <externalUrl>/downloads/plexd/linux-amd64 answers 502 with plexd_download_upstream_unavailable	The API cannot reach the plexd GitHub release, or the mirror provisioning.plexdDownloadSourceUrl names	Allow the API Pods egress to github.com and release-assets.githubusercontent.com on TCP 443, or set provisioning.plexdDownloadSourceUrl to a copy of the release assets the API can reach. The plexsphere container log carries the upstream error beside the problem's correlation id.
The plexsphere container exits with plexsphere: PLEXSPHERE_PROVISIONING_BROKER_CA_BUNDLE_FILE must hold PEM certificates only: …	The key provisioning.caBundle names holds something other than PEM certificates; the rest of the message names the rule it breaks	Put only the CA certificates into the key and roll the API out, as in Provisioned machines and a private CA.
A provisioned machine's cloud-init log shows certificate signed by unknown authority, or the plexd install failing on an SSL certificate problem	The control plane serves a certificate from a CA the machine does not trust	Set provisioning.caBundle as in Provisioned machines and a private CA. The self-enrolment runbook walks through the diagnosis.
The edge answers 503 or times out on every path, while the Pods are Ready and a port-forward works	networkPolicy.gatewayFrom is empty or does not select the Pods of the Gateway, ingress controller or mesh in front of the release	Read the labels off those Pods with kubectl -n <namespace> get pods --show-labels and set them as in Restrict network access.
helm install times out on the migrate hook, and Pods of the release stay in ErrImagePull or ImagePullBackOff	The nodes hold no credential for the image registry, imagePullSecrets names no Secret, or the Secret is missing, in another namespace, or holds a credential the registry refuses	kubectl -n "$NS" describe pod <pod> names the image and the pull error. Create the Secret as in The registry credential and name it in imagePullSecrets (and spicedb.imagePullSecrets for a mirrored SpiceDB), then run helm install again.
A Pod stays in CreateContainerConfigError	A Secret, or a key in it, is missing	kubectl -n "$NS" describe pod <pod> names it; create it as above.
Pods stay in ContainerCreating with secret "spicedb-tls" not found	cert-manager is not installed, or its Certificate has not gone Ready	Install cert-manager, read kubectl -n "$NS" describe certificate spicedb-tls for the reason it has not issued, or issue the certificate yourself and name it in spicedb.tls.existingSecret as in The SpiceDB certificate.
helm install fails with no matches for kind "Certificate"	The cert-manager CRDs are absent from the cluster	Install cert-manager first, or set spicedb.tls.existingSecret so the chart renders no cert-manager resource.
The plexsphere container logs certificate signed by unknown authority on its SpiceDB dial	The Secret carries no ca.crt, or the CA was rotated and the API still holds the root it read at boot	Add the key, or roll the API onto the new root with kubectl -n "$NS" rollout restart deployment/plexsphere.
The signer restarts right after its first install	The core migrations have not run yet	Install plexsphere first; the signer recovers once its migrate hook completed.
helm install fails with a named message	A value combination the chart refuses	The values reference lists each message and its fix.

See also

• Helm charts: every value, the Secret contract, and the resource names.
• Run database migrations
• Rotate the NSK wrap key
• Rotate the signing key
• Disaster recovery: the backup coverage matrix, the ordered restore checklist, and the restore drill.