plexsphere

Appearance

Lab cluster runbook ​

The lab is a real Kubernetes cluster, a Gardener shoot on Cleura's OpenStack, that Flux reconciles from main of this repository. Everything the lab runs lives in the namespace plexsphere: the Flux Operator and Flux, an OpenBao that holds every secret of the lab, the External Secrets Operator that materialises those secrets, cert-manager with a private lab CA, Envoy Gateway serving one Gateway on the public address the cloud assigns to it, under *.<ip-dashed>.nip.io, the four backing services the control plane needs (Postgres under the CloudNativePG operator, NATS with JetStream, SeaweedFS as the S3 object store, and Dex as the OIDC issuer), Crossplane with its composition Functions as the management fleet's provisioning substrate, and the control plane itself with its signer, installed from the edge channel that every merge to main publishes. The one exception are the namespaces the management fleet creates per Project it places on the cluster; see What lives outside plexsphere.

This page covers the one-time bootstrap with deploy/lab/scripts/bootstrap.sh, the Flux Kustomizations and their order, the backing services, the control plane and its provisioning substrate, how the lab follows main and upstream releases, the Postgres backups, how a machine comes to trust the lab CA, how an operator signs in and issues a bootstrap token, how the first node outside the cluster joins, the end-to-end check with deploy/lab/scripts/verify.sh, emptying the cluster with deploy/lab/scripts/teardown.sh, rebuild and recovery, and troubleshooting.

The desired state is the tree under deploy/lab/. In this page <ip> stands for the public address (a.b.c.d) and <ip-dashed> for its dashed form (a-b-c-d), the label nip.io resolves back to the address.

What the cluster provides ​

The lab runs on a Gardener shoot on Cleura's OpenStack (region Fra1), created and sized in Cleura's Gardener dashboard. What the bootstrap relies on:

  • Kubernetes 1.35 with two amd64 workers of the flavor b.2c8gb (2 vCPUs and 8 GiB each), Cilium, and the default StorageClass default (Cinder CSI, expandable) with the VolumeSnapshotClass csi-cinder marked default. Every volume of the lab uses the default class, and the nightly Postgres snapshot the default snapshot class.
  • LoadBalancer Services through the OpenStack cloud controller: for each one it creates an Octavia load balancer, attaches a floating IP from the external network two to three minutes after the Service exists, and reports the address in the Service's status. A floating IP cannot be reserved for the cluster ahead of the Service, so no manifest names an address: the Gateway's EnvoyProxy asks for a LoadBalancer Service alone, and bootstrap step 9b reads the assigned address from the Gateway into the ConfigMap lab-cluster, which every hostname of the lab derives from. Deleting the Service deletes the load balancer and releases the floating IP; a rebuilt lab gets a new address and new hostnames.
  • No egress firewall: Pods reach any address and port on the internet, the OpenStack's API ports among them. Gardener's own NetworkPolicies apply in kube-system only. Flux reaches GitHub over SSH on ssh.github.com:443 all the same, which works with and without a firewall that closes TCP 22.
  • An admin kubeconfig issued by the Gardener dashboard. It expires; renew it there. Until Flux runs in the cluster it is the only way to change anything.

Prerequisites ​

On the host that runs the bootstrap:

  • kubectl, helm (3.13 or later, for helm get metadata), jq, openssl (OpenSSL 1.1.1 or later, for req -addext and for deriving an Ed25519 public key from a raw seed; the LibreSSL macOS ships does the first and not the second), ssh-keygen, curl, htpasswd and cosign (2.6 or later, for cosign verify --new-bundle-format) on PATH. macOS ships htpasswd; Debian and Ubuntu have it in apache2-utils. The bootstrap needs its -B (bcrypt) and -i (password on stdin) flags. macOS has cosign in Homebrew, and the sigstore release page carries binaries. The first cosign verify on a host fetches the Sigstore trust root from tuf-repo-cdn.sigstore.dev.
  • A cluster-admin kubeconfig kept outside the repository, or at /kubeconfig in the checkout root, which .gitignore ignores and a workspace gate keeps ignored. LAB_KUBE_CONTEXT names its context.
  • The inputs of the two scripts, as environment variables or in /lab.env in the checkout root, which .gitignore ignores and the same gate keeps ignored: one LAB_<NAME>=<value> line each, the value raw to the end of the line (no quotes, no export), blank lines and # comments allowed, mode 0600. A variable set in the environment wins over the file, and LAB_ENV_FILE names another file. The file holds the ghcr.io token and the OAuth App secret, so it belongs in the password manager with the state directory.
  • A GitHub classic personal access token with read:packages, for pulling plexsphere charts and images from ghcr.io: the kubelet and Flux pull with it through the Secret plexsphere-registry. Create it at https://github.com/settings/tokens/new (your Settings, Developer settings, Personal access tokens, Tokens (classic), Generate new token (classic)) with the one scope read:packages and an expiry. It has to be a classic token: GitHub Packages accepts no fine-grained token, and ghcr.io answers one with 403 on every pull. If the organisation enforces SAML single sign-on, authorise the token for plexsphere on its row afterwards (Configure SSO). GitHub shows the token once. It goes to LAB_GHCR_TOKEN, with the GitHub user name it belongs to in LAB_GHCR_USERNAME.
  • A green run of the edge lane on main, so the channel holds a chart the control plane can install: crane ls ghcr.io/plexsphere/charts/plexsphere lists a 0.0.0-edge.… version.
  • A GitHub OAuth App for Dex, created under the plexsphere organisation at https://github.com/organizations/plexsphere/settings/applications/new (the organisation's Settings, Developer settings, OAuth Apps, New OAuth App) with any name, the homepage URL https://dex.<ip-dashed>.nip.io and the authorization callback URL https://dex.<ip-dashed>.nip.io/callback; the device flow stays off. The app's page then shows its client id, and Generate a new client secret shows a secret once. They go to LAB_GITHUB_CLIENT_ID and LAB_GITHUB_CLIENT_SECRET. Members of the organisation sign in to the lab through the app; each grants it access to the organisation on the first sign-in.
  • The right to add a read-only deploy key to the repository.
  • A password manager for the state directory the bootstrap fills.

The cluster must not carry another stack's controllers: the bootstrap installs Flux, cert-manager, External Secrets, Envoy Gateway, CloudNativePG and Crossplane itself and refuses to start while any of their CRD groups already exists on a cluster it has not bootstrapped.

The bootstrap ​

Write the inputs to lab.env in the checkout root and run the bootstrap from there:

text
LAB_KUBE_CONTEXT=<context>
LAB_GHCR_USERNAME=<github-user>
LAB_GHCR_TOKEN=<classic-token-with-read-packages>
LAB_GITHUB_CLIENT_ID=<oauth-app-client-id>
LAB_GITHUB_CLIENT_SECRET=<oauth-app-client-secret>
bash
deploy/lab/scripts/bootstrap.sh

Every input is also an environment variable, which wins over the file:

VariableRequiredMeaning
LAB_KUBE_CONTEXTyesThe kubeconfig context of the cluster; every kubectl and helm call passes it.
LAB_GHCR_USERNAME, LAB_GHCR_TOKENon the first runThe ghcr.io credential seeded into OpenBao. A later run with both set rewrites it.
LAB_GITHUB_CLIENT_ID, LAB_GITHUB_CLIENT_SECRETon the first runThe GitHub OAuth App Dex signs organisation members in with, seeded into OpenBao. A later run with both set rewrites it; one without either keeps it. Setting only one of the two stops the run. The first run on a lab bootstrapped before Dex existed needs both too, or step 9 stops with OpenBao holds no lab/dex.
LAB_VOLUME_SNAPSHOT_CLASSnoOverrides the VolumeSnapshotClass read from the cluster for the nightly Postgres snapshot.
LAB_STATE_DIRnoDefault $HOME/.plexsphere-lab/$LAB_KUBE_CONTEXT, mode 0700; every generated secret is a 0600 file in it.
LAB_OPENBAO_ROOT_TOKENnoThe OpenBao root token, for a run without the openbao-init.json that holds it.
LAB_ADDRESS_TIMEOUT_SECONDSnoDefault 900; how long step 9b waits for the cloud to assign the Gateway's address. A value that is not a whole number of seconds above zero stops the run.
LAB_SEED_TIMEOUT_SECONDSnoDefault 300; how long step 10b waits for the seed Job to complete. A value that is not a whole number of seconds above zero stops the run.
LAB_ENV_FILEnoDefault lab.env in the checkout root; the file both scripts read LAB_<NAME>=<value> lines from before anything else. A line that is not such an assignment stops the run.
LAB_OPENSTACK_RC_FILEnoAn OpenStack rc file, absolute or relative to the checkout root; see The OpenStack Cloud. Step 10c creates an application credential in the rc file's project with its user's password and registers the Cloud with it. It replaces the four LAB_OPENSTACK_* inputs below other than LAB_OPENSTACK_REGION, which overrides the rc file's region; one of the four beside it stops the run.
LAB_OPENSTACK_AUTH_URL, LAB_OPENSTACK_REGION, LAB_OPENSTACK_PROJECT_ID, LAB_OPENSTACK_APPLICATION_CREDENTIAL_ID, LAB_OPENSTACK_APPLICATION_CREDENTIAL_SECRETno, all five or none without an rc fileThe lab's OpenStack: the Keystone endpoint (https:// only), the region, the id of the OpenStack project the machines land in (the Cloud's external id), and an application credential scoped to that project. A partial set stops the run with set all five LAB_OPENSTACK_* inputs or none.
LAB_OPENSTACK_APPLICATION_CREDENTIAL_NAMEnoDefault plexsphere-lab; the name of the application credential step 10c creates from the rc file.
LAB_OPENSTACK_CLOUD_SLUG, LAB_OPENSTACK_CLOUD_NAMEnoDefault lab-openstack and Lab OpenStack; the slug and the display name of the Cloud step 10c registers. The slug also names the Cloud's credential and is what verify.sh check 23 looks the Cloud up by. A slug the API would refuse, other than at most 63 characters of lowercase letters, digits and inner hyphens, stops the run.
LAB_OPENSTACK_EXTERNAL_NETWORK_IDnoThe id of the external network the landing zone's router gateways on; see The OpenStack landing zone. Without it step 10e reads the one external network the OpenStack project sees.

The Git URL and branch come from deploy/lab/flux/fluxinstance.yaml alone. Run the bootstrap with </dev/null to skip the pause after the deploy key is printed.

Every passed step prints one line on stdout; every message and error goes to stderr prefixed bootstrap:. The first failing step stops the run with exit code 1. The steps, in order:

  1. Prerequisites: the host tools, the cluster reachable through LAB_KUBE_CONTEXT (the kubectl error is quoted when it is not), whole-number timeouts in LAB_ADDRESS_TIMEOUT_SECONDS and LAB_SEED_TIMEOUT_SECONDS, that htpasswd writes a bcrypt hash from stdin, that openssl derives an Ed25519 public key from a raw seed, on the first run both ghcr.io variables and both GitHub OAuth App variables, on every run both or neither of the GitHub pair, an rc file that names an https:// Keystone, a literal password, a user and a project, each with a domain or as an id, and a region, or all five LAB_OPENSTACK_* inputs or none, an https:// LAB_OPENSTACK_AUTH_URL, and a valid LAB_OPENSTACK_CLOUD_SLUG. Nothing is created on the cluster before this step passes.

  2. Refusal while another stack holds the cluster: if a CRD of fluxcd.controlplane.io, source.toolkit.fluxcd.io, cert-manager.io, gateway.envoyproxy.io, gateway.networking.k8s.io, external-secrets.io, postgresql.cnpg.io, pkg.crossplane.io or apiextensions.crossplane.io exists and the namespace plexsphere does not carry plexsphere.io/lab=bootstrapped, the run stops with bootstrap: the cluster already carries <group>; remove the stack that installed it before bootstrapping the lab.

  3. Namespace: deploy/lab/namespace.yaml is applied and the namespace labelled plexsphere.io/lab=bootstrapped, which marks the cluster as the lab's from then on. The marker has a key of the lab's own: the Flux Operator keeps app.kubernetes.io/part-of=flux on the namespace of its FluxInstance against any patch.

  4. Snapshot class: without LAB_VOLUME_SNAPSHOT_CLASS the snapshot class is the cluster's only VolumeSnapshotClass, or the one annotated snapshot.storage.kubernetes.io/is-default-class: "true" among several; anything else stops the run with bootstrap: expected exactly one VolumeSnapshotClass or one marked default, found: <names>; set LAB_VOLUME_SNAPSHOT_CLASS. The ConfigMap lab-cluster receives LAB_VOLUME_SNAPSHOT_CLASS, and the step prints the class with where it came from; a re-run with a changed class updates it. The address is step 9b's: a re-run keeps the one the ConfigMap already holds until then, so the Kustomizations that substitute it keep reconciling meanwhile, and a first run writes the ConfigMap without it.

  5. Git credential: unless the Secret lab-git exists, an ed25519 key pair is generated (or reused from the state directory), GitHub's published SSH host keys are read from https://api.github.com/meta over TLS as the known_hosts entries for ssh.github.com on port 443, and the Secret is created with identity, identity.pub and known_hosts. The public key is printed; register it as a read-only deploy key on the repository and press Enter. A read of lab-git that fails for any reason other than the Secret being absent stops the run, so a failed read never mints a new deploy key.

  6. Unseal key: unless the Secret openbao-unseal exists, it is created from openbao-unseal.key in the state directory, and that key is generated (32 bytes) only while OpenBao has no volume. A missing key beside an existing volume data-openbao-0 stops the run: a new key never unseals the data sealed under the old one. A read of either object that fails for any reason other than the object being absent (an expired token, a missing permission, a timeout) stops the run too, so a failed read never mints a new key.

  7. Lab CA: unless the state directory holds lab-ca.crt, an ECDSA P-256 CA (CN=plexsphere lab CA, ten years) is generated. A state directory with lab-ca.key but no lab-ca.crt, or a cluster that already publishes a CA in lab-ca-public, stops the run instead; restore the files from the password manager. A failed read of lab-ca-public stops the run the same way. The ConfigMap lab-ca-public receives the certificate under response.body, and its SHA-256 fingerprint is printed.

    7b. Backing-service and control-plane credentials: whatever of s3-access-key (16 random bytes, hex), s3-secret-key (32 bytes, hex), dex-operator.password and dex-member.password (24 bytes, base64), spicedb-preshared-key (32 bytes, hex), signer-seed (32 raw bytes), api-keys.json (the thirteen cursor HMAC keys of 32 bytes, hex, and NSK_WRAP_KEY_B64 of 32 bytes, base64) and the signer's mutual-TLS chain the state directory lacks is generated, mode 0600, and one line per file says generated or exists. The chain is six files, generated together when signer-ca.crt is absent with the commands of the install how-to: signer-ca.crt and signer-ca.key, a server leaf signer-server.crt and signer-server.key naming plexsphere-signer, plexsphere-signer.plexsphere.svc and plexsphere-signer.plexsphere.svc.cluster.local, and a client leaf signer-client.crt and signer-client.key carrying the SPIFFE ID spiffe://plexsphere.lab/core-binary, all valid for ten years. The public key of signer-seed is derived on every run. A file that exists but holds no value stops the run, before any file is generated, with bootstrap: <state>/<file> is empty; restore it from the password manager, or delete it to generate a new one; a signer-seed of another size than 32 bytes stops it the same way. So does signer-ca.crt beside a missing leaf, with bootstrap: signer-ca.crt exists but <file> is missing; restore signer-server.* and signer-client.* from the password manager, or to rotate the chain instead, delete signer-ca.crt signer-ca.key signer-server.crt signer-server.key signer-client.crt signer-client.key (not signer-seed) and lab/signer-mtls (bao kv metadata delete -mount=lab signer-mtls as root in openbao-0), and a file of the chain beside a missing signer-ca.crt, which a new chain would overwrite, with bootstrap: signer-ca.crt is missing but <file> exists; restore signer-ca.crt from the password manager, or to rotate the chain instead, …. Nothing touches the cluster here.

  8. Flux: the Flux Operator is installed with Helm into plexsphere within the version range of its OCIRepository, the FluxInstance applied, and the run waits for the FluxInstance and the GitRepository lab. An unregistered deploy key stops the run here, with the GitRepository's condition message printed. Once Flux is Ready the step reads the namespace's labels back: the Flux Operator applies a Namespace object of its own for the namespace of its FluxInstance and adopts the labels kubectl set, so the lab's labels survive only through the Namespace patch in deploy/lab/flux/fluxinstance.yaml. A namespace without pod-security.kubernetes.io/enforce=restricted or the marker stops the run with bootstrap: namespace plexsphere lost its labels under Flux: ….

  9. OpenBao: once lab-openbao and the Pod openbao-0 are Ready, OpenBao is initialised if it is not yet, and the recovery keys and root token go to openbao-init.json in the state directory, never into the cluster; an init that does not finish leaves its output in openbao-init.json.partial. The bootstrap reads OpenBao's state from the JSON bao status prints on stdout alone, since kubectl reports bao's exit code 2 for a sealed or uninitialised server on stderr. A status it cannot read stops the run before bao operator init with bootstrap: bao status in openbao-0 printed no status JSON: … or bootstrap: bao status in openbao-0 failed with exit code …: …, the second with kubectl's error. The static seal unseals it. With the root token the bootstrap enables the KV v2 mount lab and the kubernetes auth method, writes the policy and role lab-external-secrets for the ServiceAccount external-secrets, seeds lab/ca from lab-ca.crt and lab-ca.key in the state directory unless it exists, and writes lab/registry from the ghcr.io variables. It then writes lab/s3 from the two key files on every run, without a trailing newline a restored file may carry; a key that is not the hex string step 7b generates stops the run with bootstrap: the S3 key files in <state> are malformed; restore them from the password manager. It writes lab/dex with the bcrypt hashes of the two password files, computed with htpasswd on the spot, and the GitHub pair; a run without the pair patches only the hashes and keeps the pair. bcrypt salts every hash, so lab/dex changes on each run while Dex keeps verifying the same passwords. It writes the control plane's four keys on every run as well: lab/api-keys (the fourteen values of api-keys.json and SIGNING_PUBLIC_KEY_B64, the public key of signer-seed), lab/spicedb (preshared_key), lab/signer (seed_b64, the seed in base64) and lab/signer-mtls (ca.crt, server.crt, server.key, client.crt and client.key); an api-keys.json that lacks one of its keys stops the run with bootstrap: <state>/api-keys.json is malformed; restore it from the password manager. Every value reaches OpenBao on stdin. A file that step 7b had to generate while OpenBao already holds the key it is seeded into (lab/s3, lab/dex, lab/spicedb, lab/signer, lab/api-keys or lab/signer-mtls) means the original was lost: the run removes the files it generated and stops before any OpenBao write with bootstrap: OpenBao holds lab/s3 but <state>/s3-access-key is missing; restore it from the password manager. Last, it writes the audit pepper store the API reads: the KV v2 mount plexsphere unless it exists, the policy plexsphere-audit-pepper, which creates and reads plexsphere/data/audit/pepper/* and withholds update so the API cannot overwrite a Domain's pepper, and the Kubernetes auth role plexsphere-audit-pepper for the ServiceAccount plexsphere in plexsphere (TTL 20 minutes, at most an hour). The role requires the audience vault, like lab-external-secrets: the API presents a token projected for that audience, never its ServiceAccount's own token, which the Kubernetes API accepts with the chart's ClusterRole. Then the cloud credential store: the KV v2 mount cloud-credentials unless it exists; the policy plexsphere-cloud-credentials, which creates and updates cloud-credentials/data/clouds/* and reads nothing, since the API only writes credentials; its Kubernetes auth role for the ServiceAccount plexsphere (TTL 20 minutes, at most an hour); the policy lab-cloud-credentials, which only reads the same paths; and its role for the ServiceAccount external-secrets (TTL an hour), which the ClusterSecretStore plexsphere-cloud-credentials logs in with. Both roles require the audience vault. The policy keeps the API from reading credentials out of OpenBao only: the API's ClusterRole reads Secrets in every namespace, so it can read each credential the broker materialises as a Secret in a Project namespace.

    9b. Public address: the run asks Flux to reconcile lab-gateway and waits up to LAB_ADDRESS_TIMEOUT_SECONDS (900) for the Gateway lab to report an IPv4 address of type IPAddress in its status: the OpenStack cloud controller assigns it to the Gateway's Service through an Octavia load balancer and a floating IP, two to three minutes after the Service exists, and Envoy Gateway copies it into the Gateway. The ConfigMap lab-cluster then receives LAB_PUBLIC_IP and LAB_PUBLIC_IP_DASHED beside the class, and the step prints configmap lab-cluster: LAB_PUBLIC_IP=<ip> from the Gateway lab; lab-issuer, lab-dex and lab-control-plane, which substitute the two, proceed from there. A Gateway without such an address when the time is up stops the run with bootstrap: gateway lab reports no IPv4 address after <n>s; kubectl --namespace plexsphere describe services -l gateway.envoyproxy.io/owning-gateway-name=lab shows the cloud controller's events. The address is never an input: a floating IP cannot be reserved ahead of the Service, and the teardown releases it, so a rebuilt lab gets a new address and new hostnames, and the summary prints the OAuth App callback to register anew. A re-run reads the live address again and rewrites the ConfigMap when it changed.

  10. Convergence: the run waits up to 15 minutes each for lab-external-secrets, lab-cert-manager, lab-envoy-gateway, lab-crossplane, lab-crossplane-packages, lab-secrets, lab-issuer, lab-gateway, lab-cloudnative-pg, lab-nats, lab-postgres, lab-seaweedfs, lab-dex and lab-control-plane, and after lab-postgres up to 10 minutes for the Cluster plexsphere-db to report Ready. Before each wait it requests a reconciliation of that Kustomization (reconcile.fluxcd.io/requestedAt): Flux marks a Kustomization stalled when a health check fails early, as lab-secrets does before OpenBao holds anything and lab-postgres while the Cluster still starts, and looks again after the Kustomization's one-minute retryInterval. A stall after the request clears on that retry: on a first install lab-issuer stalls while cert-manager races its own first request for lab-edge-tls, and lab-gateway while the Gateway waits for its address. On a timeout it prints all Kustomizations and the failing one's conditions.

    10b. Seed the first Domain: unless the Job plexsphere-seed exists, the run waits up to 15 minutes for the HelmRelease plexsphere to be Ready and reads the appVersion, status and revision of the Helm release plexsphere with helm get metadata. A release it cannot read stops the run with bootstrap: cannot read the Helm release plexsphere; is lab-control-plane Ready?, and one whose upgrade and migrate hook still run with bootstrap: the Helm release plexsphere is pending-upgrade, not deployed; re-run once its upgrade settles. It then applies a Job that runs ghcr.io/plexsphere/plexsphere-bootstrap:<appVersion>, the seed image of the control plane's own commit, against the manifest in the ConfigMap plexsphere-seed-manifest, and waits up to LAB_SEED_TIMEOUT_SECONDS, five minutes unless set, for it to complete, reading its state every few seconds. The timeout counts time: a wait the API server refuses pauses for a second, and a read it refuses prints kubectl's error and keeps waiting. The Job is kept: a re-run that finds it succeeded prints seed job plexsphere-seed: exists, succeeded and applies nothing. A Job that failed has its log printed and stops the run at once with bootstrap: the seed Job failed; read its log, fix the cause, delete job/plexsphere-seed and re-run, and one that did not complete in time the same way with bootstrap: the seed Job did not complete within <seconds>s; …. A Job deleted while the run waits stops it with bootstrap: job/plexsphere-seed disappeared while the bootstrap waited for it; re-run to apply it again. A release that moved to another revision while the seed ran stops the run with bootstrap: the Helm release plexsphere moved from revision <n> to <m> while the seed ran; run 'kubectl -n plexsphere rollout restart deployment/plexsphere' so the API applies its ReBAC schema again: the API of the upgrade applied the newer ReBAC schema at boot, which the seed's older image may have replaced.

    The sign-in: steps 10c to 10e call the API at https://plexsphere.<ip-dashed>.nip.io as operator@example.com, signed in by deploy/lab/scripts/lab-session.sh the way the console signs in, with the password in the state directory's dex-operator.password; see The lab API session. It prints lab API: operator@example.com signed in at platform scope and in the Domain lab (<domain id>), and a sign-in that fails stops the run with cannot sign in as operator@example.com at platform scope: <why>, cannot find the Domain lab: <why> or cannot sign in as operator@example.com in the Domain lab: <why>.

    10c. Register the OpenStack Cloud: without LAB_OPENSTACK_RC_FILE or the five LAB_OPENSTACK_* inputs the step logs skipping the OpenStack Cloud: set LAB_OPENSTACK_RC_FILE or the five LAB_OPENSTACK_* inputs and re-run (docs/contributing/lab-cluster.md, The OpenStack Cloud) and the run continues. With an rc file it first creates, or takes from the state directory, the application credential and proves it against Keystone, and prints openstack application credential <name> (<id>) of <user> in <project> (<project id>): <created|kept>, authenticates. Then it looks each object up with the platform session before it creates it: the Cloud LAB_OPENSTACK_CLOUD_SLUG names (lab-openstack unless set), the active credential of the same name on it, and the assignment of both to the Project openstack. See The OpenStack Cloud for what it sends and how it fails. It ends with openstack cloud <slug> (<cloud id>): <created|exists>, credential <slug> (<credential id>): <issued|exists>, assigned to project openstack.

    10d. Install and import the OpenStack blueprints: it pulls the official catalog's bundle ghcr.io/plexsphere/blueprints:v1, verifies its signature with cosign and ends that with official catalog ghcr.io/plexsphere/blueprints:v1 @ sha256:<digest>: signed by the official release identity. It then installs openstack-instance and openstack-landing-zone from the bundle, each when the bundle offers the version the lab pins, and ends each with openstack blueprint <slug> <version>: XRD <xrd> Established, Composition <slug> applied, from the official catalog @ sha256:<digest>; a slug the bundle offers at another version, or not at all, logs skipping the OpenStack blueprint: … and is skipped on its own. With the domain session it registers the official catalog as a source of the Domain lab unless it exists and imports the installed blueprints from it. See The OpenStack blueprints for what it fetches and sends, when it skips and how it fails. Each imported blueprint ends with its own line: openstack blueprint openstack-instance v1alpha2 (version <version id>): <imported|unchanged> into the Domain lab, from the official catalog (<catalog id>): <registered|exists> and openstack blueprint openstack-landing-zone v1alpha1 (version <version id>): <imported|unchanged> into the Domain lab, from the official catalog (<catalog id>): <registered|exists>.

    10e. Provision the OpenStack landing zone: without the Cloud of step 10c the step logs skipping the OpenStack landing zone: no OpenStack Cloud is registered (step 10c skipped), without the landing-zone blueprint of step 10d skipping the OpenStack landing zone: openstack-landing-zone is not imported (step 10d skipped it), and the run continues. Otherwise it reuses the landing zone of the Project openstack and prints openstack landing zone <resource id>: exists, network <name>, phase <phase>, with the network its composite reports (or not reported by the composite pr-<id> yet). The name is a Domain member's networkName parameter, so one with quotes, blanks or control characters is not printed: the line says named by the composite pr-<id> outside the pattern ^[A-Za-z0-9._-]+$ instead. Without one it finds the external network through Neutron, or checks the one LAB_OPENSTACK_EXTERNAL_NETWORK_ID names, provisions a landing zone with the domain session, and prints openstack landing zone <resource id>: created, network plexsphere-project-019a0816-0000-7000-8000-000000000001 on the external network <id> (<name>), phase <phase>. It does not wait for Ready; verify.sh reports the phase. See The OpenStack landing zone for the lookup and the Resource. The step stops with Keystone's catalog lists no public network endpoint in the region <region>; provider-openstack cannot manage networks there, GET <url> answered <status> for a list Neutron refuses, the OpenStack project sees no external network; set LAB_OPENSTACK_EXTERNAL_NETWORK_ID, the OpenStack project sees <n> external networks (<names>); set LAB_OPENSTACK_EXTERNAL_NETWORK_ID to one of them, LAB_OPENSTACK_EXTERNAL_NETWORK_ID names <id>, which the OpenStack project sees as no external network (GET <url> answered <status>) or the landing zone <id> of the Project openstack is Failed; delete it (plexctl resource delete <id>) and re-run.

  11. Summary: the address, the hostnames ca.<ip-dashed>.nip.io, dex.<ip-dashed>.nip.io, plexsphere.<ip-dashed>.nip.io and *.<ip-dashed>.nip.io, the GitHub OAuth App callback https://dex.<ip-dashed>.nip.io/callback to register on the app, the two Dex static users with the paths of their password files, the control plane URL https://plexsphere.<ip-dashed>.nip.io with the note that /v1/version there names the running edge version, that operator@example.com signs in to the console on the Domain lab and as Platform Operator, the CA fingerprint, the access file and the state directory. When steps 10c and 10d both passed, two provision: lines follow: the plexctl login into the Domain lab and the plexctl resource create with the Project, the version id and the credential id filled in, as Provision an OpenStack machine uses them. Its networkName is the network step 10e printed: plexsphere-project-019a0816-0000-7000-8000-000000000001 for a landing zone it provisioned, the composite's for one it reused. It stays <network> when step 10e skipped, the composite reports no network yet, or it reports a name outside that pattern.

    The step also writes the access file access.md into the state directory, printed as access file: <state>/access.md: the URLs of the console, of the console's sign-in with the Domain lab's id filled in, of the API and its /v1/version, of Dex, of the OAuth App callback and of the CA download; the two static users with their passwords in clear and their roles; the ids of the Domain lab and of the Projects lab-nodes and openstack; and the plexctl login into the Domain. It is derived from the Gateway's address, the state directory and the seed, so every run rewrites it whole, a 0600 file like the password files it quotes. See The access file.

Store every file of the state directory in a password manager:

FileWhat it is
lab-git, lab-git.pubThe deploy key Flux pulls the repository with.
openbao-unseal.keyThe static unseal key of OpenBao.
lab-ca.key, lab-ca.crtThe lab CA.
openbao-init.jsonOpenBao's recovery keys and root token.
s3-access-key, s3-secret-keyThe S3 key pair of SeaweedFS and the control plane.
dex-operator.password, dex-member.passwordThe passwords of Dex's static users operator@example.com and member@example.com.
spicedb-preshared-keyThe gRPC preshared key of the control plane's SpiceDB.
signer-seedThe signer's 32-byte Ed25519 seed, the Domain's signing key.
api-keys.jsonThe API's thirteen cursor HMAC keys and its NSK wrap key.
signer-ca.crt, signer-ca.key, signer-server.crt, signer-server.key, signer-client.crt, signer-client.keyThe signer's mutual-TLS chain: its CA, its server leaf and the API's client leaf.

access.md beside them needs no copy: it is derived from the other files and the Gateway's address, every bootstrap rewrites it and the teardown deletes it. It holds the two passwords in clear, so it stays in the state directory all the same.

Then delete the local copies of lab-git, lab-ca.key and signer-ca.key. A re-run needs neither lab-git nor lab-ca.key while the cluster keeps its Secrets and OpenBao its volume, and never needs signer-ca.key, which could mint a client certificate the signer admits as the API. Nothing the bootstrap generates is written into the checkout or committed.

The Flux Kustomizations ​

The FluxInstance flux creates the GitRepository lab and the Kustomization lab, which applies deploy/lab/ itself: the namespace, Flux's own definition, the nine HelmRepositories and the fifteen child Kustomizations. A HelmRelease that installs CRDs never shares a Kustomization with the custom resources that need them, so every first apply is clean.

The namespace itself is the Flux Operator's. It applies a Namespace object of its own for the namespace of its FluxInstance, labelled app.kubernetes.io/part-of=flux and annotated kustomize.toolkit.fluxcd.io/ssa: Ignore, which keeps the Kustomization lab from applying namespace.yaml over it, and it adopts whatever labels kubectl set before. The lab's labels, the three Pod Security labels at restricted and the marker plexsphere.io/lab=bootstrapped, reach the namespace through a kustomize patch on that Namespace object in deploy/lab/flux/fluxinstance.yaml. namespace.yaml carries the same labels for the bootstrap's first apply, and a workspace gate holds the two equal. The first real cluster showed what happens without the patch: the operator's apply dropped enforce and audit, so the namespace enforced nothing, and replaced the marker, so every re-run of the bootstrap refused the lab's own cluster.

KustomizationPathDepends onApplies
labdeploy/labNamespace, FluxInstance, the HelmRelease flux-operator, HelmRepositories, the fifteen children
lab-openbaodeploy/lab/openbaoHelmRelease openbao
lab-external-secretsdeploy/lab/external-secretsHelmRelease external-secrets
lab-cert-managerdeploy/lab/cert-managerHelmRelease cert-manager
lab-envoy-gatewaydeploy/lab/envoy-gatewayHelmRelease envoy-gateway
lab-crossplanedeploy/lab/crossplaneHelmRelease crossplane
lab-crossplane-packagesdeploy/lab/crossplane-packageslab-crossplaneDeploymentRuntimeConfig default, Functions function-patch-and-transform and function-auto-ready, all cluster-scoped
lab-secretsdeploy/lab/secretslab-openbao, lab-external-secretsSecretStore lab, ExternalSecrets lab-ca, plexsphere-registry, seaweedfs-s3-config, plexsphere-object-store, dex-credentials, plexsphere-cursor-keys, spicedb-credentials, plexsphere-signer-seed, plexsphere-signer-tls, plexsphere-signer-client-ca and plexsphere-access-signer-client
lab-issuerdeploy/lab/issuerlab-cert-manager, lab-secretsIssuer lab-ca, Certificate lab-edge-tls
lab-gatewaydeploy/lab/gatewaylab-envoy-gatewayGatewayClass, EnvoyProxy and Gateway lab, the redirect and CA routes
lab-cloudnative-pgdeploy/lab/cloudnative-pgHelmRelease cloudnative-pg
lab-natsdeploy/lab/natsHelmRelease nats, NetworkPolicy nats
lab-postgresdeploy/lab/postgreslab-cloudnative-pgCluster plexsphere-db, ScheduledBackup plexsphere-db-nightly, the CronJob plexsphere-db-backup-prune with its RBAC, NetworkPolicy plexsphere-db
lab-seaweedfsdeploy/lab/seaweedfslab-secretsHelmRelease seaweedfs, NetworkPolicy seaweedfs
lab-dexdeploy/lab/dexlab-secrets, lab-gatewayHelmRelease dex, HTTPRoute dex, NetworkPolicy dex
lab-control-planedeploy/lab/control-planelab-secrets, lab-gateway, lab-postgres, lab-nats, lab-seaweedfs, lab-dexOCIRepositories plexsphere-edge and plexsphere-signer-edge, HelmReleases plexsphere and plexsphere-signer, BackendTrafficPolicy plexsphere-api, ConfigMap plexsphere-seed-manifest

lab-issuer, lab-dex and lab-control-plane substitute ${LAB_PUBLIC_IP} and ${LAB_PUBLIC_IP_DASHED} from the ConfigMap lab-cluster, which carries them only once bootstrap step 9b has read the address from the Gateway; until then each reports Ready False with the kustomize-controller's message naming the variable. lab-postgres substitutes ${LAB_VOLUME_SNAPSHOT_CLASS} from it, which step 4 writes first. lab-gateway substitutes nothing and depends on lab-envoy-gateway alone: the Gateway must exist before the cloud assigns the address, so its listeners name no hostname (the routes carry them) and its https listener names the Secret lab-edge-tls before lab-issuer writes it; Envoy Gateway programs the http listener meanwhile and takes the Secret up when it appears. Until the bootstrap has seeded OpenBao, the SecretStore or an ExternalSecret reports Ready False with the reason SecretSyncedError, lab-secrets stays not Ready and holds lab-issuer, lab-seaweedfs, lab-dex and lab-control-plane back; that is the intended gate. lab-seaweedfs, lab-dex and lab-control-plane wait for it because their HelmReleases mount Secrets that External Secrets writes, and lab-postgres waits for lab-cloudnative-pg because the operator's HelmRelease installs the CRDs of the Cluster. lab-control-plane waits for every backing service too: its routes attach to the Gateway, its migrate hook dials Postgres, and the API dials NATS, the object store and Dex at boot. Its health check reads the Ready condition of both HelmReleases, and its timeout is 15 minutes, because the first install waits on the migrate hook and four image pulls from ghcr.io, then on the signer.

Every child reconciles every ten minutes (interval: 10m) and retries a failed apply or health check after one minute (retryInterval: 1m). Flux stops a health check early on an object that reports a failure, even one that heals seconds later: on a first install cert-manager reports the Certificate lab-edge-tls Failed while it races its own first CertificateRequest, and the Gateway lab reports Failed until the cloud has assigned its address. Without the retry interval Flux would look again only after ten minutes, and every Kustomization behind the stalled one would wait as long. A workspace gate keeps retryInterval on every child, at most two minutes and shorter than interval.

The backing services ​

The control plane's charts point at a Postgres, a NATS, an S3 object store and an OIDC issuer and install none of them. The lab runs each as one instance in plexsphere, on a volume of the default StorageClass, with its credentials from OpenBao through External Secrets. Each has one ingress NetworkPolicy that admits the Pods labelled app.kubernetes.io/part-of: plexsphere, which all three control plane charts stamp, and nothing else in the namespace beyond what the service itself needs. A workload without that label times out on all four ports. The policies select only the services' Pods, so everything else in plexsphere keeps its traffic.

Postgres ​

The HelmRelease cloudnative-pg runs the CloudNativePG operator. The Cluster plexsphere-db is one Postgres 16 instance, the Pod plexsphere-db-1, from the rolling image ghcr.io/cloudnative-pg/postgresql:16-minimal-trixie, which every Pod restart pulls again, on the 20Gi PVC plexsphere-db-1. Its one database plexsphere, owned by the role plexsphere, holds the core schema and the SpiceDB datastore, with max_connections at 200. The operator serves it on 5432 through the Services plexsphere-db-rw, plexsphere-db-ro and plexsphere-db-r.

The operator generates the owner's password and writes the Secret plexsphere-db-app (type kubernetes.io/basic-auth), whose key uri is postgresql://plexsphere:<password>@plexsphere-db-rw:5432/plexsphere. OpenBao is not involved. There is no superuser Secret; admin work runs as postgres over the local socket:

bash
kubectl exec -n plexsphere -it plexsphere-db-1 -c postgres -- psql -U postgres

The NetworkPolicy plexsphere-db admits the control plane on 5432 and the operator on 5432 and 8000, where it reads each instance's status. The backups are in Backups.

NATS ​

The HelmRelease nats runs one NATS server, the Pod nats-0, with JetStream on the 10Gi PVC nats-js-nats-0. The Service nats serves clients on 4222 and the monitor on 8222. The server has no credentials: the API's NATS client takes only a credentials file, and the NetworkPolicy nats, which admits the control plane on 4222 and 8222, is the admission control. The API creates its streams itself.

SeaweedFS ​

The HelmRelease seaweedfs runs SeaweedFS in all-in-one mode: the Deployment seaweedfs-all-in-one runs master, volume, filer and S3 in one process on the 20Gi PVC seaweedfs-all-in-one-data. The Service seaweedfs-all-in-one serves S3 on 8333, plaintext inside the namespace.

The bootstrap seeds one S3 key pair into OpenBao under lab/s3. The ExternalSecret seaweedfs-s3-config renders it into the Secret of the same name, whose one key seaweedfs_s3_config is the identities file SeaweedFS loads: one identity plexsphere with the actions Admin, Read, Write, List and Tagging. With an identity in place SeaweedFS refuses every unsigned request. The ExternalSecret plexsphere-object-store writes the same pair as S3_ACCESS_KEY and S3_SECRET_KEY, the shape the control plane chart reads. The chart's hook Job seaweedfs-bucket-hook creates the bucket plexsphere-action-output on every install and upgrade when it is missing; the API never creates it.

The NetworkPolicy seaweedfs admits the control plane on 8333, and SeaweedFS's own Pods, the hook Job among them, on every port.

Dex ​

The HelmRelease dex runs Dex with its memory store, served through the HTTPRoute dex on the Gateway's https listener at https://dex.<ip-dashed>.nip.io. A Dex restart drops logins in flight and its signing keys; the API keeps its own sessions in Postgres, so a signed-in operator stays signed in.

Users sign in two ways:

  • The static users operator@example.com and member@example.com, with the user ids of the dev stack, so the subject the dev bootstrap manifest carries for the first Platform Operator is the same here. Their passwords are dex-operator.password and dex-member.password in the state directory; OpenBao holds only their bcrypt hashes under lab/dex.
  • GitHub, for members of the plexsphere organisation, through the OAuth App the bootstrap seeds under lab/dex as well.

The ExternalSecret dex-credentials writes the two hashes and the OAuth App's client id and secret into the Secret dex-credentials, which Dex reads from its environment. The one client plexsphere is public and has no secret, because the API's code exchange is PKCE-only. The NetworkPolicy dex admits the Gateway's Envoy proxy and the control plane on 5556. In-cluster callers use the public hostname too, which needs the cluster to route its own LoadBalancer address back to the proxy; verify.sh check 15 proves it, and Troubleshooting names the fallback.

What the control plane leg configures ​

The names the control plane's charts are pointed at. <ip-dashed> is LAB_PUBLIC_IP_DASHED of the ConfigMap lab-cluster.

WhatValue
Control-plane hostname (API and console)plexsphere.<ip-dashed>.nip.io
OIDC issuerhttps://dex.<ip-dashed>.nip.io (discovery at /.well-known/openid-configuration)
OIDC clientid plexsphere, public, no secret, redirect URI https://plexsphere.<ip-dashed>.nip.io/v1/auth/callback
Static usersoperator@example.com (userID 0c10d000-0000-4000-8000-0000000000c0, Dex sub CiQwYzEwZDAwMC0wMDAwLTQwMDAtODAwMC0wMDAwMDAwMDAwYzASBWxvY2Fs) and member@example.com (userID 0e110e00-0000-4000-8000-00000000000e)
Postgres DSNSecret plexsphere-db-app, key uri (postgresql://plexsphere:<password>@plexsphere-db-rw:5432/plexsphere), for database.existingSecret/database.key of both charts and datastore.existingSecret/datastore.key of the spicedb chart
NATSnats://nats.plexsphere.svc:4222, no credentials
Object storeendpoint http://seaweedfs-all-in-one.plexsphere.svc:8333, region us-east-1, path style, insecure endpoint allowed, Secret plexsphere-object-store with S3_ACCESS_KEY and S3_SECRET_KEY, bucket plexsphere-action-output
Audit pepper storehttp://openbao.plexsphere.svc:8200, KV v2 mount plexsphere, Kubernetes auth role plexsphere-audit-pepper
Peers the NetworkPolicies admitevery Pod labelled app.kubernetes.io/part-of: plexsphere

The control plane ​

The Kustomization lab-control-plane installs the control plane on top of the backing services:

  • The HelmRelease plexsphere runs the API (Deployment plexsphere), the console (Deployment plexsphere-console) and the bundled SpiceDB (Deployment spicedb, datastore in plexsphere-db, serving TLS from a cert-manager certificate in the Secret spicedb-tls). Its migrate hook, the Job plexsphere-migrate, runs before the API on every install and upgrade. On the first install SpiceDB starts beside the API: the API answers /livez, stays not ready and waits for SpiceDB to answer, without restarting, then applies the authorization schema and serves.
  • The HelmRelease plexsphere-signer runs the signer (Deployment plexsphere-signer), which signs with the Domain's key and answers the API alone, over mutual TLS on 8443. It installs once plexsphere is Ready, so after the first migration.
  • The BackendTrafficPolicy plexsphere-api and the ConfigMap plexsphere-seed-manifest, described below.

Every Pod runs one replica, the lab's grade.

Where the charts come from ​

The two charts come from the edge channel, not from a release. The OCIRepository plexsphere-edge selects the newest oci://ghcr.io/plexsphere/charts/plexsphere whose version is 0.0.0-edge.…, and plexsphere-signer-edge the newest signer chart, with the recipe of Select an edge version with Flux and the pull secret plexsphere-registry. A chart's appVersion is the edge tag of its commit, so the API, migrate, console and signer images default to the edge images of the same commit. plexsphere-edge polls every minute and plexsphere-signer-edge every five, so on a merge the control plane usually upgrades first. A poll lists the tags and pulls a chart only when the newest version changes, once per merge.

The values and why ​

The HelmRelease plexsphere sets only what the lab needs:

  • externalUrl is https://plexsphere.<ip-dashed>.nip.io, without a trailing slash. The CSRF allowlist, the OIDC callback <externalUrl>/v1/auth/callback (the redirect URI the Dex client plexsphere has registered), the action, access and enrolment callbacks and the plexd download URL derive from it.
  • gateway routes /v1 and /downloads to the API and everything else to the console on that hostname, through the https listener of the Gateway lab only; nothing attaches to the plaintext listener, whose redirect answers http:// with a 308. networkPolicy.gatewayFrom admits the Envoy proxy Pods, labelled gateway.envoyproxy.io/owning-gateway-name: lab, and nothing else from outside the release. The chart's default deny and the SpiceDB and signer policies stay on.
  • rateLimit.clientIpSource: proxy-headers keys the rate limiter on the client Envoy appends to X-Forwarded-For, with Envoy as the one trusted hop. The proxy Service's externalTrafficPolicy: Local keeps that address the client's. With remote-addr every client would share the proxy Pod's one budget.
  • database, spicedb.datastore, nats, objectStore and actions are the names of What the control plane leg configures. The bundled SpiceDB keeps the chart's TLS chain, and the API reads its CA from spicedb-tls.
  • audit.pepper keeps each Domain's audit pepper in the OpenBao mount plexsphere. The API logs in with the Kubernetes auth role plexsphere-audit-pepper as its ServiceAccount plexsphere, presenting the token for the audience vault that api.extraVolumes projects at audit.pepper.kubernetesJwtPath. The ServiceAccount's own token, which the Kubernetes API accepts with the chart's ClusterRole, never travels to OpenBao's plaintext port. The static pepper of the dev stack stays off.
  • platformIdp and platformOperator make operator@example.com the Platform Operator: at boot the API seeds a platform binding dex-platform on the lab's Dex and grants that subject platform:plexsphere#admin. The client secret reference is a placeholder: the API's code exchange is PKCE-only and sends no secret. The user id is the dev stack's, so the two bootstrap manifests stay comparable.
  • api.extraEnv turns the node event stream on against the lab's NATS (PLEXSPHERE_SSE_NATS_URL) and sets SSL_CERT_DIR to the lab CA, which api.extraVolumes mounts from the ConfigMap lab-ca-public. Dex's issuer, token endpoint and keys sit behind the lab's edge certificate, which the lab CA signs, and the API reaches them through the public address. SSL_CERT_DIR adds the lab CA to the directories Go scans and keeps the image's bundle file, so the API still verifies GitHub for the plexd download mirror and Let's Encrypt for the bridge.
  • provisioning.caBundle names the same ConfigMap lab-ca-public, key response.body. The API mounts it at /etc/plexsphere/provisioning-ca and the broker writes the lab CA into every cloud-init document it renders, so a machine it provisions trusts the API's edge certificate before it downloads plexd and registers.
  • cloudCredentials keeps every cloud credential the API issues in the OpenBao mount cloud-credentials. The API logs in with the Kubernetes auth role plexsphere-cloud-credentials, presenting the same projected vault token as for the audit pepper (cloudCredentials.kubernetesJwtPath). defaultTtl: 2160h gives a credential issued without a lifetime of its own 90 days, the longest an API token lives.
  • provisioning.cloudCredentialsStore names the ClusterSecretStore plexsphere-cloud-credentials the broker's credential ExternalSecrets read through, and provisioning.providerConfigApiGroups lists openstack.m.crossplane.io, the group of the lab Cloud's ProviderConfig. The two grant the API externalsecrets and providerconfigs in exactly those groups, nothing else.
  • imagePullSecrets pulls the private images with plexsphere-registry; signing.keyId names the Domain's key.

Left unset on purpose: api.replicas (one Pod), auth.*, provisioning.* apart from provisioning.caBundle, provisioning.cloudCredentialsStore and provisioning.providerConfigApiGroups (the download URL derives from externalUrl, and the API reaches GitHub on TCP 443), artifacts, observability and spicedb.imagePullSecrets (the SpiceDB image is public). managementFleet is set; see The provisioning substrate.

Two responses on /v1 stay open for as long as the client keeps them: a node's event stream and the session attach WebSocket. Envoy Gateway's route timeout of 15 seconds would cut both, and a slow /downloads/plexd fetch too. The BackendTrafficPolicy plexsphere-api sets requestTimeout: 0s, no route timeout, on the API's route alone; the Dex and CA routes keep the default. The stream idle timeout stays at its default, because the API's keep-alives and pings arrive well inside it. Should a node's stream still reset every 15 seconds with the policy Accepted, set requestTimeout: 12h, the session's maximum lifetime, instead.

The Secrets ​

Six ExternalSecrets of lab-secrets write the Secrets the two charts read, from the four OpenBao keys the bootstrap seeds:

SecretWritten by the ExternalSecretFromRead by
plexsphere-cursor-keysplexsphere-cursor-keyslab/api-keysThe API: thirteen cursor HMAC keys, NSK_WRAP_KEY_B64 and SIGNING_PUBLIC_KEY_B64.
spicedb-credentialsspicedb-credentialslab/spicedbSpiceDB, the API and the seed Job: SPICEDB_GRPC_PRESHARED_KEY.
plexsphere-signer-seedplexsphere-signer-seedlab/signerThe signer: seed, decoded from base64 to the 32 raw bytes.
plexsphere-signer-tlsplexsphere-signer-tlslab/signer-mtlsThe signer's server certificate, kubernetes.io/tls.
plexsphere-signer-client-caplexsphere-signer-client-calab/signer-mtlsThe signer: client-ca.crt, which verifies the API.
plexsphere-access-signer-clientplexsphere-access-signer-clientlab/signer-mtlsThe API: client.crt, client.key and ca.crt for the signer.

The bootstrap mints the signer's chain itself rather than through cert-manager, because the charts read client.crt, client.key and client-ca.crt, names cert-manager does not write. The Postgres DSN (plexsphere-db-app), the object store pair (plexsphere-object-store) and the pull secret (plexsphere-registry) are the backing services'.

The signer's key ​

The signer runs the software key provider in dev mode (devMode.enabled, which sets PLEXSPHERE_ENV=dev on the signer alone; the API runs without a dev marker). The seed is minted once by the bootstrap, kept in the state directory and replayed into OpenBao, so a rebuilt lab signs with the same key and every enrolled node keeps trusting the Domain. The provider keeps the seed in process memory, which is not a production posture: the signer reports the token software-provider-dev-only on /livez/warnings. The signer admits one caller, the client certificate with the SPIFFE ID spiffe://plexsphere.lab/core-binary.

The seed ​

Step 10b of the bootstrap runs the Job plexsphere-seed once. It reads the manifest in the ConfigMap plexsphere-seed-manifest and creates:

  • the Domain lab with the mesh range 10.64.0.0/16;
  • its primary IdP binding dex on https://dex.<ip-dashed>.nip.io, the client plexsphere;
  • operator@example.com (the Dex subject of the static user) with the relations admin and auditor on the Domain: admin lets the operator see the Domain, create Projects and issue bootstrap tokens, auditor reads the audit log;
  • the GitHub accounts listed after it, each a Domain admin (admin and auditor) or a user (member), as Roles for GitHub accounts describes;
  • the Project lab-nodes, under the pinned id 019a0816-0000-7000-8000-000000000002, where nodes enrolled by hand join; it reserves no sub-range and draws its addresses from the Domain's mesh range;
  • the Project openstack in the Domain lab, under the pinned id 019a0816-0000-7000-8000-000000000001. Step 10c assigns the lab's OpenStack Cloud and its credential to that id, which the bootstrap keeps as SEED_PROJECT_ID, and the management fleet places the Project on the lab cluster and creates its namespace plexsphere-project-019a0816-0000-7000-8000-000000000001.

It seeds no grant or token. The Job's log ends with seed-complete domains_created=1 domains_skipped=0 on the first run; its line domain-created slug=lab id=<uuid> names the Domain's ID, which the sign-in asks for. The Job stays as the record that the seed ran, and a re-run of the bootstrap applies nothing while it exists. To run the seed again, delete it and re-run the bootstrap: the seed is idempotent, so the new Job reports domains_skipped=1 and binding-skipped, and the Domain keeps its ID. A changed mesh_cidr is refused rather than applied. A lab whose seed ran before the manifest named the Project openstack has no such Project until then: delete the Job once (kubectl -n plexsphere delete job plexsphere-seed) and re-run the bootstrap, or step 10c stops at the first assignment.

The seed Pod carries two of the API's labels, app.kubernetes.io/name: plexsphere and app.kubernetes.io/part-of: plexsphere, the peers the SpiceDB and Postgres policies admit. It mounts no ServiceAccount token and runs under the restricted Pod Security profile.

The Platform Operator ​

The same person holds two roles under two user rows, as in the dev stack: the Domain admin the seed creates, and the Platform Operator the API seeds at boot under the reserved platform-operators Domain. A Platform Operator can create Domains, and can neither list them nor issue bootstrap tokens; Signing in and issuing a bootstrap token uses the Domain admin. The GitHub accounts in platformOperator.additionalSubjects of the HelmRelease plexsphere are Platform Operators as well, and a workspace gate requires each of them to be a Domain admin in the seed manifest too, for the same reason.

The provisioning substrate ​

The API runs inside the cluster it manages, so the lab cluster is also its management cluster. The API registers it at boot as demo-management-cluster. The management fleet places Projects on it, creates a namespace plexsphere-project-<project-id> for each Project it places and installs the provider packages a Cloud declares when the Cloud is assigned to a Project. The provisioning broker renders each provisioned resource as a Crossplane composite into that namespace.

The fleet installs neither Crossplane nor the External Secrets Operator. Each time it reconciles a Project placed on the cluster, its verify gate checks that the cluster serves apiextensions.crossplane.io, pkg.crossplane.io and external-secrets.io, and that the Deployments crossplane and external-secrets are Available. A failed gate leaves the API Ready; the Project's namespace stays in Provisioning and never reaches Ready. The lab runs both controllers in plexsphere, and the HelmRelease plexsphere sets managementFleet.crossplaneNamespace and managementFleet.externalSecretsNamespace to plexsphere, so the API looks there rather than in the upstream charts' namespaces. verify.sh check 22 repeats the gate from outside.

  • The HelmRelease crossplane runs Crossplane v2 from the upstream chart: the Deployments crossplane and crossplane-rbac-manager. The chart's defaults set no runAsNonRoot, no seccomp profile and keep every capability, which the namespace's restricted Pod Security profile refuses; the values add the three fields and keep the chart's user, group and read-only root file system. The chart installs no package itself.
  • Crossplane installs its CRDs from its init container, so the Kustomization lab-crossplane-packages applies the package objects one step later. Both kinds there are cluster-scoped: the Functions function-patch-and-transform and function-auto-ready, which the blueprints' Compositions run, at the dev stack's tag and digest, and the DeploymentRuntimeConfig default.
  • Crossplane runs every package as a Deployment in its own namespace, without a seccomp profile and without dropping capabilities, so the restricted profile refuses that Pod too and the package never turns Healthy. The runtime config default adds the two fields. Every package that names no runtime config uses it: the two Functions and every provider the management fleet installs for a Cloud. A runtime config replaces Crossplane's defaults for each field it sets, so it repeats the user and group 2000, runAsNonRoot and the two privilege fields, and its container must be named package-runtime.
  • The ClusterSecretStore plexsphere-cloud-credentials, in deploy/lab/secrets beside the store lab, is how a provider in a Project namespace gets its credential. The broker writes an ExternalSecret per credential into the namespace, and ESO copies the credential's payload from cloud-credentials/clouds/<cloud>/credentials/<credential> into the Secret key credentials the ProviderConfig reads. The store's conditions admit only namespaces labelled app.kubernetes.io/managed-by: plexsphere and app.kubernetes.io/component: project-namespace, the labels the fleet stamps on every Project namespace. It logs in with the OpenBao role lab-cloud-credentials, which step 9 of the bootstrap writes: until then the store is not Ready, and lab-secrets holds everything that depends on it back. After merging a change that adds the store to a running lab, re-run the bootstrap.

The lab's Cloud is the OpenStack Cloud lab-openstack, which step 10c registers; see The OpenStack Cloud. Once it is assigned to the Project openstack, the fleet installs the Provider provider-openstack on the cluster, and the broker renders the ProviderConfig openstack and the credential's ExternalSecret into the Project's namespace.

The broker reads the API group and kind of a blueprint's composite from the XRD in the imported catalog, and renders only the composite, the ProviderConfig and the ExternalSecret; it installs neither the XRD nor the Composition. The bootstrap installs both pairs from the official catalog in step 10d, so a resource provisioned from one can reach Ready. The lab offers the official catalog's openstack-instance and openstack-landing-zone; see The OpenStack blueprints.

A machine on an image that runs cloud-init's ca_certs module trusts the lab CA from its first boot, because the broker writes the CA into the cloud-init document. The NixOS machine image of plexsphere-node reads the CA from /etc/plexd/ca.crt, which the same document writes (its README, "Provision it through the plexsphere broker").

The lab API session ​

Steps 10c to 10e of the bootstrap and checks 23 to 25 of verify.sh call the API as operator@example.com, the Dex static user the HelmRelease plexsphere names as Platform Operator and the seed manifest makes the admin of the Domain lab. Both scripts source deploy/lab/scripts/lab-session.sh, which signs that user in the way the console does:

  1. GET /v1/auth/idp-bindings?scope=platform names the platform's binding, and POST /v1/auth/sign-in with it answers an authorization URL on Dex.
  2. Dex's /auth/local leads to its password form, which is posted with the password of dex-operator.password in the state directory, on curl's stdin.
  3. Dex redirects through /v1/auth/callback, which sets the session cookie and the CSRF cookie in a cookie jar in a temporary directory the run deletes when it ends.

That is the platform session, which the Cloud, its credential and their assignments need (platform#manage). The scripts then sign in a second time, with {"domain_id": …}, for the domain session, which the Domain's catalog source needs (manage on the Domain). The API keeps the two apart: the platform session holds no right in the Domain's Projects, and does not even list the Domain. So the scripts read the id the seed generated for the Domain lab from the database, with psql in the primary of the Postgres cluster plexsphere-db, the slug passed as a variable. A database without the Domain stops the run with cannot find the Domain lab: the database holds no Domain lab; did the seed Job run?.

Every call sends the session's cookies, the API's Origin and the CSRF cookie's value in X-Plexsphere-CSRF, which the API asks of a cookie-authenticated write. No API token is minted, so nothing has to be revoked: the sessions expire on their own. verify.sh reads the password from LAB_STATE_DIR as the bootstrap does, default $HOME/.plexsphere-lab/$LAB_KUBE_CONTEXT.

A refused password stops the bootstrap with cannot sign in as operator@example.com at platform scope: the IdP refused the password of operator@example.com in <state>/dex-operator.password: the file no longer holds the password whose hash step 9 seeded into OpenBao for Dex. Restore it from the password manager.

The OpenStack Cloud ​

The OpenStack Cloud is optional. A lab without it leaves LAB_OPENSTACK_RC_FILE and the five LAB_OPENSTACK_* inputs out of lab.env: the bootstrap skips step 10c, verify.sh skips the parts of check 23 that need it, and nodes join by hand as The first node describes.

Step 10c registers the OpenStack through the API with the platform session, so the API stores the credential in OpenBao and the authz tuples follow from the API's own events. It takes the OpenStack in one of two shapes.

An rc file. LAB_OPENSTACK_RC_FILE names the rc file the OpenStack's dashboard hands out, absolute or relative to the checkout root, where the .gitignore keeps Cleura's <project>--<region>--<user>--rc and Horizon's <project>-openrc.sh out of Git. For Cleura's region Fra1:

text
LAB_OPENSTACK_RC_FILE=px001--fra1--px001--rc
LAB_OPENSTACK_CLOUD_SLUG=cleura
LAB_OPENSTACK_CLOUD_NAME=Cleura

The bootstrap reads the file's OS_* assignments and never runs it: a value that refers to another variable, as Horizon's OS_PASSWORD=$OS_PASSWORD_INPUT does, is skipped, so an rc file that prompts for the password needs a line export OS_PASSWORD=<password> of its own. The file must name an https:// OS_AUTH_URL, the password, the user (OS_USER_ID, or OS_USERNAME with its domain's name or id), the project (OS_PROJECT_ID, or OS_PROJECT_NAME with its domain's name or id) and OS_REGION_NAME, which LAB_OPENSTACK_REGION overrides. The rc file replaces the other four LAB_OPENSTACK_* inputs; one of them beside it stops the run.

With it, step 10c authenticates the rc file's user with its password against Keystone (the /v3 of OS_AUTH_URL), scoped to its project, and creates the application credential plexsphere-lab (LAB_OPENSTACK_APPLICATION_CREDENTIAL_NAME) with the user's roles in that project, so the credential reaches nothing else. The password reaches Keystone in a body on curl's stdin and the Keystone token in a header file, so neither shows in a process list or a log line. Keystone shows the credential's secret once, so the bootstrap keeps it in openstack-application-credential.json in the state directory, mode 0600, beside the auth URL, the user, the project and its id. A re-run takes the credential from that file and creates none. A file that names another user, project or Keystone than the rc file stops the run; move it aside to create a new credential. A credential of the same name the state directory lacks stops the run too, because its secret is lost: delete it with openstack application credential delete plexsphere-lab, or set another name, and re-run. The credential carries no expiry; the plexsphere credential issued from it does.

The five inputs. Without an rc file, the five LAB_OPENSTACK_* inputs name an application credential created by hand: the Keystone endpoint, the region, the id of the OpenStack project the machines land in, and the credential scoped to that project:

text
LAB_OPENSTACK_AUTH_URL=https://<keystone>/v3
LAB_OPENSTACK_REGION=<region>
LAB_OPENSTACK_PROJECT_ID=<openstack-project-id>
LAB_OPENSTACK_APPLICATION_CREDENTIAL_ID=<application-credential-id>
LAB_OPENSTACK_APPLICATION_CREDENTIAL_SECRET=<application-credential-secret>

Either way, every run proves the application credential by authenticating with it. A refusal stops the run with Keystone refused the application credential <id>: POST <keystone>/auth/tokens answered <status>; …. Keystone's answer carries the service catalog, from which step 10e reads the region's public Neutron endpoint. The cluster has no egress firewall, so provider-openstack reaches the OpenStack's API ports from the cluster as they are: Cleura's Keystone answers on 5000, Nova on 8774, Neutron on 9696 and Glance on 9292.

Then step 10c sends every request through one curl call with the lab CA, the session's cookies and the body on stdin, so the application credential secret reaches no process list or log line. It:

  1. pages GET /v1/clouds?limit=200 for the slug LAB_OPENSTACK_CLOUD_SLUG (lab-openstack unless set), and when it is missing creates it with POST /v1/clouds: the display name LAB_OPENSTACK_CLOUD_NAME (Lab OpenStack unless set), provider openstack, the OpenStack project's id as the external id, endpoint {auth_url, region}, the region as the default region, the package xpkg.crossplane.io/crossplane-contrib/provider-openstack v0.11.0 and the ProviderConfig version openstack.m.crossplane.io/v1beta1;
  2. pages the Cloud's credentials for an active one named like the Cloud's slug. When the run has just created the application credential, such a credential was issued from an earlier one, so the step revokes it (POST /v1/cloud-credentials/<id>/revoke) and logs revoked the credential <slug> (<id>), issued from an earlier application credential. When there is none, it issues one with the payload {auth_url, region, application_credential_id, application_credential_secret}, the document provider-openstack reads;
  3. assigns the Cloud and then the credential to the Project openstack (019a0816-0000-7000-8000-000000000001), accepting an assignment that already exists.

A 403 is retried every 2 seconds for up to a minute: the tuples that let the Platform Operator manage the Cloud it created and assign the credential it issued arrive through the authz outbox a moment after the write. A 401 stops the run with <method> <path> refused the platform session of operator@example.com; re-run to sign in again, and any other answer with the method, the path, the status and the Problem code.

Once the assignments exist, the fleet installs provider-openstack and the broker renders the ExternalSecret and the ProviderConfig for the first resource provisioned in the Project.

The application credential outlives the lab: emptying the cluster leaves it in Keystone and in the state directory, and a rebuild with the same state directory uses it again. To retire it, delete it in Keystone and move openstack-application-credential.json aside.

Prove the credential authenticates ​

Without a blueprint, an Observe-only managed resource proves the whole chain from inside the cluster: ESO materialises the credential, the provider reads it and reaches Keystone and Nova. Take the two ids from step 10c's pass line and the id of a flavor the project sees (openstack flavor show -f value -c id <name>), then apply:

bash
ns=plexsphere-project-019a0816-0000-7000-8000-000000000001
cloud=<cloud id> credential=<credential id> flavor=<flavor id>
kubectl -n "$ns" apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: openstack-probe
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: plexsphere-cloud-credentials}
  target: {name: openstack-probe, creationPolicy: Owner}
  data:
    - secretKey: credentials
      remoteRef: {key: cloud-credentials/clouds/$cloud/credentials/$credential, property: payload}
---
apiVersion: openstack.m.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: openstack-probe
spec:
  credentials:
    source: Secret
    secretRef: {name: openstack-probe, namespace: $ns, key: credentials}
---
apiVersion: compute.openstack.m.crossplane.io/v1alpha1
kind: FlavorV2
metadata:
  name: openstack-probe
  annotations:
    crossplane.io/external-name: $flavor
spec:
  managementPolicies: ["Observe"]
  forProvider: {}
  providerConfigRef: {kind: ProviderConfig, name: openstack-probe}
EOF

The ExternalSecret turns Ready, the Secret openstack-probe holds the issued JSON under credentials, and the FlavorV2 reaches Synced=True and Ready=True. Then delete the three objects:

bash
kubectl -n "$ns" delete flavorv2.compute.openstack.m.crossplane.io/openstack-probe \
  providerconfig.openstack.m.crossplane.io/openstack-probe externalsecret/openstack-probe

The OpenStack blueprints ​

The lab provisions OpenStack machines from the official catalog's blueprint openstack-instance, and the network they attach to from its blueprint openstack-landing-zone. The control plane never installs a blueprint on a management cluster, so step 10d of the bootstrap does it from the signed bundle ghcr.io/plexsphere/blueprints:v1, and imports the same versions from the same bundle into the catalog of the Domain lab, so the Domain's Projects can provision from them:

  • openstack-instance at v1alpha2: the XRD xopenstackinstances.blueprints.plexsphere.com, whose namespaced composite XOpenStackInstance serves v1alpha2, and the Composition openstack-instance, which composes one namespaced InstanceV2.
  • openstack-landing-zone at v1alpha1: the XRD xopenstacklandingzones.blueprints.plexsphere.com, whose namespaced composite XOpenStackLandingZone serves v1alpha1, and the Composition openstack-landing-zone, which composes a namespaced NetworkV2, SubnetV2, RouterV2 and RouterInterfaceV2.

bootstrap.sh pins the two versions in OPENSTACK_BLUEPRINT_VERSION and OPENSTACK_LANDING_ZONE_BLUEPRINT_VERSION, and the two XRD names in OPENSTACK_INSTANCE_XRD and OPENSTACK_LANDING_ZONE_XRD. Both Compositions run the two Functions of lab-crossplane-packages and compose resources of provider-openstack v0.11.0, which authenticate through the ProviderConfig openstack the broker writes into the Project namespace.

The catalog source belongs to the Domain lab, not to the catalog as a whole. A Domain's Blueprint is readable to every member of the Domain through its parent Domain: the Project openstack lists it under its blueprints, and the console resolves its versions. A catalog-global Blueprint would be readable only to the Platform Operator that imported it, a principal with no right in the Domain's Projects.

The instance blueprint takes four parameters:

ParameterRequiredSets on the server
imageNameyesThe image it boots, on the flavor's disk
flavorNameyesThe flavor
networkNameyesThe one network it attaches to
availabilityZonenoThe availability zone; the OpenStack default without it

The server is named after the composite, carries the broker's cloud-init document as its user data and the Project's cloud tags as server metadata. The landing zone's parameters are in The OpenStack landing zone.

Installing from the signed bundle ​

The host pulls and verifies the bundle itself, with curl, jq, openssl and cosign. The API declares a blueprint's XRD and Composition storage-internal and carries no digest of the bundle the host could pull by. Flux applies neither pair either: a Flux OCIRepository on the bundle selects the first layer of a media type, while the bundle carries one XRD layer per entry, and a Flux GitRepository on github.com/plexsphere/blueprints would apply a commit nothing verified, because Flux checks no cosign signature on a git source. Each registry request is bounded to 60 seconds. Step 10d:

  1. asks ghcr.io for an anonymous pull token with GET https://ghcr.io/token?scope=repository:plexsphere/blueprints:pull. The package is public; the control plane pulls it the same way;
  2. fetches the bundle's manifest by its tag with GET https://ghcr.io/v2/plexsphere/blueprints/manifests/v1 and Accept: application/vnd.oci.image.manifest.v1+json. The token travels in a header file, so it never shows in the host's process list. The bundle's digest is the SHA-256 of the manifest bytes the host received, not a header the registry sends, so a registry that served other bytes under the tag fails the signature check. The manifest must declare the media type application/vnd.oci.image.manifest.v1+json and the artifact type application/vnd.plexsphere.blueprints.catalog.v1;
  3. verifies that digest with cosign verify ghcr.io/plexsphere/blueprints@sha256:<digest> --new-bundle-format against the release identity the catalog source pins below, the SAN pattern ^https://github\.com/plexsphere/blueprints/\.github/workflows/release\.yml@refs/tags/v.*$ and the issuer https://token.actions.githubusercontent.com, and prints official catalog ghcr.io/plexsphere/blueprints:v1 @ sha256:<digest>: signed by the official release identity;
  4. for openstack-instance, then openstack-landing-zone, finds the three layers titled catalog/<slug>/xrd.yaml, catalog/<slug>/composition.yaml and catalog/<slug>/metadata.json with the media types of the bundle contract, fetches metadata.json by its digest with GET https://ghcr.io/v2/plexsphere/blueprints/blobs/<digest>, and compares its version with the pin. The registry answers a blob with a redirect to a pre-signed CDN URL, which curl follows without the token. A slug the bundle offers at another version, without a version, or not at all is skipped on its own with skipping the OpenStack blueprint: the official catalog offers <slug> <version>, the lab pins <pin>; re-run once the catalog's v1 tag serves it (docs/contributing/lab-cluster.md, The OpenStack blueprints) (or offers <slug> without a version, or offers no <slug>): a version the lab does not pin names a composite kind the lab does not expect;
  5. fetches the XRD and the Composition of a slug at its pin by their digests and checks each blob's SHA-256 against the digest the signed manifest names, so what it applies is what the release identity signed. The apply runs with the operator's rights, so before it applies either layer it reads both with kubectl create --dry-run=client -o name, and each must hold one object and nothing else: the XRD <xrd> or the Composition <slug>. A layer that also carries a ClusterRoleBinding, for instance, never reaches the cluster. It applies the bytes unchanged, the XRD first, with kubectl apply --server-side --field-manager plexsphere-lab-bootstrap --force-conflicts. It waits up to two minutes for the XRD to become Established and prints openstack blueprint <slug> <pin>: XRD <xrd> Established, Composition <slug> applied, from the official catalog @ sha256:<digest>.

Server-side apply under one field manager changes nothing on a re-run over an unchanged bundle. --force-conflicts lets the bootstrap take over the fields another manager holds, which matters once, on a lab whose pairs Flux applied (see Migrating a lab that Flux installed the pairs on).

Each of these messages stops the run before anything is imported:

  • GET <url> did not reach ghcr.io: curl could not reach the registry or its CDN, or the request took longer than 60 seconds.
  • GET https://ghcr.io/token?scope=repository:plexsphere/blueprints:pull answered <status>; the official catalog's registry issued no anonymous pull token: the registry refused an anonymous pull, for instance because the package is no longer public.
  • GET https://ghcr.io/token?scope=repository:plexsphere/blueprints:pull answered 200 without a token: the registry's token answer carries no token.
  • GET https://ghcr.io/v2/plexsphere/blueprints/manifests/v1 answered <status>; the official catalog's registry does not serve ghcr.io/plexsphere/blueprints:v1: the tag does not exist, or the registry refused the token.
  • ghcr.io/plexsphere/blueprints:v1 is not a blueprint catalog bundle (mediaType <value>, artifactType <value>): the tag names an image index or another artifact.
  • cosign refused ghcr.io/plexsphere/blueprints@sha256:<digest>: <cosign's last line>; the bundle is not signed by the official release identity: no signature of the release identity covers the manifest the host received, or cosign could not reach the registry or the Sigstore trust root.
  • the official catalog's layer catalog/<slug>/<file> answered <status>: the registry does not serve a blob the manifest names.
  • the official catalog's layer catalog/<slug>/<file> has the digest sha256:<got>, the manifest names <digest>: the registry or its CDN served other bytes than the signed manifest names.
  • kubectl cannot read the official catalog's layer catalog/<slug>/<file>: the layer is no stream of Kubernetes objects, or the API server does not know their kind; kubectl's own message is printed above it.
  • the official catalog's layer catalog/<slug>/<file> holds <objects>, not the one object <object>: the layer carries another object than the XRD or the Composition the lab installs, or more objects than that one.
  • kubectl apply of the XRD of <slug> failed or kubectl apply of the Composition of <slug> failed: the API server refused the object; kubectl's own message is printed above it.
  • the XRD <xrd> of <slug> did not become Established: Crossplane did not establish the XRD's CRD within two minutes. The XRD's conditions are printed before the message. The pair stays applied, and a re-run applies it again.

Registering and importing ​

Step 10d sends every request to the lab API the way step 10c does, with the domain session of The lab API session: one curl call with the lab CA, the session's cookies and the body on stdin. A 403 is retried for up to a minute and a 401 stops the run. It:

  1. lists the catalog sources (GET /v1/blueprint-catalogs) for one of the Domain lab on ghcr.io, repository plexsphere/blueprints, tag v1. When there is none, it registers it as Official Blueprint catalog of the Domain with POST /v1/blueprint-catalogs, pinned to the release identity of the official catalog: the SAN pattern ^https://github\.com/plexsphere/blueprints/\.github/workflows/release\.yml@refs/tags/v.*$ and the issuer https://token.actions.githubusercontent.com. It checks every source of the Domain on that bundle, not only the one it imports through. A source that is not pinned to exactly that identity, one registered by hand as unsigned for instance, stops the run. So does one that tracks the tag, whose tracking sweep would import every version the tag publishes, and a disabled one. The run stops with the catalog source <id> for ghcr.io/plexsphere/blueprints:v1 is not an active source that pins the official release identity without tracking its tag; deregister it and re-run (docs/contributing/lab-cluster.md, The OpenStack blueprints);
  2. imports the slugs it installed, in that order, in one POST /v1/blueprint-catalogs/<id>/import whose body is {"slugs": ["openstack-instance", "openstack-landing-zone"]}, or the one slug it installed. When it installed neither, it imports nothing, and step 10e skips the landing zone. It checks every outcome before it looks a version up. imported and unchanged continue when the outcome's version is the slug's pin. The import resolves the v1 tag again, so a tag that moved since the host pulled the manifest stops the run with the official catalog moved to <slug> <version> during the import, which is now published to the Domain lab; the lab pins and installed <pin>, so bump the pin and re-run before a Resource uses <version>. The import publishes that version before it answers, and the API retires no version, so it stays in the Domain's catalog. drift stops the run with the official catalog changed <slug> <pin> after the lab imported it; the catalog must publish the change under a new version: the import never replaces a published version, so changed content under the same label needs a new label upstream, then a pin bump here. conflict and error stop the run with the outcome's status and reason;
  3. finds, for each imported slug, the Domain's Blueprint and its version at the pin, and prints the version id: the instance's is the one the next sections provision machines with, the landing zone's the one step 10e provisions with. The list leaves out a Blueprint the user cannot read yet, and the tuples that let the Domain read the one the import registered arrive through the authz outbox a moment later, so the lookup is retried every 2 seconds for up to a minute.

Every object is looked up or applied idempotently, so a re-run is safe, also after a run that stopped at any point, and on an unchanged bundle each import line reads unchanged.

The installed XRDs and Compositions are in no Flux inventory, so no prune reaches them. Deleting an XRD deletes its CRD, every composite of its kind and, through Crossplane, every server or network they composed. A blueprint therefore leaves the lab only by hand: delete its resources with plexctl resource delete, then run kubectl delete compositeresourcedefinition with its XRD, xopenstackinstances.blueprints.plexsphere.com or xopenstacklandingzones.blueprints.plexsphere.com, and kubectl delete composition with its Composition, as Emptying the cluster does.

Refreshing the fixtures ​

tests/integration/testdata/blueprints/<slug>/ holds the xrd.yaml, composition.yaml and metadata.json of each entry the lab installs, equal to the catalog's layers of the same titles. The lab never applies them. They are fixtures: the broker-render integration tests, the envtest bootstrap-chain tests, the workspace contract tests and the provisioning-broker e2e scenario on a real Crossplane read them, so the repository proves that the broker renders the catalog's OpenStack entries and that Crossplane admits them. The workspace gates TestLabManifests_OpenStackBlueprint and TestLabManifests_OpenStackLandingZoneBlueprint hold each fixture XRD's served version to its pin in bootstrap.sh and its metadata.name to OPENSTACK_INSTANCE_XRD or OPENSTACK_LANDING_ZONE_XRD, so a pin bump fails them until the fixtures follow in the same commit.

To move the lab to a new version of an entry:

  1. Bump its pin in bootstrap.sh.
  2. Pull the three layers of the new version with the token, manifest and blob requests step 10d sends, or with oras pull ghcr.io/plexsphere/blueprints:v1 on a host that has oras, and copy them over the fixture.
  3. When the XRD's schema changed, update its envtest CRD under tests/integration/testdata/crds/, which a workspace gate holds to the XRD, and regenerate the digests there with sha256sum *.yaml > crds.sha256.

Migrating a lab that Flux installed the pairs on ​

Before step 10d installed the pairs, a Flux Kustomization of the lab applied them. Removing it from the tree makes Flux delete that Kustomization, and its finalizer deletes its objects except the two XRDs, which carry kustomize.toolkit.fluxcd.io/prune: disabled. The two Compositions are gone until the bootstrap runs again. Meanwhile no composite of either kind, a landing zone's XOpenStackLandingZone or a server's XOpenStackInstance, can select its Composition, and verify.sh checks 24 and 25 fail their composition line. The composed Neutron networks and Nova servers belong to the composite, not to the Composition, and stay. Run deploy/lab/scripts/bootstrap.sh once the change is on main and Flux has reconciled: step 10d applies the Compositions again and takes the XRDs over from kustomize-controller. The label app.kubernetes.io/part-of and the prune annotation the Kustomization added stay on the XRDs and have no effect.

The OpenStack landing zone ​

A server of openstack-instance attaches to a network by name, and a fresh OpenStack project has none it can attach to: Cleura's Fra1 project lists only the external network ext-net. A Project gets that network once, as a Resource of the kind landing-zone provisioned from openstack-landing-zone. Its composite XOpenStackLandingZone composes four managed resources in the Project namespace: a private network, a subnet on it, a router whose gateway sits on the cloud's external network, and the router interface that plugs the subnet into the router. The servers on the network reach the internet through the router; nothing reaches them from outside. No security group is composed: the servers land in the project's default group, which allows all egress.

The blueprint is standalone: the broker mints no bootstrap token for it and walks its Resource through Pending, Provisioning and Ready, never Enrolling. It takes four parameters:

ParameterRequiredSets
externalNetworkIdyesThe id of the external network the router's gateway sits on.
networkNamenoThe name of the network, the subnet and the router; the Project's namespace plexsphere-project-<project id> without it.
cidrnoThe subnet's IPv4 range; 192.168.42.0/24 without it.
dnsNameservernoThe one DNS server the subnet hands out; the cloud's default resolver without it.

The name is what an instance's networkName names, so the operator knows it before the landing zone exists: for the Project openstack it is plexsphere-project-019a0816-0000-7000-8000-000000000001. Nothing enforces one landing zone per Project. A second one of the same Project needs its own networkName: Neutron allows two networks of one name, and Nova's lookup of the name would then be ambiguous.

The cidr must not overlap the Domain's mesh CIDR 10.64.0.0/16 (mesh_cidr in deploy/lab/control-plane/seed-manifest.yaml), the range the Domain's nodes take their mesh addresses from. The default 192.168.42.0/24 lives in the Composition, because the broker writes only the parameters a request supplies.

A machine resolves the lab's nip.io hostnames through the subnet's resolver. When the cloud's default resolver does not resolve public names, plexd join fails on name resolution; a landing zone with dnsNameserver set to a public resolver, such as 1.1.1.1, gives the machines one that does.

Step 10e provisions the landing zone of the Project openstack:

  1. It lists the Project's Resources (GET /v1/projects/019a0816-0000-7000-8000-000000000001/resources) for one of the kind landing-zone that is not Deprovisioning or Deleted. It reuses that one, unless it is Failed, which stops the run. The API answers no parameters of a Resource, so the step reads the network's name from the composite's status.networkName.

  2. Without one, it needs the external network's id, which it reads from Neutron as the application credential of step 10c, at the region's public network endpoint of Keystone's catalog. With LAB_OPENSTACK_EXTERNAL_NETWORK_ID set, it reads GET <neutron>/v2.0/networks/<id>, which must answer 200 with router:external true. Without it, it lists GET <neutron>/v2.0/networks?router:external=true and takes the one network the project sees; none, or more than one, stops the run and asks for LAB_OPENSTACK_EXTERNAL_NETWORK_ID. On Cleura's Fra1 the one external network is ext-net.

  3. It provisions the landing zone with POST /v1/projects/019a0816-0000-7000-8000-000000000001/resources:

    json
    {"origin": "provisioned", "kind": "landing-zone", "blueprint_version_id": "<landing-zone version id>",
 "cloud_credential_id": "<credential id>", "parameters": {"externalNetworkId": "<external network id>"}}

A run that provisions the landing zone sends Neutron one read; a run that reuses one sends none. The OpenStack project pays for the network, the subnet and the router until the landing zone is deleted or the lab is emptied.

A Project on another OpenStack provisions its landing zone the same way, with the landing zone's version id from its Domain's catalog and the credential of its Cloud:

bash
SSL_CERT_FILE=lab-ca.crt plexctl resource create \
  --project-id <project id> --kind landing-zone \
  --blueprint-version-id <landing-zone version id> --cloud-credential-id <credential id> \
  --parameters '{"externalNetworkId":"<external network id>"}'

The composite reports what it composed in its status: networkName, networkId, subnetId and routerId, beside Crossplane's conditions.

bash
kubectl -n plexsphere-project-019a0816-0000-7000-8000-000000000001 \
  get xopenstacklandingzones.blueprints.plexsphere.com -o yaml

A wrong externalNetworkId leaves the Resource in Provisioning for good: the Composition raises no ProvisioningFailed, so the broker never sees a failure. The RouterV2's conditions name Neutron's answer: kubectl -n <namespace> describe routerv2s.networking.openstack.m.crossplane.io.

Tear the landing zone down after the machines on it, with the API:

bash
SSL_CERT_FILE=lab-ca.crt plexctl resource delete <landing zone resource id> --yes

Instances first, then the landing zone: inside one landing zone the router interface goes first, which cuts a live server's egress, and Neutron refuses to delete a subnet or a network a server's port still uses. Deleting an instance leaves the landing zone standing, as it leaves the ProviderConfig openstack. Once the Resource reaches Deleted, kubectl -n plexsphere-project-019a0816-0000-7000-8000-000000000001 get networkv2s,subnetv2s,routerv2s,routerinterfacev2s.networking.openstack.m.crossplane.io lists nothing and openstack network list shows only the external network.

Provision an OpenStack machine ​

With the pass lines of steps 10c, 10d and 10e in hand, provision a machine in the Project openstack. It boots a stock Ubuntu 24.04 cloud image, which runs the broker's cloud-init document as written: ca_certs installs the lab CA, /etc/plexd/install-plexd.sh fetches plexd from https://plexsphere.<ip-dashed>.nip.io/downloads/plexd, and plexd join registers the node. The machine needs outbound TCP 443 to the lab's address and nothing inbound: no floating IP, key pair or security group rule. The bootstrap's two provision: lines are steps 4 and 5 below with the ids and the network filled in; the image and the flavor are the OpenStack project's.

  1. Find the image, or upload it. openstack image list shows what the project sees; to upload, download noble-server-cloudimg-amd64.img from https://cloud-images.ubuntu.com/noble/current/ and run:

    bash
    openstack image create --disk-format qcow2 --container-format bare \
  --file noble-server-cloudimg-amd64.img ubuntu-24.04

  2. Pick a flavor whose root disk holds the image: compare openstack flavor list with the virtual_size of openstack image show <image>.

  3. Take the landing zone's network plexsphere-project-019a0816-0000-7000-8000-000000000001, which step 10e provisions and the second provision: line names. Its Resource must be Ready (verify.sh check 25), and openstack router show plexsphere-project-019a0816-0000-7000-8000-000000000001 -c external_gateway_info shows the router's gateway on the external network. Without step 10e, provision the landing zone as The OpenStack landing zone shows.

    On Cleura's Fra1 the public image Ubuntu 24.04 Noble Numbat x86_64 needs no upload, and a flavor such as 1C-2GB-20GB holds it.

  4. Sign in with plexctl as operator@example.com, as Signing in and issuing a bootstrap token describes:

    bash
    SSL_CERT_FILE=lab-ca.crt plexctl login --server https://plexsphere.<ip-dashed>.nip.io --domain-id <domain-id>

  5. Create the resource with the credential id of step 10c's pass line and the version id of step 10d's:

    bash
    SSL_CERT_FILE=lab-ca.crt plexctl resource create \
  --project-id 019a0816-0000-7000-8000-000000000001 --kind node \
  --blueprint-version-id <version id> --cloud-credential-id <credential id> \
  --parameters '{"imageName":"<image>","flavorName":"<flavor>","networkName":"plexsphere-project-019a0816-0000-7000-8000-000000000001"}'

  6. Watch the phase walk Pending, Provisioning, Enrolling and Ready, within 15 minutes:

    bash
    while :; do
  phase="$(SSL_CERT_FILE=lab-ca.crt plexctl resource get <resource id> --output json | jq -r '.provisioning.phase')"
  echo "$(date +%T) $phase"
  case "$phase" in Ready | Failed) break ;; esac
  sleep 15
done

    Meanwhile the composite and the server's managed resource appear in the Project namespace, and the InstanceV2 reaches Synced=True and Ready=True:

    bash
    kubectl -n plexsphere-project-019a0816-0000-7000-8000-000000000001 \
  get xopenstackinstances.blueprints.plexsphere.com,instancev2s.compute.openstack.m.crossplane.io

  7. Read the first boot: openstack console log show <server>, the server named after the composite, prints the cloud-init output with the plexd install lines and the node_id: and mesh_ip: lines of plexd join. The console's node list of the Project openstack shows the node, and plexctl bootstrap-token list --project 019a0816-0000-7000-8000-000000000001 shows the token the broker minted for the resource as consumed.

Ready means registered. The broker's document installs no plexd unit (see the cloud-init document), so on this image no plexd keeps running after join: the node does not heartbeat and joins no mesh traffic.

Tear the machine down with the API, never with kubectl:

bash
SSL_CERT_FILE=lab-ca.crt plexctl resource delete <resource id> --yes

plexctl resource get <resource id> shows the teardown phases until the phase reaches Deleted and the Resource is gone. The composite and the InstanceV2 leave the namespace, and openstack server list no longer lists the server. The ProviderConfig openstack, the credential's ExternalSecret and the landing zone stay, because the Project's other resources share them.

How the lab updates ​

  • Flux fetches main every minute, and every child Kustomization applies a new revision as it arrives. A merged change to deploy/lab/ reaches the cluster without anyone running kubectl. The GitRepository's artifact holds deploy/lab/ only, so a commit elsewhere in the repository costs the controllers no more than the fetch.
  • Every third-party HelmRelease carries a semver range within one major version, and the HelmRepositories refresh hourly, so the charts follow new upstream patch and minor releases without a commit. The FluxInstance follows Flux 2.x. A workspace gate keeps every range in the form >=X.Y.Z <X+1.0.0. The two composition Functions are the exception: they carry the dev stack's tag and digest, and a workspace gate holds the two trees equal, so they move with a bump there.
  • To move the lab past a broken release, raise the lower bound of the range in a pull request once a fixed release exists. To hold the lab on the release it runs until then, set spec.suspend: true on that HelmRelease in a pull request, and drop it again afterwards.
  • To stop applying a whole directory, set spec.suspend: true on its child Kustomization in deploy/lab/kustomizations.yaml through a pull request. A suspended Kustomization keeps its objects; the HelmReleases it applied keep reconciling on their own.
  • Every Kustomization prunes: an object removed from the tree is deleted from the cluster. The namespace and the Postgres Cluster plexsphere-db are the exceptions; they carry kustomize.toolkit.fluxcd.io/prune: disabled because the namespace holds OpenBao's volume and Flux itself, and the Cluster owns the database's volume. A Cluster that is meant to go is deleted by hand once it is out of the tree.

The control plane follows main through the edge channel instead of a version range:

  • The edge lane publishes the charts and images of every merge to main. Within a minute of the publish (five for the signer) the OCIRepositories pick the new edge version up, and the HelmReleases upgrade. /v1/version on https://plexsphere.<ip-dashed>.nip.io names the version the API runs, v0.0.0-edge.<stamp>.g<sha>, whose <sha> is the commit's, and kubectl -n plexsphere get helmreleases shows the chart version each release applied.
  • A failed upgrade is retried and never rolled back (the upgrade strategy RetryOnFailure, every five minutes, no remediation). The migrate hook has already moved the schema by the time the API fails to roll out, migrations do not go down, and the previous API Pods keep serving against the new schema, so /v1/version keeps naming the old version. helmrelease/plexsphere reports Ready: False with the cause until a retry or the next merge succeeds. Read the failed attempt with kubectl -n plexsphere describe helmrelease plexsphere and the log of the migrate Job, which the chart keeps for ten minutes after it finishes: kubectl -n plexsphere logs job/plexsphere-migrate. A schema error is fixed forward on main.
  • The signer upgrades after the control plane in the common case, not always. Its dependsOn holds it back while helmrelease/plexsphere is not Ready, but cannot compare the two charts' versions, because helm-controller checks a dependency before it loads the new chart. A signer that upgrades first onto a migration the control plane has not applied yet crash-loops until the control plane's migrate hook runs, which its one-minute source makes happen inside the signer's five-minute upgrade wait; the signer's upgrade then completes. Should the wait run out first, RetryOnFailure tries again five minutes later. A first install keeps the order strictly.
  • The edge lane keeps the newest three edge versions of each package, so a lab that was suspended for more than three merges resumes on the newest.

Backups ​

The lab backs up Postgres and nothing else. Postgres holds what a loss would cost for real: the enrolled nodes and the audit chains. NATS streams keep a replay window of 24 hours by design, action output in SeaweedFS is disposable, and every secret in OpenBao is replayed from the bootstrap's state directory.

The ScheduledBackup plexsphere-db-nightly takes a cold volume snapshot of plexsphere-db at 02:00 UTC every night, through the VolumeSnapshotClass in LAB_VOLUME_SNAPSHOT_CLASS of the ConfigMap lab-cluster. Cold means the operator fences the single instance for the seconds the snapshot takes, and Postgres refuses connections for that long; in exchange the snapshot restores without WAL archiving, which the lab does not run. Each run creates a Backup that owns its VolumeSnapshot. No object owns the Backups, so they outlive the ScheduledBackup, and the Cluster is kept out of Flux's prune, so a change to the tree never deletes the database or a backup of it. At 02:30 UTC the CronJob plexsphere-db-backup-prune deletes every completed Backup of the cluster beyond the newest seven, and their VolumeSnapshots with them. A failed Backup does not count toward the seven; it is deleted once a newer Backup has completed, so a run of failed nights never costs the last restorable snapshot. CloudNativePG's own retention covers only object-store backups.

List the backups and their snapshots:

bash
kubectl get backups.postgresql.cnpg.io -n plexsphere
kubectl get volumesnapshots.snapshot.storage.k8s.io -n plexsphere

Take one by hand, for example before a risky change:

bash
kubectl apply -f - <<'YAML'
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: plexsphere-db-manual
  namespace: plexsphere
spec:
  cluster:
    name: plexsphere-db
  method: volumeSnapshot
  online: false
YAML
kubectl wait -n plexsphere backups.postgresql.cnpg.io/plexsphere-db-manual \
  --for=jsonpath='{.status.phase}'=completed --timeout=5m

A completed Backup made by hand counts toward the seven the prune job keeps.

To restore, create a new Cluster from a Backup by hand, as CloudNativePG's recovery documentation describes, with the storage of the original:

bash
kubectl apply -f - <<'YAML'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: plexsphere-db-restore
  namespace: plexsphere
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:16-minimal-trixie
  storage:
    size: 20Gi
  enableSuperuserAccess: false
  postgresql:
    parameters:
      max_connections: "200"
  bootstrap:
    recovery:
      backup:
        name: <backup>
      database: plexsphere
      owner: plexsphere
YAML

Once it is Ready, point database.existingSecret and spicedb.datastore.existingSecret of the HelmRelease plexsphere, and database.existingSecret of the HelmRelease plexsphere-signer, at its Secret plexsphere-db-restore-app instead of plexsphere-db-app, in one pull request. The core rows, the SpiceDB tuples and the signer's keys live in the same database and stay consistent only when all three move together. The upgrade of plexsphere runs the migrate hook against the restored schema; the signer crash-loops until it has, and RetryOnFailure retries it. The seed Job reads plexsphere-db-app too, but a restored database already holds the Domain lab. The restored Cluster is not part of the Flux tree and the NetworkPolicy plexsphere-db does not select its Pods; bring both into deploy/lab/postgres/ through a pull request before the lab relies on it. CloudNativePG can also restore from a VolumeSnapshot named directly in bootstrap.recovery.volumeSnapshots.storage.

Trusting the lab CA ​

The Gateway serves the lab CA certificate at /lab-ca.crt on ca.<ip-dashed>.nip.io, over plaintext HTTP for a machine that does not trust the CA yet:

bash
curl -fsSLo lab-ca.crt http://ca.<ip-dashed>.nip.io/lab-ca.crt
openssl x509 -in lab-ca.crt -noout -fingerprint -sha256

Compare the fingerprint with the one the bootstrap printed and the last line of verify.sh. Only a matching certificate is trusted:

  • Debian and Ubuntu:

    bash
    sudo cp lab-ca.crt /usr/local/share/ca-certificates/plexsphere-lab.crt
sudo update-ca-certificates

  • macOS:

    bash
    sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain lab-ca.crt

  • Firefox keeps its own store: Settings, Privacy & Security, Certificates, View Certificates, Authorities, Import, and trust the certificate for websites.

  • plexctl has no CA flag and Go reads the system store. On Linux without the system-wide step, point Go at the file for one call: SSL_CERT_FILE=lab-ca.crt plexctl …. On macOS the keychain step covers it.

The access file ​

The bootstrap's last step writes access.md into the state directory ($HOME/.plexsphere-lab/<context>/access.md by default), the one file to open before signing in or to hand a colleague who trusts the lab CA. It carries, as Markdown tables:

  • The URLs: the console, the console's sign-in with the Domain lab's id already filled in (https://plexsphere.<ip-dashed>.nip.io/?domain_id=<domain-id>), the API and its /v1/version, Dex, the GitHub OAuth App callback and the CA download, with the address, the wildcard hostname and the CA's SHA-256 fingerprint.
  • The users: operator@example.com and member@example.com with their passwords in clear, read from dex-operator.password and dex-member.password, and their roles. The operator is the Platform Operator and the admin and auditor of the Domain lab; the member holds no role until a Domain admin grants one. GitHub accounts are not listed, they sign in with their own credentials.
  • The ids of the Domain lab, read from Postgres as The lab API session does, and of the seeded Projects lab-nodes and openstack.
  • The plexctl login into the Domain, with SSL_CERT_FILE pointing at the state directory's lab-ca.crt.

The file is derived, not a source: every bootstrap rewrites it whole, so a rebuild with a new address or a restored state directory with other passwords is reflected on the next run, and the teardown deletes it because its URLs name the released address. It is a 0600 file in the state directory, like the password files it quotes, and belongs in no checkout and no chat.

Signing in and issuing a bootstrap token ​

Sign in from a browser that trusts the lab CA (see Trusting the lab CA). The console's sign-in asks for the ID of the Domain lab, which access.md in the state directory names beside the sign-in URL (see The access file), and which the seed Job's log names too:

bash
kubectl -n plexsphere logs job/plexsphere-seed | grep domain-created

When the Job was re-created since, read the ID from Postgres instead:

bash
kubectl exec -n plexsphere plexsphere-db-1 -c postgres -- \
  psql -U postgres -d plexsphere -tAc "SELECT id FROM plexsphere.domains WHERE slug = 'lab'"
  1. Open https://plexsphere.<ip-dashed>.nip.io/?domain_id=<domain-id>, or open https://plexsphere.<ip-dashed>.nip.io and type the ID.
  2. Sign in at Dex as operator@example.com with the password in dex-operator.password of the state directory. The console lands on the Domain lab, with its Audit section readable.
  3. Open the Project lab-nodes, which the seed created, and its Bootstrap tokens tab.
  4. Issue a token of kind node, env prefix lab, valid for one hour. The console shows the token once; copy it to the node.

A member of the plexsphere GitHub organisation can sign in to the same Domain through the GitHub connector. An account listed in Roles for GitHub accounts lands with its role; any other holds none, and its Domain list is empty with empty_reason: no_rebac_membership.

The same steps with plexctl on a host that trusts the lab CA (on Linux without the system-wide step, prefix each command with SSL_CERT_FILE=lab-ca.crt):

bash
plexctl login --server https://plexsphere.<ip-dashed>.nip.io --domain-id <domain-id>
plexctl bootstrap-token issue --project 019a0816-0000-7000-8000-000000000002 --kind node --env-prefix lab --ttl 1h
plexctl bootstrap-token list --project 019a0816-0000-7000-8000-000000000002

plexctl login prints a verification URL to approve in the browser. A further Project comes from plexctl project create --domain <domain-id> --slug <slug> --display-name <name> --sub-range-cidr 10.64.1.0/24; its range lies inside the Domain's mesh range 10.64.0.0/16. See the plexctl project and plexctl bootstrap-token references for their flags.

To sign in as Platform Operator, choose Sign in as a Platform Operator below the Domain-ID form and sign in at Dex with the same account, as Sign in as a Platform Operator describes. That session can create Domains; it lists no Domain and cannot issue a bootstrap token, which needs a role on the Project's Domain. plexctl login --platform is refused on the lab, which does not set PLEXSPHERE_AUTH_PLATFORM_DEVICE_LOGIN.

Roles for GitHub accounts ​

Members of the plexsphere organisation sign in through Dex's GitHub connector, at the Domain lab and at platform scope alike. Their role comes from two files in this repository, keyed by the account's Dex subject:

RoleWhereWhat it grants
Platform OperatorplatformOperator.additionalSubjects in deploy/lab/control-plane/helmrelease-plexsphere.yaml, and a Domain admin entryplatform:plexsphere#admin and managementfleet:fleet#admin: Domains, Clouds, catalog-global blueprint catalogs, cluster registration
Domain admindomain_admins in deploy/lab/control-plane/seed-manifest.yaml, relations: [admin, auditor]the Domain lab: its Projects, bootstrap tokens, identities and groups, and its audit log
Userdomain_admins in the same file, relations: [member], then a role on a Projectreading the Domain; what the user may do in a Project comes from the Project role

The Dex subject of a GitHub account encodes the account's numeric GitHub id and the connector id github as a small protobuf message in base64url without padding. Derive it from the login:

bash
login=<github-login>
id="$(curl -fsS "https://api.github.com/users/$login" | jq -r .id)"
printf '\x0a%b%s\x12\x06github' "$(printf '\\x%02x' "${#id}")" "$id" | base64 | tr -d '\n=' | tr '+/' '-_'; echo

For berendt (id 419091) it prints CgY0MTkwOTESBmdpdGh1Yg. With the static user's id and the connector local the same encoding gives the subject of operator@example.com the manifests carry.

Add the subject with a comment naming the login. A Platform Operator goes into both files:

yaml
# helmrelease-plexsphere.yaml, spec.values
platformOperator:
  additionalSubjects:
    - <subject> # <github-login>
yaml
# seed-manifest.yaml, manifest.yaml
domain_admins:
  - domain_slug: lab
    external_subject: <subject> # <github-login>
    relations: [admin, auditor]   # [member] for a user

Once the change is merged, Flux applies both files. The HelmRelease change rolls the API out, and the new API grants the listed Platform Operators at boot. The seed manifest reaches the ConfigMap plexsphere-seed-manifest but runs only when the bootstrap creates the seed Job, so delete the Job and run the bootstrap again; the seed adopts a row the account's earlier sign-in created and adds the relations:

bash
kubectl -n plexsphere delete job plexsphere-seed
deploy/lab/scripts/bootstrap.sh </dev/null

A user also needs a role on a Project, which a Domain admin grants once the user has signed in. plexctl identity list --domain <domain-id> shows the user's id; then, for the Project lab-nodes:

bash
plexctl rebac tuple add --project 019a0816-0000-7000-8000-000000000002 \
  --subject user:<user-id> --relation maintainer \
  --resource project:019a0816-0000-7000-8000-000000000002

viewer, operator and admin are the Project's other relations. Removing an account from either file revokes nothing: the API and the seed only ever add grants, so a former role stays until its tuples are deleted by hand.

The first node ​

The first node is a Linux host (amd64 or arm64) outside the cluster, installed by hand. It needs outbound TCP 443 to the lab, and UDP 51820 open for the WireGuard peers that later nodes add. It reaches the control plane on the one origin only: POST /v1/register with the bootstrap token, then the heartbeat, the endpoint report, the state pull, the ingest and the event stream, each authenticated with the key the registration hands out.

  1. Trust the lab CA in the host's store, as Trusting the lab CA describes for Debian and Ubuntu. plexd has no option for an extra CA yet; it reads the system store.

  2. Fetch plexd and its checksum list from the lab, check the binary, and install it:

    bash
    arch=amd64 # or arm64
base=https://plexsphere.<ip-dashed>.nip.io/downloads/plexd
curl -fsSLo "plexd-linux-$arch" "$base/linux-$arch"
curl -fsSLo checksums.sha256 "$base/checksums.sha256"
grep " plexd-linux-$arch\$" checksums.sha256 | sha256sum -c -
sudo install -m 0755 "plexd-linux-$arch" /usr/local/bin/plexd

    The API serves the plexd release /PLEXD_VERSION pins, fetched from GitHub on the first request for each platform. While the API cannot reach GitHub, a binary answers 502 with plexd_download_upstream_unavailable.

  3. Write the token to /etc/plexd/bootstrap-token, mode 0600, and install plexd against the lab:

    bash
    sudo install -d -m 0700 /etc/plexd
# Paste the token, then press Ctrl-D.
sudo sh -c 'umask 077; cat >/etc/plexd/bootstrap-token'
sudo plexd install --api-url https://plexsphere.<ip-dashed>.nip.io \
  --token-file /etc/plexd/bootstrap-token

  4. Add the Project and a name for the node to /etc/plexd/config.yaml:

    yaml
    registration:
  project_id: <project-id>
  resource_handle: <name>

  5. Start it:

    bash
    sudo systemctl enable --now plexd
journalctl -u plexd -f

On success the journal shows the registration with the node's ID and mesh IP, then the event stream connecting. The console's node list of the Project shows the node online, and the API logs sse: stream opened with the node's ID once per connection the node keeps. A line every 15 seconds means the Gateway's request timeout is cutting the stream; see Troubleshooting.

A bootstrap token is single-use: starting plexd again with the same token file is refused with token_consumed (HTTP 403); write a fresh token to recover.

Two limits of the lab hold for this node:

  • A mediated session to the node does not connect. The API dials the node's mesh address from its Pod, and the Pod has no route into the mesh; the API logs the failed dial.
  • A plexd inside the cluster is out of scope. WireGuard between two nodes like this one is the next step.

Verification ​

deploy/lab/scripts/verify.sh checks the bootstrapped lab end to end:

bash
deploy/lab/scripts/verify.sh

It reads LAB_KUBE_CONTEXT from lab.env in the checkout root the way the bootstrap does, or from the environment.

It prints one ok: line on stdout per passed check, one skip: <check>: <why> line on stdout per check that does not apply to this lab, and one FAIL: <check>: <what was found> line on stderr per failed check, runs every check, and exits 1 when any failed. The checks:

  1. configmap lab-cluster: the ConfigMap exists, LAB_PUBLIC_IP is an IPv4 address and LAB_PUBLIC_IP_DASHED its dashed form.
  2. flux: the FluxInstance flux, the GitRepository lab, the Kustomization lab and the fifteen children are Ready, and the GitRepository serves a refs/heads/main@sha1: revision, named after the ref the FluxInstance syncs.
  3. helmreleases: flux-operator, openbao, external-secrets, cert-manager, envoy-gateway, crossplane, cloudnative-pg, nats, seaweedfs, dex, plexsphere and plexsphere-signer are Ready.
  4. openbao: bao status reports initialised and unsealed. A status the script cannot read fails the check with FAIL: openbao: bao status in openbao-0 printed no status JSON: … or FAIL: openbao: bao status in openbao-0 failed with exit code …: ….
  5. external-secrets: the SecretStore and the eleven ExternalSecrets are Ready, the Secret lab-ca is kubernetes.io/tls with the certificate of lab-ca-public, the Secret plexsphere-registry is kubernetes.io/dockerconfigjson with an auths entry, the key seaweedfs_s3_config of seaweedfs-s3-config is JSON with one identity, both keys of plexsphere-object-store are non-empty, dex-credentials carries four non-empty keys of which the two password hashes are bcrypt ($2…), plexsphere-cursor-keys carries its fifteen keys non-empty, spicedb-credentials carries SPICEDB_GRPC_PRESHARED_KEY, the key seed of plexsphere-signer-seed holds 32 bytes, plexsphere-signer-tls is kubernetes.io/tls with a certificate naming plexsphere-signer, plexsphere-signer-client-ca carries client-ca.crt, and the client certificate of plexsphere-access-signer-client carries URI:spiffe://plexsphere.lab/core-binary and verifies against the ca.crt beside it.
  6. certificate: the Issuer lab-ca and the Certificate lab-edge-tls are Ready, and the certificate names *.<ip-dashed>.nip.io and <ip-dashed>.nip.io and verifies against the lab CA.
  7. gateway: the Gateway lab is Programmed on <ip>, exactly one LoadBalancer Service carries <ip>, and the proxy Deployment in plexsphere has one ready replica.
  8. endpoints, over the public network: the CA download answers the same bytes over HTTP (Content-Type: application/x-pem-file) and over HTTPS under the lab CA, the certificate served on port 443 is the one in the Secret lab-edge-tls (compared by fingerprint, so a machine that already trusts the lab CA gets the same result), http://verify.<ip-dashed>.nip.io/ redirects with 308 to https, which keeps the method and body, and https://verify.<ip-dashed>.nip.io/ answers 404 because no route claims that host yet.
  9. pods: every Pod in plexsphere runs with every container ready (a finished Job's Pod is not counted), and no lab Pod or Envoy proxy runs in another namespace.
  10. cluster scope: the namespace plexsphere carries pod-security.kubernetes.io/enforce, audit and warn at restricted and the marker plexsphere.io/lab=bootstrapped, the lab's only GatewayClass is lab, and no flux-system, cert-manager, envoy-gateway-system, external-secrets or crossplane-system namespace exists.
  11. postgres: the Cluster plexsphere-db is Ready, the Secret plexsphere-db-app carries uri, username and password, show max_connections in plexsphere-db-1 prints 200, the ScheduledBackup plexsphere-db-nightly exists, every Backup of the cluster is completed with each of its VolumeSnapshots readyToUse, and the last Job of the CronJob plexsphere-db-backup-prune, if any, succeeded.
  12. nats: the Pod nats-0 is Ready, and a Pod labelled app.kubernetes.io/part-of=plexsphere reads http://nats.plexsphere.svc:8222/healthz?js-enabled-only=true with 200.
  13. seaweedfs: the Deployment seaweedfs-all-in-one has one ready replica, s3.bucket.list lists plexsphere-action-output, and from a labelled Pod the bucket answers a request signed with the key pair of plexsphere-object-store with 200 and an unsigned one with 403.
  14. dex, over the public network: the discovery document at https://dex.<ip-dashed>.nip.io/.well-known/openid-configuration names that issuer, its jwks_uri serves at least one key, /auth for the client plexsphere answers 200 (the login page), and /auth/local for the same client and redirect URI leads to the password form with 200. The second request is the one that proves the client and its redirect URI are registered: Dex shows the login page to any client and checks the client once a connector is chosen, answering an unknown client with 404 and an unregistered redirect URI with 400. The GitHub login itself is not driven.
  15. dex in-cluster: a labelled Pod reads the same discovery document through the public hostname with the lab CA, the way the API reaches the issuer.
  16. network policies: a Pod without labels cannot connect to plexsphere-db-rw 5432, nats 4222, seaweedfs-all-in-one 8333 and dex 5556, and a labelled Pod connects to all four. A service that admits the unlabelled Pod fails with FAIL: network policies: <service> admits an unlabelled Pod on <port>.
  17. control plane: the Deployments plexsphere, plexsphere-console, spicedb and plexsphere-signer have one ready replica; the API container sets PLEXSPHERE_AUTH_TRUST_PROXY_HEADERS=true, PLEXSPHERE_SSE_NATS_URL, PLEXSPHERE_AUDIT_PEPPER_MOUNT=plexsphere and PLEXSPHERE_PROVISIONING_BROKER_CA_BUNDLE_FILE=/etc/plexsphere/provisioning-ca/ca.crt, PLEXSPHERE_CLOUD_CREDENTIALS_KV_MOUNT=cloud-credentials, PLEXSPHERE_CLOUD_CREDENTIALS_OPENBAO_K8S_ROLE=plexsphere-cloud-credentials, PLEXSPHERE_CLOUD_CREDENTIALS_DEFAULT_TTL=2160h and PLEXSPHERE_PROVISIONING_BROKER_CLOUD_CREDENTIALS_STORE=plexsphere-cloud-credentials, and neither PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER, PLEXSPHERE_CLOUD_CREDENTIALS_OPENBAO_TOKEN nor PLEXSPHERE_ENV, while the signer sets PLEXSPHERE_ENV=dev; the API's volume provisioning-ca reads the ConfigMap lab-ca-public, key response.body; the Job plexsphere-seed succeeded; and the HTTPRoutes plexsphere-api and plexsphere-console are Accepted with ResolvedRefs.
  18. public origin, over the public network: /v1/version on https://plexsphere.<ip-dashed>.nip.io names v and the chart version helmrelease/plexsphere last applied (its status.history, the newest deployed entry's chartVersion before the +), / serves the console as text/html, /downloads/plexd offers the plexd version /PLEXD_VERSION of the checkout pins, /v1/auth/idp-bindings?scope=platform lists exactly one binding on dex.<ip-dashed>.nip.io, /v1/projects without a session answers 401, and http://plexsphere.<ip-dashed>.nip.io/ redirects with 308 to https. A stopped API shows here as the Gateway's 503.
  19. streams: the BackendTrafficPolicy plexsphere-api is Accepted; a policy that is not prints its condition, and a missing one reports Accepted absent. The enrolled node's event stream is the live proof; see The first node.
  20. rate limiter: fifteen POST /v1/register in a burst from the host. The first ten answer 4xx other than 429, and a later one answers 429 with Retry-After: the auth surface admits a burst of ten per client and refills one request every three seconds. Run verify.sh again only half a minute after a previous run, or this check finds the host's budget spent. Repeating the burst from the enrolled node right after shows ten more admitted: the API keys on the client the proxy names, not on the proxy.
  21. control-plane network policies: a Pod without labels and a Pod labelled app.kubernetes.io/part-of=plexsphere both fail to connect to plexsphere 8080, plexsphere-console 80, spicedb 50051 and plexsphere-signer 8443. The API and the console admit the Gateway's proxy alone, which check 18 proves, and SpiceDB and the signer the Pods labelled app.kubernetes.io/name=plexsphere.
  22. management fleet substrate: what the management fleet's verify gate checks for every Project the cluster hosts, read from outside. The cluster serves apiextensions.crossplane.io, pkg.crossplane.io and external-secrets.io, the Deployments crossplane and external-secrets are Available in plexsphere, the API container sets PLEXSPHERE_MANAGEMENT_FLEET_CROSSPLANE_CONTROLLER_NAMESPACE and PLEXSPHERE_MANAGEMENT_FLEET_EXTERNAL_SECRETS_CONTROLLER_NAMESPACE to plexsphere, and the Functions function-patch-and-transform and function-auto-ready are Installed and Healthy. The fleet folds a failed gate into a flag and the API stays Ready, so this check is where a missing substrate shows. Before check 23, lab api: after a pause that refills the requests check 20 drained from the auth surface's rate limiter, five of which the two sign-ins spend, operator@example.com signs in at platform scope and in the Domain lab as The lab API session describes, with the password in LAB_STATE_DIR. A sign-in that fails is one FAIL: lab api: line, and checks 23 to 25 then skip what they would read through the API.
  23. cloud credentials: the ClusterSecretStore plexsphere-cloud-credentials is Ready; kubectl auth can-i as system:serviceaccount:plexsphere:plexsphere answers yes for create providerconfigs.openstack.m.crossplane.io and create externalsecrets.external-secrets.io in plexsphere-project-019a0816-0000-7000-8000-000000000001; that namespace exists; the Provider provider-openstack is Installed and Healthy; and, read through the API with the platform session and the lab CA of check 18, the Cloud LAB_OPENSTACK_CLOUD_SLUG names (lab-openstack unless set) carries an active credential of the same name whose expires_at is more than 7 days ahead. The OpenStack Cloud is optional, as in step 10c: without LAB_OPENSTACK_RC_FILE or all five LAB_OPENSTACK_* inputs the check keeps the store, the ExternalSecret right and the namespace, which Flux and the seed bring on their own, and prints skip: cloud credentials: no OpenStack Cloud is configured (…) for the ProviderConfig right, the Provider and the Cloud. Prove the credential authenticates goes one step further, by hand.
  24. openstack blueprint: the XRD xopenstackinstances.blueprints.plexsphere.com is Established; the Composition openstack-instance composes blueprints.plexsphere.com/<served> XOpenStackInstance, where <served> is the version the XRD serves; kubectl auth can-i as system:serviceaccount:plexsphere:plexsphere answers yes for create xopenstackinstances.blueprints.plexsphere.com in the Project namespace of check 23; and, read through the API with the domain session, the catalog of the Domain lab holds openstack-instance with the version <served>. Until the official catalog serves that version, the last line fails with the Domain lab's catalog holds no openstack-instance <served>; run bootstrap.sh once the official catalog serves it. An XRD that serves no version fails it with the XRD serves no version; the catalog cannot be matched against it instead.
  25. openstack landing zone: the same four reads as check 24 for the landing-zone blueprint: the XRD xopenstacklandingzones.blueprints.plexsphere.com is Established, the Composition openstack-landing-zone composes blueprints.plexsphere.com/<served> XOpenStackLandingZone, the API may create xopenstacklandingzones.blueprints.plexsphere.com in the Project namespace, and the catalog of the Domain lab holds openstack-landing-zone with the version <served>. Then the landing zone of step 10e, read through the API with the domain session: the Project openstack holds a Resource of the kind landing-zone that is not Deprovisioning or Deleted, printing ok: openstack landing zone: the landing zone <id> is Ready, and its composite pr-<provisioned resource id> reports the network it composed in status.networkName, printing ok: openstack landing zone: the composite pr-<id> composes the network <name>. A name with quotes, blanks or control characters is not printed: the line says composes a network named outside the pattern ^[A-Za-z0-9._-]+$ instead. A landing zone in another phase fails with the landing zone <id> is <phase>, not Ready, a Project without one with the Project openstack holds no landing zone; run bootstrap.sh with LAB_OPENSTACK_RC_FILE or the LAB_OPENSTACK_* inputs, and a composite without the name with the composite pr-<id> in plexsphere-project-019a0816-0000-7000-8000-000000000001 reports no network name. Without the OpenStack inputs the Resource half prints skip: openstack landing zone: no OpenStack Cloud is configured (…).

Checks 12, 13, 15 and 16 run throwaway Pods (lab-verify-<n>) of the image docker.io/alpine/k8s in plexsphere, and check 21 reads what the two Pods of check 16 found on the control plane's ports. The Pods run under the restricted Pod Security profile and without a ServiceAccount token; the lab CA and the S3 keys reach them on stdin. A Pod that admission refuses fails its check with kubectl's message. Every probe gives up after 10 seconds, and retries a connection refused while the network plugin is still admitting the new Pod.

The last line on stdout is the CA fingerprint.

Emptying the cluster ​

To build the lab again from nothing on the same cluster, empty the cluster first: the bootstrap refuses a cluster that carries one of the lab's CRD groups without the marked namespace (step 2), and a fresh OpenBao needs a fresh volume. deploy/lab/scripts/teardown.sh empties it:

bash
deploy/lab/scripts/teardown.sh --yes

It reads LAB_KUBE_CONTEXT, LAB_STATE_DIR and LAB_ENV_FILE the way the bootstrap does, and refuses to run without --yes: the run deletes every lab object, the volumes of OpenBao, Postgres, NATS and SeaweedFS, and the Postgres backups, whose VolumeSnapshots live in plexsphere and go with it. Everything Gardener installed stays: Cilium, the snapshot controller and the platform's CRDs. Every passed step prints one line on stdout, every message goes to stderr prefixed teardown:, and the first failing step exits 1. Running it again is safe: every step skips what is already gone, so a run that stopped halfway finishes on the next. An emptied cluster passes in a few seconds.

Before it deletes anything, the script checks two things:

  • The namespace plexsphere carries the marker plexsphere.io/lab=bootstrapped the bootstrap sets. A namespace without it belongs to another stack and stops the run. No namespace is a cluster an earlier run emptied as far as that; the run goes on with what is left outside it.
  • No provisioned Resource is left on the cluster. Every composite of a blueprint XRD is a Resource of a Project, and deleting it with kubectl would take its servers and networks unrecorded. The script deletes the Resources through the API instead, signed in as operator@example.com in the Domain lab through deploy/lab/scripts/lab-session.sh, with lab-ca.crt and dex-operator.password from the state directory and the address from the ConfigMap lab-cluster: the machines first, then the landing zones, each waited for until it reaches Deleted, for up to LAB_TEARDOWN_RESOURCE_TIMEOUT_SECONDS (default 900). Without the state directory's files or the ConfigMap it stops and asks for plexctl resource delete by hand, as Provision an OpenStack machine and The OpenStack landing zone describe.

Decide about the state directory before the run. Keep it to rebuild with the same CA, deploy key, unseal key and credentials, as Rebuild and recovery describes for a lost OpenBao volume. Move it aside to start over: the bootstrap then generates everything anew, prints a new deploy key to register, and the old deploy key can be removed from the repository. The application credential an rc file created stays in Keystone; when the state directory goes, delete it too (openstack application credential delete plexsphere-lab), or the next bootstrap stops before it creates one of the same name.

Then the script empties the cluster in this order, which deletes every custom resource while its controller still runs to serve its finalizer:

  1. It stops Flux: it annotates the FluxInstance with fluxcd.controlplane.io/reconcile=disabled, which keeps the Flux Operator from reverting what follows, and suspends every Kustomization and HelmRelease.
  2. It deletes the custom resources of the lab's controllers: the XRDs, Compositions, Functions, the Provider, the DeploymentRuntimeConfig, the activation policy and the lock of Crossplane; the HTTPRoutes, the BackendTrafficPolicy, the Gateway, its class and the EnvoyProxy; the Postgres backups and the cluster plexsphere-db, not waited for, because its Pod waits for the API's connections, which the next step closes; the ExternalSecrets, the SecretStore and the ClusterSecretStore; the Certificates and Issuers. It then waits for Crossplane's package revisions to go, while the package manager still serves their finalizers.
  3. It uninstalls the releases, the control plane first and the controllers last, and deletes every volume left in the namespace.
  4. It deletes the Flux objects, which prune and uninstall nothing while suspended, the FluxInstance, the Flux Operator's release and the namespace plexsphere.
  5. It deletes the namespaces plexsphere-project-<project id> the management fleet created, one per Project it placed on the cluster.
  6. It deletes what the charts and Crossplane leave outside the namespace: the webhook configurations whose Services went with the namespace, and the cluster RBAC of Flux and of Envoy Gateway's certificate job.
  7. It deletes the CRDs of the lab's groups: cert-manager, External Secrets, Flux and the Flux Operator, CloudNativePG, Crossplane and its providers, Envoy Gateway, the Gateway API and blueprints.plexsphere.com, printed to stderr before the delete. An instance that still holds its CRD at this point has a finalizer nobody serves any more, because every controller went in step 3; the script removes the finalizer, says so on stderr, and deletes the CRD again.
  8. It checks that nothing is left: no CRD of the lab's groups, no ClusterRole, ClusterRoleBinding or webhook configuration naming plexsphere, crossplane, cert-manager, cnpg, external-secrets, envoy-gateway or flux, and no lab namespace. Anything left stops the run with the list.

The last lines name what stays: the state directory, lab.env, the deploy key registered on GitHub and the application credential in Keystone. The public address does not: the cloud controller deletes the load balancer with the Gateway's Service and releases its floating IP, which the step says in its address: line. The access file access.md goes with it, since its URLs name that address: the step deletes it from the state directory and says so in a removed: line; the passwords it quoted stay in their files. The next bootstrap reads the address the cloud assigns to the new Gateway, prints the new hostnames and the OAuth App callback to register, writes access.md again, and every machine that trusts the lab CA keeps trusting it under the new names.

Rebuild and recovery ​

  • Same cluster, OpenBao volume intact: run the bootstrap again. It skips every object that exists, never overwrites lab/ca, rewrites lab/registry when the ghcr.io variables are set, and replays every wait. Of the CA it needs only lab-ca.crt in the state directory. Deleting the ConfigMap lab-cluster or lab-ca-public is repaired the same way.

  • A new cluster: restore the state directory from the password manager first. The bootstrap reuses its deploy key, unseal key and CA, so the trust every machine already has survives. The new Gateway's address changes the hostnames but not the CA; access.md is written anew with them.

  • A lost OpenBao volume: restore lab-ca.key into the state directory, then run the bootstrap again. It initialises the fresh server, overwrites openbao-init.json with the new recovery keys and root token, and replays every seed: the CA, the S3 key pair and the Dex password hashes from the state directory, the registry credential and the GitHub OAuth App from their variables, which this run needs set. The Secret lab-ca ends up with the same certificate as before. Without lab-ca.key the run stops before it seeds lab/ca.

  • A lost openstack-application-credential.json: the secret cannot be read from Keystone again. Delete the credential there (openstack application credential delete plexsphere-lab) and re-run with the rc file, which creates a new one. Step 10c then revokes the Cloud's credential issued from the old one and issues a new one.

  • A lost openbao-init.json on an initialised server: set LAB_OPENBAO_ROOT_TOKEN; the bootstrap stops before any OpenBao write without one.

  • Lost credential files: restore s3-access-key, s3-secret-key, dex-operator.password, dex-member.password, spicedb-preshared-key, signer-seed, api-keys.json and the signer-* files but signer-ca.key from the password manager before a re-run. A run that finds one missing while OpenBao holds its key stops before any OpenBao write rather than replace a credential the workloads use.

  • The control plane's material is replayed from the state directory like the CA, so a rebuilt lab serves the same signing key, the same cursor and NSK keys and the same signer chain, and the nodes it knew keep trusting it. To rotate the signer's chain, delete its six files signer-ca.crt, signer-ca.key, signer-server.crt, signer-server.key, signer-client.crt and signer-client.key, and lab/signer-mtls (bao kv metadata delete -mount=lab signer-mtls as root in openbao-0), then re-run the bootstrap. Keep signer-seed, which signer-* matches too: it is the signing key, not part of the chain, and a run that finds it missing while OpenBao holds lab/signer stops.

  • A lost Postgres volume restored from a snapshot, with all three references repointed as Backups describes, keeps the Domain lab, its binding and the enrolled nodes. The bootstrap creates the seed Job only while job/plexsphere-seed is absent; delete the Job to have a re-run seed again.

  • A lost Postgres volume: restore the newest snapshot as Backups describes, or rebuild the control plane's data and enrol the lab's nodes again.

  • A lost NATS volume loses the replay window and nothing else; the API creates its streams again.

  • A lost SeaweedFS volume loses the action output. The bucket comes back with the chart's hook on the next upgrade of the HelmRelease; to run that upgrade now, force it, because an unchanged release is otherwise left alone:

    bash
    flux reconcile helmrelease seaweedfs -n plexsphere --force

What lives outside plexsphere ​

Every namespaced object of the lab, the controllers and the Gateway's proxy included, lives in plexsphere. Outside it:

  • The GatewayClass lab, which is cluster-scoped by the Gateway API.
  • The DeploymentRuntimeConfig default and the two Functions, which are cluster-scoped by Crossplane. A workspace gate allows these two kinds and keeps them in deploy/lab/crossplane-packages/.
  • The CRDs the charts, Crossplane and the Flux Operator install, among them CloudNativePG's postgresql.cnpg.io and Crossplane's apiextensions.crossplane.io and pkg.crossplane.io, which follow their chart versions.
  • The namespaces plexsphere-project-<project-id> the management fleet creates, one per Project it places on the cluster, each with the Role, RoleBinding, ServiceAccount and ResourceQuota the fleet reconciles and the composites the broker renders. They are the one deliberate exception to the single namespace: the per-Project namespace is the boundary the fleet scopes a Project's provisioning to. None exists until an operator assigns a Project.
  • The cluster RBAC of the charts: ClusterRoles and bindings of the controllers, Crossplane's included, OpenBao's system:auth-delegator binding for the Kubernetes auth method, and the admission webhooks and policies the charts render. The Dex and SeaweedFS charts' own cluster RBAC is switched off: neither service calls the Kubernetes API. The control plane chart renders the ClusterRole and ClusterRoleBinding plexsphere-api-plexsphere for the API's ServiceAccount.

Everything the lab pulls is on TCP 443, and the OpenStack's API ports are open from the cluster as well; the lab adds nothing outside the namespace for either.

Troubleshooting ​

SymptomCauseFix
GitRepository lab not Ready, SSH authentication failureThe deploy key is not registered, or the URL's host and port differ from the known_hosts entry in Secret lab-git (ssh.github.com:443).Register lab-git.pub as a read-only deploy key; keep the URL on ssh.github.com:443.
A re-run stops with the cluster already carries fluxcd.controlplane.io although the lab bootstrapped the clusterThe namespace lost the marker plexsphere.io/lab=bootstrapped: the Namespace patch in deploy/lab/flux/fluxinstance.yaml is missing, or the Flux Operator has not applied it yet.Restore the patch; kubectl label namespace plexsphere plexsphere.io/lab=bootstrapped lets the run pass meanwhile.
Bootstrap step 8 or verify.sh check 10 reports the namespace without enforce=restrictedThe same cause: the Flux Operator owns the namespace and applied it without the lab's labels. The namespace enforces no Pod Security profile until they are back.The same fix; kubectl label namespace plexsphere pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/audit=restricted covers the gap.
Gateway lab Programmed: False with AddressNotAssignedThe cloud controller has not created the load balancer for the Gateway's Service yet (two to three minutes), or could not: a quota of load balancers or floating IPs in the OpenStack project, or no external network to allocate from.kubectl -n plexsphere describe services -l gateway.envoyproxy.io/owning-gateway-name=lab shows the controller's events; bootstrap step 9b waits LAB_ADDRESS_TIMEOUT_SECONDS for the address.
SecretStore lab not Ready with InvalidProviderConfig or ValidationFailedOpenBao is sealed or uninitialised, the role lab-external-secrets is missing, or its audience differs from the store's.Run the bootstrap; it initialises OpenBao and rewrites the role with audience vault.
Certificate lab-edge-tls not Ready with IncorrectIssuer or PendingThe Secret lab-ca is missing or holds no CA.Check the ExternalSecret lab-ca and lab/ca in OpenBao; the bootstrap seeds it from lab-ca.crt and lab-ca.key.
A Kustomization Ready: False with no matches for kindA custom resource landed in a Kustomization that applies before its CRD exists.Restore the dependency order of deploy/lab/kustomizations.yaml; the workspace gate names the missing dependency.
A Kustomization Ready: False naming the ConfigMap lab-clusterThe bootstrap has not written the ConfigMap, or it was deleted.Run the bootstrap.
A FailedCreate event naming a restricted Pod Security ruleA chart's Pod violates the namespace's restricted profile.Set that chart's values in its HelmRelease to satisfy the rule; do not lower the namespace's enforce label.
The Cluster plexsphere-db or a Backup reports a VolumeSnapshotClass that does not existThe class in the ConfigMap lab-cluster is not on the cluster. The operator checks it when a backup runs, not at install.Set LAB_VOLUME_SNAPSHOT_CLASS to an existing class, or mark one default, and run the bootstrap again.
A Backup in phase failed although the class existsThe VolumeSnapshotClass belongs to another CSI driver than the PVC plexsphere-db-1.Pick the class of the PVC's driver with LAB_VOLUME_SNAPSHOT_CLASS and run the bootstrap again.
The Dex Pod crash-loops with no password hash providedThe Secret dex-credentials is missing or lacks a hash.Check the ExternalSecret dex-credentials and lab/dex in OpenBao; the bootstrap writes it.
A GitHub sign-in ends with Dex reporting the user is not in an allowed organisationThe user is not a member of plexsphere, or has not granted the OAuth App access to the organisation.Join the organisation, or grant the app access on GitHub's authorized OAuth Apps page.
The hook Job seaweedfs-bucket-hook fails, or seaweedfs-all-in-one crash-loopsThe identities file in seaweedfs-s3-config is malformed, which stops the server, or does not name the key pair, which answers the hook with 403.Read kubectl logs -n plexsphere job/seaweedfs-bucket-hook and the server's log; check the ExternalSecret seaweedfs-s3-config and lab/s3.
A workload times out on 5432, 4222, 8333 or 5556Its Pods lack the label app.kubernetes.io/part-of: plexsphere the NetworkPolicies admit.Label the workload's Pods; the control plane charts do so themselves.
verify.sh check 15 fails while check 14 passesThe cluster does not route its own LoadBalancer address back to the Envoy proxy, so a Pod cannot reach the issuer through the public hostname, and the API cannot verify a sign-in.No chart value adds hostAliases. Add a spec.postRenderers kustomize patch to the HelmRelease plexsphere that gives the Deployment plexsphere a hostAliases entry pointing dex.<ip-dashed>.nip.io at the ClusterIP of the proxy's Service (kubectl -n plexsphere get services -l gateway.envoyproxy.io/owning-gateway-name=lab).
helmrelease/plexsphere not Ready, the Job plexsphere-migrate failedA migration fails against the lab's schema.Read kubectl -n plexsphere logs job/plexsphere-migrate within ten minutes of the attempt. A schema error is fixed forward on main; the release retries every five minutes and never rolls back.
The API crash-loops with could not reach the OpenBao audit pepper mountThe mount plexsphere or the role plexsphere-audit-pepper is missing, the role requires another audience than vault, or OpenBao is sealed.Run the bootstrap; step 9 writes both.
The API crash-loops with could not reach the OpenBao cloud credential storeThe role plexsphere-cloud-credentials is missing, requires another audience than vault, or OpenBao is sealed.Run the bootstrap; step 9 writes the mount, both policies and both roles.
The bootstrap stops with cannot sign in as operator@example.com at platform scope: the IdP refused the password …dex-operator.password in the state directory is not the password Dex holds: a state directory of another lab, or one restored from an older copy.Restore the file from the password manager, or point LAB_STATE_DIR at the lab's state directory, and re-run.
Step 10c stops with Keystone refused the application credential <id>The application credential was deleted or expired in Keystone.With an rc file, move openstack-application-credential.json out of the state directory and re-run, which creates a new one; with the five inputs, create a new credential and put it into lab.env.
Step 10c stops with <user> already holds an application credential named plexsphere-lab, whose secret … lacksA previous lab created the credential and its state directory is gone.Delete it with openstack application credential delete plexsphere-lab, or set LAB_OPENSTACK_APPLICATION_CREDENTIAL_NAME to another name, and re-run.

| Step 10c stops with answered 403 after a minute | The token's user lacks the right: the token was not minted from the platform session, or the authz outbox has not delivered the Cloud's or the credential's tuples. | Mint the token with the Platform Operator's identity_ref; check the API log for the authz outbox, then re-run. | | Step 10c stops at POST /v1/clouds/<id>/cloud-assignments | The Project openstack does not exist: the seed Job ran before the manifest named it. | kubectl -n plexsphere delete job plexsphere-seed and re-run the bootstrap. | | The ClusterSecretStore plexsphere-cloud-credentials, or an ExternalSecret cloud-credentials-<hash> in a Project namespace, is not Ready | The role lab-cloud-credentials is missing or its policy does not read cloud-credentials/data/clouds/*; the namespace lacks one of the two Project labels the store's conditions require; or the credential's KV entry has no payload. | Run the bootstrap for the role; kubectl get namespace <ns> --show-labels for the labels; kubectl -n <ns> describe externalsecret <name> names the failing read. | | The API logs certificate signed by unknown authority for the issuer | The API container does not trust the lab CA: the lab-ca volume or SSL_CERT_DIR is missing. | Check api.extraVolumes, api.extraVolumeMounts and api.extraEnv of the HelmRelease plexsphere, and the ConfigMap lab-ca-public. | | A sign-in ends at Dex with the redirect_uri rejected | The Dex client's redirect URI differs from <externalUrl>/v1/auth/callback. | Keep externalUrl of the HelmRelease plexsphere and the client of the HelmRelease dex on the same hostname. | | helmrelease/crossplane not Ready, a FailedCreate event on the ReplicaSet of crossplane naming a restricted Pod Security rule | The chart's defaults, without the Pod and container security contexts of the HelmRelease crossplane. | Keep podSecurityContextCrossplane, securityContextCrossplane and the two RBAC manager counterparts in the HelmRelease; a workspace gate holds them. | | A Function stays Healthy: False, its Deployment shows FailedCreate naming capabilities or seccompProfile | The package runs without the runtime config default, or the config lost a field. | Check kubectl get deploymentruntimeconfig default -o yaml against deploy/lab/crossplane-packages/deploymentruntimeconfig.yaml; the Deployment re-renders once the config is restored. | | verify.sh check 22 reports the API lacks a PLEXSPHERE_MANAGEMENT_FLEET_* namespace | The HelmRelease plexsphere lost managementFleet, so the API looks for the controllers in crossplane-system and external-secrets and every placed Project stays unverified. | Restore managementFleet.crossplaneNamespace and managementFleet.externalSecretsNamespace at plexsphere. | | The signer crash-loops with --key-provider=software is refused | The signer runs without PLEXSPHERE_ENV=dev. | Keep devMode.enabled: true in the HelmRelease plexsphere-signer. | | plexd logs a TLS verification error and does not register | The node does not trust the lab CA. | Add the lab CA to the host's store, as Trusting the lab CA describes. | | A Kustomization reports Ready: False with failed early due to stalled resources naming an object that is healthy by now | Flux checked the object during a short-lived failure, such as cert-manager's first request for lab-edge-tls or the Gateway before its address. | Nothing: Flux retries after the Kustomization's one-minute retryInterval. flux reconcile kustomization <name> -n plexsphere retries at once. | | The node's event stream reconnects every 15 seconds, and the API logs sse: stream opened for it each time | The Gateway's request timeout cuts the stream: the BackendTrafficPolicy plexsphere-api is missing or not Accepted. | kubectl -n plexsphere describe backendtrafficpolicy plexsphere-api; Flux restores a deleted policy within ten minutes. | | A provisioned resource stays in Pending, and the API logs provisioning broker: substrate not ready; skipping resource until its XRD is registered | The cluster does not serve the composite's kind: the bootstrap has not installed the XRD (step 10d skipped or stopped), or the resource names a blueprint version the lab does not pin. | kubectl get compositeresourcedefinition <name>; re-run the bootstrap, and provision only from the version step 10d printed. | | An InstanceV2 in the Project namespace reports Synced=False naming an image, a flavor or a network it cannot find | The parameters name an object the OpenStack project does not see, or misspell it. | Check the names with openstack image list, openstack flavor list and openstack network list; delete the resource and create it with the right parameters. | | A provisioned resource stays in Enrolling, and openstack console log show <server> shows a failed fetch of /downloads/plexd or a certificate error | The machine cannot reach the lab on TCP 443, the network has no router to the external network, or the machine does not trust the lab CA. | Give the network a router to the external network; check that the cloud-init output lists the lab CA under ca_certs, and that provisioning.caBundle of the HelmRelease plexsphere names it. |