Dev stack runbook

This runbook brings up the full local development stack — every dependency the plexsphere API and the plexd sibling product talk to in production — on a single-node kind cluster, seeds a deterministic demo tenant, authenticates plexctl against the seeded Dex IdP, runs the end-to-end golden flow against the dev cluster, and is the reference for the manifest-to-CI mapping, the SpiceDB wiring, the canonical troubleshooting matrix, and the reset recipe. It is self-contained — every step from git clone to a green golden-flow run is below.

Reach for this runbook when you need the full stack rather than the lean cluster the Set up your local plexsphere tutorial brings up:

• You are onboarding to the repository and want the shortest path from git clone to a usable /v1 API and CLI.
• You are reproducing a CI failure on a feature branch without re-creating the cluster on every test iteration.
• You are triaging a dev-stack-smoke red run and want to map the failing job back to a single make target on your laptop.

See also:

• plexsphere local dev-stack reference — the per-dependency reference (one section per service, image pin, dev-mode flags, prod-delta). Open it whenever a Pod is unhealthy and you need to know what flags it boots with and how that diverges from production.
• CI pipeline — jobs, triggers, and reproducing failures — the per-job CI guide, including the dev-smoke schedule + label gate and the SHA-pinned third-party actions each job depends on.
• Testing — Pyramid, Build Tags, and the Shared Harness — the test-pyramid guide; the integration tier under tests/integration/dev_* and the E2E tier under tests/e2e/dev/ both exercise the same manifests this guide applies.
• How to seed tenancy Domains with plexsphere-bootstrap — the operator runbook for seeding tenancy Domains via the plexsphere-bootstrap Job.
• How to exercise the /v1/auth/* surface, How to exercise the /v1/admin/* surface, How to exercise the /v1/label-* surface, and How to exercise the /v1/domains/{id}/identities surface — per-surface walkthroughs of the four /v1/{auth,admin,label-*, domains/{id}/identities} HTTP families now wired against the production binary.

Prerequisites

The dev stack runs entirely on the contributor's laptop — no cloud credentials, no external services. The four tools below must be on $PATH before make dev is invoked. Pinned versions match the hosted runner image the CI dev-stack-smoke job uses; a newer patch release will almost always work, but the listed version is the version the test suite is proven against.

Tool	Pinned version	Purpose
Go	1.26.6	Pinned by .go-version; drives make docker-build and make e2e.
Docker	>= 24.0	Container runtime for the kind node and every dev-stack image.
kind	v0.26.0 (action helm/kind-action@v1.14.0)	Runs the local single-node Kubernetes cluster shared by all dev targets.
kubectl	>= v1.32.0	Applies the dev overlay and waits on Deployment + Job readiness.

The dev overlay layers seven additional dependencies on top of the narrower user-facing core (API + Postgres / NATS / Dex / SpiceDB): OpenBao (secrets), SeaweedFS (object store), Mimir (metrics), Loki (logs), Crossplane (control plane), External Secrets Operator (secret sync), and plexd (the bootstrap-token registration agent). Every image is SHA-pinned under deploy/local/base/<svc>/ — see plexsphere local dev-stack reference for the per-dependency reference.

The plexd image tag is the version recorded in the repo-root PLEXD_VERSION file. Bumping plexd is a single-line edit; the dev-stack drift gate in tests/workspace/plexd_version_test.go enforces the pin.

Pinned container images applied by the lean overlay (all SHA-referenced in deploy/local/base/*/):

• postgres:16-alpine — core + spicedb logical databases.
• nats:2.10.20 — messaging plane with JetStream enabled.
• ghcr.io/dexidp/dex:v2.41.1 — OIDC provider with the static plexsphere-test client and admin@example.com static user; the constants mirror internal/platform/testutil/containers/dex.go so local behaviour matches integration tests.
• authzed/spicedb:v1.39.0 — ReBAC authorisation engine, gRPC-only on port 50051, Postgres datastore shared with plexsphere, preshared-key auth (token test-key). Pin mirrors internal/platform/testutil/containers/common.go::SpiceDBImage.

The plexsphere images (plexsphere:dev, plexsphere-migrate:dev) are built locally by make docker-build and side-loaded into the kind node — no registry push is required.

Bring up the dev stack

A single command builds every plexsphere image, creates the kind cluster, applies the dev overlay (lean core + the six currently-wired dev-stack dependencies), waits for every Deployment to become Ready, and seeds the demo tenant:

make dev

make dev is the inner-loop entrypoint and is a one-shot bring-up with no separate seed step. It provisions the kind cluster, builds the plexsphere images and side-loads them, preloads the SHA-pinned upstream dependency images through the host Docker cache (scripts/dev-preload-upstream-images.sh — the cache survives make dev-down, so a cluster re-create does not re-download them), hydrates Gateway API + Envoy Gateway, applies the deploy/local/overlays/dev overlay, and runs the plexsphere-bootstrap Job. The Job is the single seed surface: it plants the three demo Domains (Acme Corp, Beta LLC, Gamma Cooperative), binds each one to the in-cluster Dex, pre-creates the canonical admin@example.com user under every Domain keyed by the Dex sub claim, and writes the admin + auditor ReBAC tuples on each Domain for that user.

Independently of the bootstrap Job, the main binary's boot-time seeds also plant a platform-shared IdP binding against the same Dex fixture (from the PLEXSPHERE_PLATFORM_IDP_* keys on the plexsphere-config ConfigMap), so the Console's "Sign in as a Platform Operator" path — a Domain-independent login with no Domain selected — works out of the box. See Sign in as a Platform Operator.

Note — plexd. The dev overlay renders the plexd Deployment, and make dev puts its image on the kind node before the apply: make docker-build-plexd pulls the release pinned by /PLEXD_VERSION from ghcr.io/plexsphere/plexd and retags it locally, make kind-load-plexd side-loads that tag. Without those two steps the Pod wedges in ErrImageNeverPull, because it runs under imagePullPolicy: Never. make dev-up-plexd re-runs the pair, re-arms the seeded bootstrap token, and rolls the Deployment. See plexsphere local dev-stack reference for the image pipeline, the Pod's environment, and the registration prerequisites.

The target is idempotent at every layer. Re-running make dev against an existing plexsphere cluster re-applies the overlay, re-waits for readiness, and re-runs the bootstrap Job. The bootstrap binary's Domain pass treats a slug-conflict re-apply as a no-op (domain-skipped reason=already-exists), the IdP-binding pass takes the partial-unique-index conflict branch on duplicate (domain_id, issuer) pairs, and the admin-seed pass loads the existing User row via UNIQUE (domain_id, external_subject) and re-touches the same ReBAC tuples through SpiceDB's OPERATION_TOUCH semantics — so a re-run is a deterministic no-op even on a fully populated database.

Startup ordering

The dev overlay applies every manifest in one kubectl apply -k pass, so the API and signer Deployments — and the bootstrap Job — are created at the same moment as Postgres, NATS, SpiceDB, and Dex. To keep the plexsphere-built workloads from CrashLoopBackOff-ing against a not-yet-ready dependency, each one gates its boot with initContainers that hold the main container until the services it dials are online:

• plexsphere runs wait-for-deps — a TCP probe of NATS, SpiceDB, and Dex — followed by wait-for-migrate, which waits for Postgres and for the migrate Job to have applied every schema migration.
• plexsphere-signer runs wait-for-migrate: it reads the signing-key tables, so it must not boot against an unmigrated schema.
• plexsphere-bootstrap runs wait-for-deps (a TCP probe of SpiceDB) followed by wait-for-migrate (Postgres and the migrate Job). The bootstrap binary writes Domains and IdP bindings through Postgres and ReBAC tuples through SpiceDB, so it must not run until both are reachable.

A reachable spicedb:50051 proves the SpiceDB process is listening, not that the ReBAC schema is installed. The schema is installed via a gRPC WriteSchema call, which both the plexsphere API (on boot) and the bootstrap binary perform. Because the dev overlay imposes no ordering between the API Deployment and the bootstrap Job, the bootstrap binary applies the schema itself before its admin-seed pass writes any tuple — the apply is idempotent, so whichever workload reaches SpiceDB first installs the schema and the other short-circuits. Without that self-applied schema the Job would race the API and exit with object definition "domain" not found.

A Pod sitting at Init:0/1 (or Init:0/2) in kubectl get pods is therefore waiting on a dependency, not failing — kubectl logs <pod> -c wait-for-deps (or -c wait-for-migrate) prints which endpoint it is still polling.

plexd is the one data-dependent workload with no such gate. It registers against the Project, Resource, and bootstrap-token rows the bootstrap Job seeds, and it carries only the dev-warning init container — a Pod that starts before the seed lands exits and is restarted with backoff until the rows exist. make dev compensates by awaiting the bootstrap Job's Complete condition before it waits for deployment --all, so the seed is committed before plexd's Available condition is demanded.

Every Pod under deploy/local/base/ whose main container exists because the dev stack relaxes a production posture announces that relaxation at boot — exactly one level=WARN line with attributes env=dev, dependency=<svc>, prod_delta=<sentence>. Tail the log of any Pod to confirm the warning is emitted:

kubectl -n default logs deployment/openbao -c dev-warning

The dev overlay applies into the default namespace (make dev invokes kubectl apply -k deploy/local/overlays/dev without an -n flag), so log queries for the bundled dependencies target default rather than the plexsphere namespace some older dev prose still references.

plexsphere API env-var contract

The plexsphere API container reads a fixed set of PLEXSPHERE_* env vars at boot — without them the corresponding /v1/* surface short- circuits to a 501 Not Implemented stub. The dev overlay wires every required var (Secret plexsphere-cursor-keys, ConfigMap plexsphere-config, and the dev overlay's JSON6902 patches). The full table — env var, dev source, factory file, /v1/* surface required by, and the prod-delta rotation contract — lives in the plexsphere section of the local dev-stack reference.

After make dev reports success, verify the container exposes every required env var:

kubectl exec deployment/plexsphere -c plexsphere -- env | grep PLEXSPHERE_

The drift gate tests/workspace/dev_overlay_factory_env_drift_test.go walks every productionXxxConfigFromEnv FuncDecl in cmd/plexsphere/ and fails closed if a new factory adds an env reader without a corresponding manifest wiring — running go test ./tests/workspace/... from the worktree root catches the gap before it reaches CI.

Seeded Domain and binding IDs

A fresh make dev plants three demo Domains — Acme Corp, Beta LLC, Gamma Cooperative — and one active IdP binding each, all keyed to the in-cluster Dex. Their UUIDs are deterministic fixtures the bootstrap Job writes to Postgres; plexctl login and every --domain-scoped call consume them.

Read the seeded slugs and Domain IDs straight from the dev Postgres StatefulSet:

kubectl exec statefulset/postgres -- \
  env PGPASSWORD=plexsphere psql -U plexsphere -d plexsphere -tAc \
  "SELECT slug, id FROM plexsphere.domains ORDER BY slug"

Expected output lists the three slugs acme-corp, beta-llc, gamma-coop in ascending order, each with its domain_id. A second make dev run re-seeds idempotently — the Domain pass takes the already-exists no-op branch and the IdP-binding pass takes the partial-unique-index conflict branch — so the IDs are stable across re-runs.

Each demo Domain's single binding is seeded as the Domain primary and aliased dex (the svc-demo binding is aliased oidc-cc). That means plexctl login --domain-id "$DOMAIN_ID" resolves the primary without a binding id, and --idp-binding dex resolves the same binding by name. The dashboard is the same: its sign-in chooser fetches the Domain's providers from GET /v1/auth/idp-bindings and either proceeds straight through (one binding, or a primary) or lets you pick one. So no IdP binding id ever has to be read out of Postgres by hand to sign in — in dev or in production.

Sign-in with the seeded Dex test user

The bootstrap manifest at deploy/local/base/plexsphere-bootstrap/configmap.yaml plants three demo Domains, binds each one to the in-cluster Dex IdP, AND pre-creates the canonical admin@example.com user under every Domain with admin + auditor ReBAC tuples — so a real Dex-driven sign-in adopts the seed user via UNIQUE (domain_id, external_subject) and inherits its permissions. No follow-up seed step is required.

The same seeded Dex IdP backs the CLI. Authenticate with the RFC 8628 device-code flow, passing just the Domain UUID you read from Postgres above — plexctl login resolves the Domain's single active binding server-side. Because that binding is the Domain primary and aliased dex, you can also pin it by name:

plexctl login --domain-id "$DOMAIN_ID"                  # resolves the primary
plexctl login --domain-id "$DOMAIN_ID" --idp-binding dex  # resolves by alias

plexctl login prints a verification URL and a user code, then polls until the device code is approved. The seeded static-user credentials are admin@example.com / password — the canonical fixture mirrored across the integration suite, with the bcrypt hash in deploy/local/base/dex/configmap.yaml matching dexPasswordHash in internal/platform/testutil/containers/dex.go. The Dex internal userID (08a8684b-db88-4b73-90a9-3cd1661f5466) drives the canonical sub claim the bootstrap manifest's domain_admins[].external_subject keys the seeded user on; bumping the Dex userID is the only change that requires updating the manifest.

Open the printed /v1/device?user_code=… link in a browser to complete the approval: the control plane serves the verification page itself, so this stack needs no external dashboard. Sign in through Dex with the static-user credentials above and click Approve; the waiting plexctl poll then receives its token. See Cookie sessions — plexsphere_session contract on every /v1/* route for the per-request session contract the /v1/* routes enforce.

The four operator how-to guides (auth, admin, label-*, identities) all target the three bootstrap Domains — the same seed admin's tuples make every surface respond 200 without an extra grant step.

Sign in with plexctl

The in-tree web dashboard has been removed: the dev overlay no longer renders a web Deployment, the kind config no longer binds a host port for it, and there is no SPA to open on this local stack. The kind config in deploy/local/kind.yaml binds the host ports the surviving workloads use — 8080 (API) and 5556 (Dex).

Drive the platform through the CLI and the /v1 API instead. The Drive the API with plexctl section below walks the same surfaces a dashboard would have shown — Domains, Projects, Identities, Audit, Groups, Labels — gated by the same per-Domain permissions.

The per-request authentication middleware recognises the plexsphere_session cookie (or a bearer token) on every /v1/* route. See Cookie sessions — plexsphere_session contract on every /v1/* route for the session contract — cookie name, scope, lifetime, SameSite posture, and the bypass list the middleware short-circuits on for unauthenticated probes (/v1/health, /v1/version, /v1/openapi.json, /v1/auth/, /v1/register, /v1/docs).

Two node surfaces are wired against the API:

• Registered nodes — make dev schedules the plexd Pod, which registers itself via POST /v1/register on startup, so plexctl node list shows the registered node against a freshly-seeded dev cluster.
• Reachability — when plexd dies (Pod Terminating), the node transitions to unreachable within the configured TTL, observable through the same /v1 node surface.

If a /v1 call hangs or fails the most common cause is a missing or failing dependency Pod — kubectl get pods should show every workload Running. The dev overlay applies into the default namespace.

Drive the API with plexctl

The seeded Dex IdP backs the CLI. make dev does not compile the operator CLI itself — build it now, prepend $PWD/bin to $PATH for the rest of the shell session, and read the one UUID plexctl login needs out of the dev Postgres StatefulSet:

make plexctl-build
export PATH="$PWD/bin:$PATH"
export PLEXSPHERE_URL=http://localhost:8080
plexctl --version

DOMAIN_ID=$(kubectl exec statefulset/postgres -- \
  env PGPASSWORD=plexsphere psql -U plexsphere -d plexsphere -tAc \
  "SELECT id FROM plexsphere.domains WHERE slug='acme-corp'")

plexctl login --domain-id "$DOMAIN_ID"

plexctl login resolves the Domain's single active binding server-side, so no IdP binding id has to be read from Postgres.

plexctl login prints a verification URL and a user code, then polls until the device code is approved. Open the printed /v1/device?user_code=… link in a browser to complete the approval — the control plane serves the verification page itself; sign in through Dex and click Approve to release the waiting plexctl poll.

For a permanent plexctl install on $PATH (sudo / ~/.local/bin) see How to install the plexctl CLI. For Domains with zero or multiple active IdP bindings see How to complete your first plexctl login.

Drive the API once authenticated:

plexctl whoami
plexctl domain list
plexctl project list       --domain "$DOMAIN_ID"
plexctl identity list      --domain "$DOMAIN_ID"
plexctl audit entries list --domain "$DOMAIN_ID"
plexctl group list         --domain "$DOMAIN_ID"

Expected whoami output identifies the seed admin under the picked Domain:

PRINCIPAL  SCOPE   SUBJECT                               DOMAIN                                ACR  AMR
user       domain  <seed-user-uuid>                      <acme-corp-uuid>                      -    -

Every list call returns 200 — the bootstrap-seeded admin carries both admin and auditor ReBAC tuples on the addressed Domain.

For JSON output, pass --output json (long form only — plexctl does NOT bind it to -o):

plexctl project list --domain "$DOMAIN_ID" --output json

The result shape is { "items": [...] }. The full subcommand catalogue with flags, exit codes, and JSON output shapes lives in plexctl — the plexsphere CLI.

Seed a full demo environment with make dev-demo

The steps above bring up the stack and drive the API by hand. When you just want a ready-to-explore environment — a couple of Projects, the provisioning catalog, and a real cloud Resource running on the local floci emulator — make dev-demo does the whole provisioning tutorial for you in one command:

make dev-demo

It brings the stack up (make dev is idempotent, so an already-running cluster is reused), builds plexctl, and then replays the provisioning-track lessons end to end: it creates the build-demo and provision-demo Projects, imports the aws-s3-bucket Blueprint, creates the demo AWS Cloud and its vaulted Credential, runs the two-party credential assignment (requested by the project owner, approved by the platform operator), and finally provisions an S3 bucket that the broker carries to Ready against floci. The bucket is left running.

The flow runs without a browser. plexctl login is device-code only, so make dev-demo never uses it; instead it mints two short-lived plexsphere API tokens straight against the dev Postgres — one for the seeded tenant admin (acme-corp / admin@example.com), one for the seeded platform operator (platform-operators / operator@example.com) — using the mint-token operation of the identity fixture binary that make dev already side-loads into the kind node. Each token is a real, server-validated bearer; the script feeds the matching one to every plexctl command through the PLEXSPHERE_TOKEN environment variable. The two identities are load-bearing: the credential assignment cannot be self-approved, so the requester and approver must differ.

Two things to expect on a first run: importing the Blueprint pulls the catalog image from ghcr, which can be slow on a throttled link, and the bucket takes a few minutes to reach Ready through the broker, Crossplane, and floci.

The minted tokens are written to bin/dev-demo/admin.token and bin/dev-demo/operator.token (mode 0600, under the gitignored bin/ tree), so you can keep driving the CLI non-interactively afterwards:

export PLEXSPHERE_URL=http://localhost:8080
export PLEXSPHERE_TOKEN=$(cat bin/dev-demo/admin.token)
plexctl resource list --project-id <provision-demo-id>

make dev-demo is idempotent — every artefact is looked up by its slug or name before it is created — so a re-run reuses what the previous run planted and only repeats the resource poll. Tear it all down with make dev-down.

Run the golden flow

The end-to-end golden flow is the canonical "the stack works" assertion. It is the same test the dev-stack-smoke CI job runs nightly. On the laptop, point make e2e at the dev cluster the laptop loop already brought up rather than letting the harness spin up a per-suite cluster:

DEV_STACK=1 make e2e

DEV_STACK=1 mode skips the per-suite kind-load.sh and kind create cluster steps and runs the chainsaw catalog directly against kind-plexsphere. A run takes ~40 seconds against a warm cluster vs. ~3 minutes for the default cold-start mode.

The golden flow runs as the chainsaw catalog tests/e2e/dev/golden-flow/chainsaw-test.yaml — the nine-step catalog that exercises the full bootstrap-token register → Domain create → Project create → IdP binding → Identity sign-in → reachability transition flow against the /v1 API.

The scenario is self-contained: it applies the dev overlay into its own plexsphere-golden-flow namespace, so against a live make dev cluster it stands up a second stack instance beside the one the laptop loop is already running. Two instances can only coexist because the scenario prepares its own: it grants its namespace the cluster-scoped RBAC the overlay's default-pinned ClusterRoleBindings do not reach, and scales the one-per-cluster controllers down — see running a second stack instance in one cluster. When a run goes red, the scenario's failure handler dumps every container's log, including the previous instance of a restarting one, before chainsaw deletes the namespace — reading a failed dev-stack scenario covers what lands in the log and why the workflow's job-level collection cannot replace it.

The default make e2e (without DEV_STACK=1) behaviour is unchanged — the harness still creates a fresh cluster per suite, which is what CI uses for clean-room reproducibility. The tests/integration/dev_stack_e2e_mode_test.go integration test asserts that default-mode is unchanged and that DEV_STACK=1 against a missing cluster fails fast with a tagged diagnostic.

Manifest-to-CI mapping

Every manifest applied by make dev has a mirror in the hosted CI pipeline. The table below is the authoritative cross-reference between a local manifest directory and the CI job that exercises it. The drift gate in tests/docs/docs_dev_local_kind_test.go asserts every deploy/local/base/<service>/ directory appears as a row below; adding a new service without a row fails CI.

Manifest path	Purpose	CI job / test that exercises it
deploy/local/kind.yaml	kind cluster config; binds host ports 8080/5556 and patches the kubelet to pull images in parallel (serializeImagePulls: false, maxParallelImagePulls: 6) so a cold boot does not queue every upstream image pull behind a single slow download.	dev-smoke (schedule + label-gated PR) and tests/workspace/dev_manifests_test.go (kubelet-patch drift gate).
deploy/local/base/postgres/	Postgres 16 StatefulSet + Service + Secret; core + spicedb databases.	test-integration (testcontainers Postgres) and dev-smoke (cluster-local Postgres).
deploy/local/base/nats/	NATS StatefulSet + Service with JetStream enabled; mirrors the testcontainers fixture.	test-integration and dev-smoke.
deploy/local/base/dex/	Dex Deployment + Service + ConfigMap with the static plexsphere-test client and admin@example.com user.	test-integration (testcontainers Dex) and dev-smoke (sign-in through the Gateway).
deploy/local/base/oidc-cc-provider/	In-tree OIDC provider (cmd/oidc-cc-provider) serving discovery plus the OAuth2 client_credentials token endpoint with one static client. Backs POST /v1/auth/service/token for the seeded svc-demo machine identity because Dex does not implement that grant. ClusterIP only — reached server-side by the plexsphere binary, never from the host.	test-integration (tests/integration/identity_service_token_test.go covers the handler) and tests/workspace/dev_manifests_test.go (drift gate).
deploy/local/base/spicedb/	SpiceDB Deployment + Service + Secret; ReBAC engine with gRPC on port 50051, Postgres datastore shared with plexsphere, preshared-key auth, single replica with dispatch disabled.	test-integration (testcontainers SpiceDB via internal/platform/testutil/containers/spicedb.go) and dev-smoke (cluster-local SpiceDB serving the plexsphere API's authz checks).
deploy/local/base/migrate/	One-shot Job running tools/migrate up against the in-cluster Postgres; initContainer waits for readiness.	dev-smoke + tests/integration/dev_kind_test.go::TestDevKind_MigrateCompletesBeforeAPIReady.
deploy/local/base/plexsphere/	API Deployment + Service + ConfigMap; mounts Postgres/NATS/Dex connection env vars.	dev-smoke (readiness probe via Gateway /readyz).
deploy/local/base/plexsphere-signer/	Signer Deployment + Service + ServiceAccount + NetworkPolicy + Secret; the gRPC signing service that the API calls over the in-cluster plexsphere-signer:8443 endpoint.	test-integration (signer binary tests under tests/integration/signer_*_test.go) and tests/workspace/signer_manifests_test.go (manifest drift gate).
deploy/local/base/plexsphere-bootstrap/	One-shot Job that seeds the initial tenancy Domain aggregates via cmd/plexsphere-bootstrap AND seeds one IdP binding per demo Domain against the in-cluster Dex IdP (http://localhost:5556/dex, client_id: plexsphere-test), then pre-creates the canonical admin@example.com user under every Domain with admin + auditor ReBAC tuples. Reads an inline ConfigMap manifest and is idempotent on re-apply (Domain pass takes the already-exists no-op branch; IdP-binding pass takes the partial-unique-index conflict branch). initContainer waits for the migrate Job.	test-integration (tests/integration/bootstrap_seed_idempotent_test.go) + tests/workspace/dev_manifests_test.go (drift gate).
deploy/local/base/console/	Static-asset Console Deployment + ClusterIP Service: an unprivileged nginx serving the npm run build bundle (console/Dockerfile) with an index.html SPA fallback and a Content-Security-Policy response header. Served behind the api:8080 Gateway listener via the console HTTPRoute, sharing the single browser-facing origin with the API.	dev-smoke (tests/e2e/dev/chainsaw-test.yaml asserts the console HTTPRoute resolves and an in-cluster curl Pod proves GET / returns 200 with the CSP header + SPA fallback) + tests/integration/dev_kind_test.go::TestDevKind_DevOverlayReachesReady/ServesConsoleWithCSP + tests/workspace/gateway_console_split_test.go (offline split drift gate).
deploy/local/base/gateway/	EnvoyProxy (pins data-plane Service to NodePort 30080) applied directly into envoy-gateway-system + GatewayClass (envoy-gateway, references the EnvoyProxy via parametersRef) + Gateway + two HTTPRoutes splitting the api listener (8080): httproute-api.yaml keeps /v1 + the root /livez /readyz /metrics endpoints on plexsphere, httproute-console.yaml routes the rest to the Console. The EnvoyProxy is NOT part of the dev overlay because kubectl apply -n plexsphere -k … rejects cross-namespace manifests; make dev and the chainsaw install step apply it separately with kubectl apply -f deploy/local/base/gateway/envoyproxy.yaml.	dev-smoke + tests/e2e/dev/chainsaw-test.yaml (asserts GatewayClass.Accepted=True, then Gateway.Accepted=True / Gateway.Programmed=True, then each HTTPRoute.Accepted=True / HTTPRoute.ResolvedRefs=True — Gateway API v1 publishes ResolvedRefs only at Listener / HTTPRoute level, not on the Gateway itself).
deploy/local/vendor/gateway-api-crds.yaml	SHA-pinned Gateway API v1 CRDs, applied before the overlay.	dev-smoke bootstrap step.
deploy/local/vendor/envoy-gateway.yaml	SHA-pinned Envoy Gateway install manifest.	dev-smoke bootstrap step.
deploy/local/overlays/dev/kustomization.yaml	Dev overlay; pins :dev image tags and sets imagePullPolicy: Never for plexsphere containers.	dev-smoke + tests/workspace/dev_manifests_test.go (parse-YAML drift gate).
deploy/local/base/openbao/	OpenBao 2.0.0 Deployment + Service + ConfigMap + Secret in -dev mode (in-memory storage, pre-unsealed, single root token); the cluster-local secrets engine.	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/seaweedfs/	SeaweedFS 3.75 single-replica StatefulSet + Service running master + volume + filer + s3 with emptyDir-backed storage; the in-cluster S3-compatible object store.	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/mimir/	Grafana Mimir 2.14.3 single-binary Deployment + Service + ConfigMap with filesystem-backed storage and multi-tenancy disabled; the in-cluster metrics backend.	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/loki/	Grafana Loki 3.3.2 single-binary monolithic Deployment + Service + ConfigMap with filesystem-backed chunks and in-memory ring; the in-cluster logs backend.	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/crossplane/	Crossplane v2.0.2 core controller (real core start) Deployment + ServiceAccount + scoped crossplane ClusterRole/Binding + webhook TLS Secrets + Service; the management-fleet control plane (no providers shipped).	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/external-secrets/	External Secrets Operator v0.18.2 Deployment + ServiceAccount + ClusterRoleBinding + Service; pulls Secrets from cluster-external backends in production overlays.	dev-stack-smoke + tests/integration/dev_stack_kind_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).
deploy/local/base/crossplane-functions/	Crossplane composition Functions (function-patch-and-transform + function-auto-ready) plus the composite-RBAC ClusterRole. The Function package CRs install only once Crossplane core has registered the pkg.crossplane.io CRDs, so make dev applies functions.yaml in its substrate step rather than via the overlay's one-shot apply; only the CRD-independent composite-rbac.yaml rides the overlay. function-auto-ready propagates composed-resource readiness onto the XR the broker observes.	provisioning-broker (e2e reach-Ready through the cloudless substrate) + tests/workspace/dev_manifests_test.go (image SHA-pin + drift gate).
deploy/local/base/provider-kubernetes/	crossplane-contrib provider-kubernetes (Provider + DeploymentRuntimeConfig package CRs in provider.yaml), its ServiceAccount RBAC (rbac.yaml), and the InjectedIdentity ProviderConfig (providerconfig.yaml). Like the Functions, the package CRs and the ProviderConfig install only after their CRDs are served, so make dev applies provider.yaml then providerconfig.yaml in its substrate step; only rbac.yaml rides the overlay. Renders the cloudless blueprint's in-cluster substrate objects with no cloud account.	provisioning-broker (e2e reach-Ready through the cloudless substrate) + tests/workspace/dev_manifests_test.go (image SHA-pin + drift gate).
deploy/local/base/blueprint-cloudless/	Dev-stack copy of the cloudless Kubernetes-node blueprint's xrd.yaml + composition.yaml. Crossplane needs the CompositeResourceDefinition and Composition installed before it can compose a broker-rendered composite, and the broker only reads the imported XRD blob to derive the composite's API kind — it never applies them. These two files are not in the overlay (the apiextensions.crossplane.io CRDs are runtime-installed by Crossplane core), so make dev applies them with kubectl apply -f in its substrate step, after core is Healthy. The embedded seed catalog they once lived in was removed when the OCI import flow became the production path; see the DECISION block in xrd.yaml.	provisioning-broker (e2e reach-Ready through the cloudless substrate) + tests/workspace/dev_targets_test.go (apply-path existence gate).
deploy/local/base/floci/	floci, a lightweight local AWS emulator (a LocalStack alternative) serving AWS-shaped services at http://floci:4566. Lets a Crossplane AWS provider drive a managed resource against an emulated AWS API with no cloud account, token, or cost. Distributed on Docker Hub (the fast registry path on the constrained dev machines), SHA-pinned, with enableServiceLinks: false so the injected FLOCI_PORT Service-link env var does not collide with floci's own listen-port config. Its Deployment + Service are pinned to the default namespace so the emulator stays co-located with the provider-aws runtime that resolves it at floci.default.svc.cluster.local; a namespace-pinned object cannot ride the overlay's kubectl apply -n <ns> -k apply, so they are NOT in the base kustomization — make dev applies them directly (same constraint as the EnvoyProxy).	tests/workspace/dev_manifests_test.go (image SHA-pin + drift gate) + tests/workspace/dev_warning_initcontainer_test.go.
deploy/local/base/provider-aws/	The upjet provider-aws-s3 v2.x family substrate: the digest-pinned member Provider, the tag-only shared provider-family-aws Provider (a Crossplane dependency target — a digest pin there leaves the member UnhealthyPackageRevision and blocks make dev), and their DeploymentRuntimeConfigs (which inject the floci endpoint via AWS_ENDPOINT_URL_* env vars) in provider.yaml, the cluster-scoped ClusterProviderConfig.aws.m.upbound.io named plexsphere-aws in providerconfig.yaml, scoped controller RBAC (rbac.yaml), and a dummy-credentials Secret (secret.yaml). Like provider-kubernetes, the package CR and ClusterProviderConfig install only after their CRDs are served, so make dev applies provider.yaml then providerconfig.yaml in its substrate step; only the cluster-scoped rbac.yaml rides the overlay. The default-pinned dummy-credentials Secret (matching the ClusterProviderConfig's secretRef.namespace) cannot ride the overlay's kubectl apply -n <ns> -k apply, so make dev applies it directly alongside floci. Serves the namespaced s3.aws.m.upbound.io Bucket the floci-backed blueprint composes. The management fleet converges the shared family Provider itself for every family member a Cloud declares, co-owning this hand-applied object under the dependency-derived name upbound-provider-family-aws. That apply carries no --force-conflicts equivalent, so it only co-owns while the two agree: the fleet derives the family at the member's tag, and a manifest family tag that differs would be refused by the apiserver and recorded as Conflict rather than rewritten. tests/workspace/seed_provider_package_pin_test.go gates the equality offline.	tests/workspace/dev_manifests_test.go (drift gate); the full render-against-floci path (provider + functions + floci + CoreDNS, asserting the composite reaches Ready) is exercised by tests/e2e/provisioning/floci-aws-s3-ready/.
deploy/local/base/blueprint-aws-s3/	Dev-stack copy of the floci-backed aws-s3-bucket blueprint's xrd.yaml + composition.yaml, byte-identical to the upstream catalog layer, mirroring blueprint-cloudless/. The composite is a namespaced XAWSS3Bucket rendering one s3.aws.m.upbound.io/v1beta1 Bucket against the plexsphere-aws ClusterProviderConfig. Not in the overlay; make dev applies the pair with kubectl apply -f after the provider package is Healthy, alongside the cloudless pair; see the DECISION block in xrd.yaml.	tests/e2e/provisioning/floci-aws-s3/ (admission contract) + tests/workspace/chainsaw_floci_aws_s3_security_context_test.go (security-context drift gate).
deploy/local/base/plexd/	plexd sibling-product Deployment + Secret; the bootstrap-token registration agent that calls plexsphere's POST /v1/register on startup. Image tag is templated from /PLEXD_VERSION.	dev-stack-smoke + tests/integration/dev_stack_plexd_register_test.go and tests/workspace/dev_stack_manifests_test.go (drift gate).

SpiceDB ReBAC engine

The dev stack ships SpiceDB as the authorisation engine behind the plexsphere API. Every identity-scoped HTTP endpoint checks a relation tuple against SpiceDB before serving a response, so the Deployment is a hard dependency for the /v1/... API surface.

• Role: relationship-based access control (ReBAC) evaluator. Stores subject/relation/object tuples and answers permission-check queries from the plexsphere API.
• Service: spicedb:50051 (in-cluster DNS), gRPC only. HTTP is disabled (--http-enabled=false) because the only consumer is the plexsphere API's authzed gRPC client.
• Auth: preshared-key (spicedb-credentials.SPICEDB_GRPC_PRESHARED_KEY, value test-key). Every plexsphere pod reads the same key from the Secret and attaches it as authzed token bearer metadata on each request.
• Datastore: Postgres. Uses postgres-credentials.DATABASE_URL — SpiceDB keeps its relation tuples in its own schema namespace inside the shared database.
• Dispatch: disabled (--dispatch-cluster-enabled=false). Single replica, no peers, so cluster dispatch would only introduce a resolution failure on boot. A multi-replica HA posture is tracked separately as a follow-up.
• Security: dev-only. Preshared-key auth is NOT production-safe — any pod that can reach spicedb:50051 can speak as any caller. A follow-up replaces this with mTLS + per-workload SPIFFE identities.

Inspect the Deployment (the dev overlay applies into the default namespace, so no -n flag is required):

kubectl get deployment spicedb
kubectl logs deployment/spicedb

Confirm the schema has loaded (the SpiceDB schema tables live in the shared Postgres database and are created by the SpiceDB migrate one-shot on first boot). Postgres is a StatefulSet — the Pod name is postgres-0, not deployment/postgres:

kubectl exec postgres-0 -- \
  psql -U plexsphere -d plexsphere -c '\dt plexsphere.*' | grep -i relation

The SpiceDB wiring rationale, the base manifests, and the full set of --datastore-* flags live in the kustomize base under deploy/local/base/spicedb/ — open the directory's README.md together with the manifests it pins for the per-flag justification.

Troubleshooting

Nine failure modes account for the vast majority of red dev runs. For each, the "how to recognise" column is what the operator typically sees first, and the "how to fix" column names the exact follow-up command.

Symptom / failure mode	How to recognise	How to fix
Port collision — host port 8080 or 5556 is already bound by another process.	make dev aborts with failed to create cluster: exit status 1 and kind logs address already in use during docker run.	Free the port (lsof -i :8080, kill the offender) OR edit deploy/local/kind.yaml to remap the host side, then rerun make dev-down && make dev.
GatewayClass never admitted — the Gateway sits with the CRD-default conditions forever because the controller has not bound to its class.	kubectl describe gateway plexsphere -n plexsphere shows Accepted/Programmed conditions with Status: Unknown, Message: "Waiting for controller", and LastTransitionTime: 1970-01-01T00:00:00Z; kubectl get gatewayclass envoy-gateway lacks Accepted=True.	Re-apply the class: kubectl apply -f deploy/local/base/gateway/gatewayclass.yaml. Confirm spec.controllerName=gateway.envoyproxy.io/gatewayclass-controller. The workspace drift gate TestDevManifests_GatewayClassMatchesGateway catches this offline.
Gateway AddressNotAssigned — listeners are Programmed=True but the Gateway reports Programmed=False / Reason: AddressNotAssigned / Message: "No addresses have been assigned to the Gateway".	kubectl describe gateway plexsphere -n plexsphere shows the AddressNotAssigned reason on the top-level Programmed condition; kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=plexsphere -o wide lists the data-plane Service as type: LoadBalancer with an empty EXTERNAL-IP.	Envoy Gateway defaults to type: LoadBalancer; a fresh kind cluster has no LB. Re-apply the EnvoyProxy that pins the Service to NodePort: kubectl apply -f deploy/local/base/gateway/envoyproxy.yaml. Confirm GatewayClass.spec.parametersRef points at EnvoyProxy/envoy-proxy-config in envoy-gateway-system. The workspace drift gate TestDevManifests_EnvoyProxyPinsKindNodePorts catches drift between the EnvoyProxy pins and the kind extraPortMappings.
Gateway readiness timeout — GatewayClass is Accepted=True but the Gateway stays Programmed=False.	kubectl get gateway shows Programmed=False; Envoy pods in envoy-gateway-system are crashlooping.	Re-apply the vendor manifests with server-side apply (the envoyproxies CRD exceeds the 256 KiB last-applied-configuration annotation cap): kubectl apply --server-side --force-conflicts -f build/dev-vendor/gateway-api-crds.yaml -f build/dev-vendor/envoy-gateway.yaml, then kubectl rollout restart deployment -n envoy-gateway-system.
kubectl wait on an HTTPRoute hangs — a custom readiness check times out even though kubectl describe httproute shows Accepted=True + ResolvedRefs=True.	Route conditions in Gateway API v1 live under .status.parents[].conditions[] (per-parent), not top-level. kubectl wait --for=condition=X=True reads only the top-level list, so the wait can never succeed against HTTPRoute / GRPCRoute / TCPRoute / TLSRoute / UDPRoute.	Use a partial-match assertion against the nested status.parents[].conditions[] structure (see tests/e2e/dev/chainsaw-test.yaml step assert-httproutes-resolved). The workspace drift gate TestChainsawTests_NoWaitAgainstGatewayAPIRoutes enforces this offline.
Dex clock skew — sign-in fails with id_token has expired immediately after issuance.	kubectl logs deployment/dex shows iat timestamps more than 60s apart from the kubelet clock.	Resync the host clock (sudo ntpdate -u pool.ntp.org on Linux; restart Docker Desktop on macOS), then make dev-down && make dev.
ErrImageNeverPull — a plexsphere pod stays in ErrImageNeverPull because the image was never side-loaded.	kubectl describe pod shows Failed to pull image "plexsphere:dev": imagePullPolicy set to Never.	Re-run make dev (it re-invokes kind load docker-image); if the issue persists, confirm the image exists locally: docker image inspect plexsphere:dev.
Dependency pods stuck in PodInitializing — on a cold boot, upstream images trickle in one at a time even though each pull is fast.	kubectl get events --field-selector reason=Pulled shows sub-second pulls with minutes of including waiting queue time; the stuck pods' init containers are long Completed.	The kubelet patch in deploy/local/kind.yaml (serializeImagePulls: false, maxParallelImagePulls: 6) removes the queueing, and the make dev preload step (scripts/dev-preload-upstream-images.sh) side-loads every SHA-pinned upstream image from the host Docker cache so a cluster re-create does not hit the network at all. A cluster created before either change keeps the old behaviour — make dev-down && make dev re-creates it. Only the very first preload on a machine still downloads each image once; a slow registry path bounds that one-time warm-up.
kind node out of disk — pods evict with DiskPressure.	kubectl describe node plexsphere-control-plane shows Conditions: DiskPressure=True.	Tear down, prune, and rerun: make dev-down && docker system prune -af && docker volume prune -f && make dev.
floci crashloops at boot — the AWS emulator pod restarts with a startup config error.	kubectl logs deployment/floci shows a Quarkus config parse error such as Expected an integer value, got tcp://...:4566.	Kubernetes injects a FLOCI_PORT=tcp://<ip>:4566 Service-link env var that floci misreads as its own listen port. The floci Deployment sets enableServiceLinks: false to stop the injection; confirm a local edit did not drop it, then kubectl rollout restart deployment/floci.
make dev hangs on "waiting for the Crossplane provisioning substrate" — the provider-Healthy wait never returns and make dev aborts after its 300s timeout, never reaching the final crossplane rollout restart step.	kubectl get provider.pkg.crossplane.io shows provider-aws-s3 INSTALLED=True / HEALTHY=False; its ProviderRevision reports RevisionHealthy=False with cannot resolve package dependencies: missing dependencies: "xpkg.upbound.io/upbound/provider-family-aws" (v2.6.1) or Invalid Semantic Version. kubectl get lock lock -o yaml shows the provider-family-aws entry with a :v2.6.1 tag baked into source and/or a sha256: version.	Reference the shared family Provider in provider-aws/provider.yaml tag-only (xpkg.upbound.io/upbound/provider-family-aws:v2.6.1), never tag+digest. Crossplane parses a name:tag@digest ref into Lock source name:tag + version <digest>, but the member's dependsOn names the bare name and constraint-checks v2.6.1: a baked-in tag misses the lookup ("missing dependencies") and a sha256: version fails the semver parse ("Invalid Semantic Version"). Only a tag-only ref yields Lock source name + version v2.6.1, which the dependency resolves. After fixing, delete the stale revision (kubectl delete providerrevision.pkg.crossplane.io -l app.kubernetes.io/name=provider-family-aws) so it recreates from the tag, or just make dev-down && make dev. The tests/e2e/provisioning/floci-aws-s3-ready/ provider-Healthy wait guards this. A member the management fleet installs does not rely on the hand-applied family object — the installer converges the family package itself, tag-only, under the dependency-derived name upbound-provider-family-aws — so this trap concerns only the hand-applied make dev path.
floci-backed Bucket never reaches Ready — an XAWSS3Bucket (or its composed s3.aws.m.upbound.io Bucket) stays in Creating/Synced=False.	kubectl describe bucket.s3.aws.m.upbound.io ... shows Synced=False reason=ReconcileError with observe failed: ... S3 Control: ListTagsForResource ... StatusCode: 403 ... AccessDenied (note the real-AWS RequestID/HostID in the body), or the controller logs show the request reaching the real AWS host (InvalidAccessKeyId) or a DNS failure for an account-prefixed host like 000000000000.floci.default.svc.cluster.local.	The upjet provider reaches floci through the AWS_ENDPOINT_URL_* env vars on the provider runtime container (see provider-aws/provider.yaml), and the S3-Control path additionally needs a CoreDNS wildcard rewrite *.floci.default.svc.cluster.local → floci so the account-prefixed host resolves. make dev installs that rewrite via scripts/floci-coredns-rewrite.sh (idempotent); if a Bucket still fails DNS, confirm the rewrite landed with kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' and re-run the script against the dev context. The broker drives a nodeless standalone blueprint (the aws-s3-bucket mesh role) straight to Ready once Crossplane reports the substrate ready — it mints no bootstrap token and skips the Enrolling phase — so a Bucket stuck short of Ready is a substrate fault, not a missing broker path. S3-Control tag read (floci-io/floci#1535, closed not-a-bug): provider-aws-s3 v2.6.1 is built on the Terraform AWS provider v6, which reads bucket tags exclusively through S3 Control ListTagsForResource on every Observe. Reaching floci on that call needs two things, both shipped: (1) the per-service override env var, which the AWS SDK derives from the model serviceId by replacing spaces with underscores and uppercasing — the s3control serviceId is S3 Control (two words), so the var must be AWS_ENDPOINT_URL_S3_CONTROL with the underscore (the no-underscore AWS_ENDPOINT_URL_S3CONTROL is silently ignored); and (2) because S3 Control prepends a hostPrefix of {AccountId}., the SDK then dials 000000000000.floci.default.svc.cluster.local, which resolves to floci only through the CoreDNS wildcard rewrite above. floci serves ListTagsForResource correctly (its #341); a 403 AccessDenied whose body carries <AccountId> and <HostId> elements is the real AWS S3-Control error shape — floci never emits it — so that response proves the Observe reached real AWS, not floci, i.e. the override did not take effect. The standard S3 GetBucketTagging succeeds on the same bucket because it uses the plain AWS_ENDPOINT_URL_S3 override and carries no host prefix; only the S3-Control variant exposes a misspelled env var or a missing rewrite. So a Bucket that CREATEs (it shows up in ListBuckets) but never reaches Ready — composed s3.aws.m.upbound.io Bucket at Synced=False reason=ReconcileError / Ready=False reason=Creating, function-auto-ready never flipping the XAWSS3Bucket XR to Ready, broker holding the Resource at Provisioning — means the S3-Control request leaked to real AWS: confirm the env var is the underscore form on the provider pod (kubectl -n default get pod -l pkg.crossplane.io/provider=provider-aws-s3 -o jsonpath='{.items[0].spec.containers[0].env}') and that the rewrite landed (command above), re-running scripts/floci-coredns-rewrite.sh against the dev context if needed.

Tear down

make dev-down deletes the plexsphere kind cluster — and nothing else. It refuses to delete a foreign cluster named plexsphere that some other tooling created: the sentinel ConfigMap planted by make dev is the only signal the target trusts:

make dev-down

The sentinel-ConfigMap probe lives in deploy/local/base/ and is applied by make dev. Three sub-tests in tests/integration/dev_stack_down_test.go enforce the contract:

• owned-deletes — make dev planted the sentinel, so make dev-down deletes the cluster.
• foreign-preserves — a kind cluster named plexsphere without the sentinel is preserved; the target prints a tagged warning and exits zero.
• force-down-bypasses — DEV_FORCE_DOWN=1 make dev-down bypasses the sentinel check (escape hatch for an inconsistent cluster the operator deliberately wants to scrub).

Reset recipe

When the stack is in an indeterminate state — a half-applied overlay, a hung Deployment, or a cluster that the troubleshooting matrix does not cover — the reliable reset is: delete the cluster, prune Docker state, and rerun make dev from a clean baseline.

make dev-down
docker system prune -f
make dev

make dev-down deletes the plexsphere kind cluster only; unrelated kind clusters on the same machine are untouched. The docker system prune -f step reclaims dangling layers and exited containers — the same hint make dev-down prints as its final line. The subsequent make dev run takes ~3 minutes from a cold cache (the plexsphere image build dominates) and ~40 seconds from a warm one.