Local dev stack

This is the per-dependency reference for the local development stack that make dev brings up. Each section describes one service: the SHA-pinned image the dev manifests apply, the dev-mode flags the container boots with, the prod-delta sentence the Pod announces at boot, and a forward link to the manifest source so the reader can diff a flag change against the operator-facing posture in one click .

The line operators tail to confirm a dev posture is active is the same one every Pod under deploy/local/base/ emits (either via a dev-warning init-container or via internal/platform/insecuredefaults.Emit for plexsphere-built Go binaries):

level=WARN msg="insecure default active in dev mode" env=dev dependency=<svc> prod_delta=<sentence>

The integration test tests/integration/dev_stack_insecure_defaults_test.go greps every Pod's logs for this line; a Pod that omits it fails the suite.

See also:

• docs/contributing/dev-stack.md — the entry-point runbook (make dev → seed → sign in with plexctl → golden flow → tear down).
• docs/tutorials/set-up-local-plexsphere.md — the narrower local-kind dev with the manifest-to-CI mapping table and the troubleshooting matrix.
• deploy/local/README.md — the kustomize base reference, including the SpiceDB wiring contract and per-service dev-only postures.

Postgres

The shared OLTP database — both plexsphere and SpiceDB store rows here, in separate schema namespaces. The kind dev introduced this dependency; the dev-stack reuses it unchanged.

Aspect	Value
Image	postgres:16-alpine@sha256:93d55776e04376e19adb2733e3ccebb4392ee7dd86d8ff238503b30fe719c84f
In-cluster	postgres:5432 (ClusterIP Service)
Replicas	1 (StatefulSet)
Storage	emptyDir — wiped on Pod restart
Auth	static password from postgres-credentials Secret

Dev-mode flags / configuration:

• single-replica StatefulSet with emptyDir storage rather than a persistent volume — the cluster lifecycle is bound to the kind container, so durability is not the goal.
• one Postgres instance hosts both the plexsphere and the SpiceDB schemas to keep the dev cluster footprint minimal.
• POSTGRES_PASSWORD ships as a static value in postgres-credentials; every workload reads the same value.

Prod-delta: production runs Postgres as a managed primary + replicas with a real durable volume, per-workload least-privilege roles, TLS-only listener, and rotating credentials sourced from the secrets manager. The SpiceDB schema lives in its own database instance.

Forward link: deploy/local/base/postgres/.

NATS

The messaging plane with JetStream enabled — plexsphere publishes domain events here and the testcontainers integration tier asserts the same JetStream contract. The kind dev introduced this dependency.

Aspect	Value
Image	nats:2.10.20@sha256:97f3bde5637e9cb75c09c15869c79c2c0d909f26c8dd371d169b175a58ff771a
In-cluster	nats:4222 (ClusterIP Service)
Replicas	1 (StatefulSet)
JetStream	enabled, emptyDir-backed
Auth	none — open NATS server

Dev-mode flags / configuration:

• --jetstream enabled; storage directory under emptyDir so streams are recreated on Pod restart.
• single replica, no clustering — dispatch is local to one Pod.
• no TLS, no NKey/JWT auth; any client that can reach nats:4222 can publish to any subject.

Prod-delta: production runs a NATS cluster with at least three replicas, durable JetStream storage on a persistent volume, TLS for both client and inter-node connections, and per-account NKey/JWT auth. Stream replication factor matches the cluster size.

Forward link: deploy/local/base/nats/.

Dex

The OIDC identity provider used by the dev sign-in flow. The kind dev introduced this dependency; the static plexsphere-test client and admin@example.com user mirror the testcontainers fixture internal/platform/testutil/containers/dex.go.

Aspect	Value
Image	ghcr.io/dexidp/dex:v2.41.1@sha256:bc7cfce7c17f52864e2bb2a4dc1d2f86a41e3019f6d42e81d92a301fad0c8a1d
In-cluster	dex:5556 (ClusterIP Service)
Replicas	1
Storage	in-memory (Dex storage.type: memory)
Static user	admin@example.com (password password)
Static client	plexsphere-test

Dev-mode flags / configuration:

• Dex storage.type: memory — every restart wipes the user / refresh token state.
• the static user and static client are baked into the Dex ConfigMap; there is no upstream IdP federation.
• the issuer URL hard-codes http://dex:5556/dex, mapped to the host via the kind extraPortMappings block; plexctl login sign-in flows go through the same URL.

Prod-delta: production wires Dex (or a managed IdP) with a real storage backend (Postgres or Kubernetes CRDs), federation against the corporate IdP, TLS-only issuer, and rotating client secrets managed by the secrets engine.

Forward link: deploy/local/base/dex/.

SpiceDB

The ReBAC engine that the plexsphere API consults on every identity-scoped HTTP endpoint. The wiring contract (preshared-key auth, in-Postgres datastore, gRPC-only on :50051) is documented at length in deploy/local/README.md.

Aspect	Value
Image	authzed/spicedb:v1.39.0@sha256:b98123b44d730cbdfef29494ff9b8ff603d64418affe8fd64d90406ee4e0d9c5
In-cluster	spicedb:50051 (gRPC, ClusterIP Service)
Replicas	1
Datastore	Postgres (shared with plexsphere)
Auth	preshared key (test-key)
Dispatch	disabled (--dispatch-cluster-enabled=false)

Dev-mode flags / configuration:

• --http-enabled=false — the only consumer is the plexsphere API's authzed gRPC client.
• --dispatch-cluster-enabled=false — single replica, no peers.
• --datastore-engine=postgres against the shared cluster Postgres.
• preshared-key auth via spicedb-credentials.SPICEDB_GRPC_PRESHARED_KEY.

Prod-delta: production replaces the preshared-key with mTLS + per-workload SPIFFE identities, runs SpiceDB as a multi-replica HA deployment with cluster dispatch enabled, and uses a dedicated Postgres instance (the production transition is tracked separately).

Forward link: deploy/local/base/spicedb/.

plexsphere

The plexsphere API itself — every /v1/* surface a contributor, an operator, or an API client talks to. The dev overlay boots the API with the full production factory chain so every /v1/* route resolves to its real handler instead of falling through to the earlier 501 Not Implemented stub. Without the env wiring documented below the cmd/plexsphere/*_factory_prod.go factories short-circuit to nil and the affected route returns 501.

Aspect	Value
Image	plexsphere:dev (kind-loaded by make docker-build)
In-cluster	plexsphere:8080 (HTTP, ClusterIP Service)
Replicas	1
Sidecar	dex-localhost-proxy (socat 127.0.0.1:5556 → dex.default.svc:5556)
Init	wait-for-migrate (gates on goose_db_version rows)
Probes	/livez + /readyz

The env-var contract every production factory in cmd/plexsphere/*_factory_prod.go reads at boot. Each row names the manifest source the dev overlay wires it through — Secret stringData, ConfigMap data, or a JSON6902 patch in the dev overlay — plus the factory file that consumes it and the /v1/* surface that returns 501 when it is unset:

Env var	Dev source	Factory	/v1/* surface required by
PLEXSPHERE_DSN	Secret postgres-credentials.DATABASE_URL	(every factory)	every /v1/* surface (datastore wiring)
PLEXSPHERE_PROJECTS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	projects_factory_prod.go	GET /v1/projects (cursor pagination)
PLEXSPHERE_CLOUDS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	clouds_factory_prod.go	GET /v1/clouds (cursor pagination)
PLEXSPHERE_DOMAINS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	domains_factory_prod.go	GET /v1/domains (cursor pagination)
PLEXSPHERE_IDENTITIES_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	identities_factory_prod.go	GET /v1/domains/{id}/identities
PLEXSPHERE_INVITATIONS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	invitations_factory_prod.go	GET /v1/domains/{id}/invitations (cursor pagination)
PLEXSPHERE_AUDIT_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	audit_factory_prod.go	GET /v1/audit (cursor pagination)
PLEXSPHERE_AUTHZ_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	authz_factory_prod.go	GET /v1/authz/relations (cursor pagination)
PLEXSPHERE_CLOUD_CREDENTIALS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	cloudcredentials_factory_prod.go	GET /v1/clouds/{id}/credentials (cursor pagination)
PLEXSPHERE_CREDENTIALS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	credentials_factory_prod.go	GET /v1/projects/{id}/credentials (cursor pagination)
PLEXSPHERE_CREDENTIAL_ASSIGNMENTS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	credentialassignments_factory_prod.go	GET /v1/projects/{id}/credential-assignments (cursor pagination)
PLEXSPHERE_NODES_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	nodes_factory_prod.go	GET /v1/nodes (cursor pagination)
PLEXSPHERE_CAPABILITIES_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	capability_inventory_factory_prod.go	GET /v1/projects/{project_id}/capabilities (cursor pagination)
PLEXSPHERE_HOOKS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	hook_catalog_factory_prod.go	GET /v1/hooks (cursor pagination)
PLEXSPHERE_INTEGRITY_VIOLATIONS_CURSOR_HMAC_KEY	Secret plexsphere-dev-cursor-keys (overlay patch)	integrity_violations_list_factory_prod.go	GET /v1/integrity-violations (cursor pagination)
PLEXSPHERE_NSK_WRAP_KEY_B64	Secret plexsphere-dev-cursor-keys (overlay patch)	registration_factory_prod.go, heartbeat_factory_prod.go	POST /v1/register (NSK wrap) and POST /v1/nodes/{id}/heartbeat (NSK unwrap / plaintext verification)
PLEXSPHERE_SIGNING_PUBLIC_KEY_B64	Secret plexsphere-dev-cursor-keys (overlay patch)	registration_factory_prod.go	POST /v1/register (signing public key)
PLEXSPHERE_SIGNING_KEY_ID	ConfigMap plexsphere-config	registration_factory_prod.go	POST /v1/register (kid)
PLEXSPHERE_CSRF_ALLOWED_ORIGINS	dev overlay JSON6902 patch	auth_factory_prod.go	every state-changing /v1/* (CSRF allowlist)
PLEXSPHERE_SESSION_IDLE_TIMEOUT	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (rolling idle window on the cookie)
PLEXSPHERE_SESSION_MAX_LIFETIME	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (absolute session cap)
PLEXSPHERE_AUTH_STATE_TTL	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (OIDC state lifetime)
PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER	ConfigMap plexsphere-config ("true" in dev)	audit_factory_prod.go	/v1/audit, /v1/domains/{id}/identities
PLEXSPHERE_ARTIFACTS_FULCIO_SAN	ConfigMap plexsphere-config (dev fixture SAN)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (release-signing SAN pin)
PLEXSPHERE_ARTIFACTS_OIDC_ISSUER	ConfigMap plexsphere-config (token.actions.githubusercontent.com)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (Fulcio OIDC issuer pin)
PLEXSPHERE_ARTIFACTS_OCI_REGISTRY	ConfigMap plexsphere-config (dev fixture registry path)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (upstream OCI source)
PLEXSPHERE_BRIDGE_ACME_DIRECTORY_URL	ConfigMap plexsphere-config (acme-staging-v02.api.letsencrypt.org)	bridge_factory_prod.go	bridge ingress create/update surfaces (ACME directory the certificate-feasibility validator probes before persisting a rule with an ACME account reference)
PLEXSPHERE_BRIDGE_ACME_PROBER_TIMEOUT	(unset — factory default 10s)	bridge_factory_prod.go	bridge ingress create/update surfaces (optional duration bounding each outbound ACME directory probe the certificate-feasibility validator runs; a non-positive or unparseable value is rejected at boot)
PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET	ConfigMap plexsphere-config (plexsphere-action-output)	actions_factory_prod.go	POST /v1/.../actions dispatch + POST /v1/nodes/{id}/executions/{exec_id} callback (object-store bucket the callback service mints over-ceiling output PUT URLs against)
PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	actions_factory_prod.go	POST /v1/.../actions dispatch (absolute base URL the dispatch service stamps onto the per-target callback a Node reports its result to)
PLEXSPHERE_S3_ENDPOINT	ConfigMap plexsphere-config (http://seaweedfs:8333)	actions_factory_prod.go, audit_factory_prod.go	POST /v1/nodes/{id}/executions/{exec_id} callback over-ceiling output PUT (shared S3 object-store endpoint the Action Orchestrator builds its client against at boot; the audit archiver reuses the family)
PLEXSPHERE_S3_REGION	ConfigMap plexsphere-config (us-east-1)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (a missing region fails the dev boot with blobstore: Config.Region is required)
PLEXSPHERE_S3_ACCESS_KEY	ConfigMap plexsphere-config (any — SeaweedFS runs unauthenticated)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (non-secret dev placeholder)
PLEXSPHERE_S3_SECRET_KEY	ConfigMap plexsphere-config (any — SeaweedFS runs unauthenticated)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (non-secret dev placeholder)
PLEXSPHERE_S3_USE_PATH_STYLE	ConfigMap plexsphere-config ("true")	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (SeaweedFS requires path-style addressing)
PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT	ConfigMap plexsphere-config ("true")	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (opts in to the plaintext in-cluster http:// SeaweedFS endpoint)
PLEXSPHERE_ACCESS_SIGNER_ENDPOINT	ConfigMap plexsphere-config (plexsphere-signer:8443)	access_factory_prod.go	session POST /v1/projects/{id}/sessions issue + POST /v1/projects/{id}/sessions/{sid}:revoke (host:port of the signer gRPC surface the issuance service signs every session JWT through)
PLEXSPHERE_ACCESS_CALLBACK_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	access_factory_prod.go	session POST /v1/projects/{id}/sessions issue (absolute base URL the issuance service stamps onto the per-session callback a target plexd reports activity to)
PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT	(unset — operator-tunable; dev reaches the in-cluster signer without client material)	access_factory_prod.go	session issue (optional mTLS client cert path for the signer gRPC dial; the unset triple defers the TLS-required error to the signer client)
PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY	(unset — operator-tunable; pairs with the client cert above)	access_factory_prod.go	session issue (optional mTLS client key path for the signer gRPC dial)
PLEXSPHERE_ACCESS_SIGNER_SERVER_CA	(unset — operator-tunable; the signer server CA the dial validates against)	access_factory_prod.go	session issue (optional server CA path for the signer gRPC dial)
PLEXSPHERE_ACCESS_CURSOR_HMAC_KEY	(unset — operator-tunable; dev falls back to the identity cursor codec)	access_factory_prod.go	GET /v1/projects/{id}/sessions (cursor pagination; a >=32-byte hex key enables the HMAC-signed cursor)
PLEXSPHERE_ACCESS_REVOCATION_TTL_FLOOR	(unset — factory default; the repo derives the revocation-list retention from the access domain floor)	access_factory_prod.go	session revoke (optional override documenting the revocation-list retention floor)
PLEXSPHERE_ACCESS_SWEEPER_TICK	(unset — factory default 60s)	access_factory_prod.go	the access-session idle/expiry reconcile / steady-state sweep cadence
PLEXSPHERE_ARTIFACTS_REFRESH_INTERVAL_SECS	(unset — factory default)	artifacts_factory_prod.go	the artifact-refresh reconcile / steady-state sweep cadence
PLEXSPHERE_PROVISIONING_BROKER_ENROL_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (absolute enrol base URL a booting node registers against during cloud-init first boot; REQUIRED when PLEXSPHERE_DSN is set or the broker refuses construction)
PLEXSPHERE_PROVISIONING_BROKER_PLEXD_DOWNLOAD_URL	ConfigMap plexsphere-config (http://localhost:8080/downloads/plexd)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (absolute URL the cloud-init first-boot runcmd fetches the plexd binary from; REQUIRED when PLEXSPHERE_DSN is set)
PLEXSPHERE_PROVISIONING_BROKER_PLEXD_IMAGE	ConfigMap plexsphere-config (ghcr.io/plexsphere/plexd:v0.1.0)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (plexd container image a helm-values blueprint deploys, pinned to the dev PLEXD_VERSION; REQUIRED when PLEXSPHERE_DSN is set)
PLEXSPHERE_MANAGED_PUSH_WRAP_KEY_B64	(unset — opt-in; the managed-push surface stays 501 until set, so the dev overlay does not wire it)	managed_push_factory_prod.go	the /v1/domains/{id}/managed-push attach/push/rollback surface (base64 of the 32-byte AES-256-GCM key the sealer binds the attached kubeconfig under at rest, with the owning Domain UUID as GCM AAD; the wrap key's presence is the managed-push opt-in, so an empty value leaves the six handlers on their 501 stub)
PLEXSPHERE_OBS_MIMIR_QUERY_URL	(unset — opt-in; the metrics-query surface stays 501 until set, so the dev overlay does not wire it)	observability_query_factory_prod.go	GET /v1/domains/{id}/metrics/query (Grafana Mimir query API base URL; setting it enables the PromQL instant / range metrics-query proxy, validated at boot, with the addressed Domain injected server-side as the upstream X-Scope-OrgID tenant)
PLEXSPHERE_OBS_LOKI_QUERY_URL	(unset — opt-in; the logs-query surface stays 501 until set, so the dev overlay does not wire it)	observability_query_factory_prod.go	GET /v1/domains/{id}/logs/query (Grafana Loki query API base URL; setting it enables the LogQL range logs-query proxy, validated at boot, with the same server-side X-Scope-OrgID tenant injection)

The drift gate tests/workspace/dev_overlay_factory_env_drift_test.go walks every productionXxxConfigFromEnv FuncDecl in cmd/plexsphere/ and fails closed if a required env var is read but not wired into one of the four manifest surfaces above.

Dev-mode flags / configuration:

• the four cursor HMAC keys are deterministic 32-byte hex fixtures (dec0deNN… repeating) — sufficient to clear the per-factory minXxxCursorHMACKeyLen floor and to make the dev cluster's cursor pagination round-trip without rotation. The substring dec0de is the "dev fixture" marker the prod-reference overlay's drift gate forbids.
• NSK_WRAP_KEY_B64 is base64 of the 32-byte ASCII fixture dev-nsk-wrap-key-do-not-use-32by; SIGNING_PUBLIC_KEY_B64 is the Ed25519 public half paired with the plexsphere-signer-seed fixture in the same overlay. The same 32-byte wrap key drives BOTH the registration factory's seal path (internal/identity/nodes/nsk/software.Provider.Issue) AND the heartbeat factory's verify path (internal/identity/nodes/nsk/software.Unwrapper.Unwrap). When the binary boots with PLEXSPHERE_DSN set but PLEXSPHERE_NSK_WRAP_KEY_B64 empty, the heartbeat factory refuses construction with ErrHeartbeatNSKWrapKeyRequired so the misconfiguration surfaces before /readyz lights up green.
• PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER=true opts the audit and identities factories into the deterministic staticPepper fallback because the dev overlay does not run a real OpenBao pepper. The per-Domain pseudonymisation still runs — the fallback is keyed off a fixed seed instead of a rotated material.
• PLEXSPHERE_CSRF_ALLOWED_ORIGINS=http://localhost:8080 is the closed allowlist the CSRF middleware compares incoming Origin headers against on every state-changing cookie-authenticated /v1/* request. With the in-tree dashboard removed, the only browser-facing origin in the dev cluster is the API behind the Gateway at host port 8080. Production overlays MUST supply their own absolute http(s) origin via the same patch shape.
• PLEXSPHERE_AUTH_CALLBACK_URL and PLEXSPHERE_AUTH_VERIFICATION_URL are intentionally left unset in the dev overlay. The auth factory falls back to a request-Host-derived callback and the relative /v1/device default, which the server now renders itself: the control plane hosts the device-verification + approval page in-tree, so interactive device-grant sign-in completes without an external dashboard. Operators who front the stack with their own UI can still point PLEXSPHERE_AUTH_VERIFICATION_URL at it.
• the dex-localhost-proxy socat sidecar exists so the API can call Dex through the same http://localhost:5556/dex issuer the browser uses — see the inline rationale in deploy/local/base/plexsphere/deployment.yaml. The chainsaw bootstrap-seed suite (tests/e2e/bootstrap-seed/chainsaw-test.yaml::signin-against-bootstrap-bindings) mirrors this sidecar pattern in CI to assert that the bootstrap-seeded IdP bindings answer /v1/auth/sign-in in-cluster.

Prod-delta: every Secret value above ships empty in the base manifests at deploy/local/base/plexsphere/secret.yaml; only the dev overlay's JSON6902 patches inject the deterministic dec0de… / dev-nsk-wrap-key-… / Ed25519 fixtures. PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER is "true" only in the dev ConfigMap. Production overlays must inject each cursor HMAC key, the NSK wrap key, and the signing public key through SealedSecrets / External Secrets (rotation surface), and counter-patch PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER to "" so the staticPepper fallback is unreachable. For the Artifact Registry, production points PLEXSPHERE_ARTIFACTS_OCI_REGISTRY at the real signed upstream plexd registry and pins the real release-signing Fulcio SAN (PLEXSPHERE_ARTIFACTS_FULCIO_SAN) and OIDC issuer (PLEXSPHERE_ARTIFACTS_OIDC_ISSUER) the release pipeline signs with; the dev ConfigMap ships fixture values instead, so there is no signed plexd release to verify and the refresh sweep stays inert — Refresh logs and skips, the genesis seed remains unverified, and the binary still boots green. PLEXSPHERE_ARTIFACTS_REFRESH_INTERVAL_SECS is left unset in dev so the factory default cadence applies; operators tune it in production when the upstream release cadence warrants. For the bridge validation pipeline, production points PLEXSPHERE_BRIDGE_ACME_DIRECTORY_URL at the operator's real ACME directory (the production CA the tenant issues certificates from), not the Let's Encrypt staging endpoint the dev ConfigMap ships; the dev value is harmless because the certificate-feasibility probe only fires for an ingress rule that carries an ACME account reference, which the dev overlay never creates. For the Action Orchestrator, production points PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL at the operator's externally reachable API origin (the absolute base a target Node resolves the per-target result callback against) rather than the dev http://localhost:8080, and PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET at the real object-store bucket the over-ceiling output uploads land in; the dev ConfigMap ships a localhost base URL and the plexsphere-action-output bucket name, both of which round-trip against the dev rig's SeaweedFS via the same PLEXSPHERE_S3_* family the audit archive uses. For the Access Orchestrator, production points PLEXSPHERE_ACCESS_CALLBACK_BASE_URL at the operator's externally reachable API origin (the absolute base a target plexd resolves the per-session activity callback against) rather than the dev http://localhost:8080, and mounts the full signer mTLS triple (PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT, PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY, PLEXSPHERE_ACCESS_SIGNER_SERVER_CA) the dev overlay leaves unset because the dev cluster reaches the in-cluster plexsphere-signer:8443 Service over a trusted in-cluster path; production also sets PLEXSPHERE_ACCESS_CURSOR_HMAC_KEY so session-list cursors are HMAC-signed rather than passed through the dev identity codec. The deploy/local/overlays/prod-reference/ overlay is the canonical reference shape — the workspace gate TestProdReferenceOverlay_DoesNotInheritDevSecrets asserts none of the dev fixture material leaks into it.

Operator-tunable knobs (optional)

The env vars in this section are read with the if raw := strings.TrimSpace(getenv("…")); raw != "" guard, so an unset value falls back to the in-binary default named in the table. The dev overlay does NOT wire them — set them only when the default does not fit the deployment. The workspace gate tests/workspace/dev_overlay_factory_env_drift_test.go's OPTIONAL allowlist pins this set so a new optional knob cannot land without being declared here.

Env var	Factory	Default	Effect when set
PLEXSPHERE_PEERS_ENDPOINT_SWEEP_INTERVAL	peers_factory_prod.go	1m	Steady-state cadence for the Peer endpoint-stale sweeper. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_PEERS_RELAY_ASSIGNER_INTERVAL	peers_factory_prod.go	30s	Heartbeat cadence for the relay-fallback assigner reconcile loop. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_SPIFFE_BUNDLE_TTL	auth_factory_prod.go	15m (spiffe.DefaultBundleCacheTTL)	Cache lifetime for the SPIFFE trust-bundle the JWT-SVID verifier consults on every POST /v1/auth/service/token. Parsed by time.ParseDuration; zero or negative falls back to the package default. Lower values shorten the window between an IdP-side bundle rotation and the API picking it up; higher values reduce upstream load.
PLEXSPHERE_APPROVALS_EXPIRE_TICK	approvals_factory_prod.go	60s (DefaultApprovalsExpireTick)	Cadence of the background sweep that expires stale pending-approval proposals. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_APPROVALS_CURSOR_HMAC_KEY	approvals_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key that binds the GET /v1/approvals list cursor to the presenting caller. When unset the list cursor falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_ACTIONS_CURSOR_HMAC_KEY	actions_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key (≥ 32 bytes) that binds the GET /v1/.../actions list cursor to the presenting caller. When unset the list cursor falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_ACTIONS_TIMEOUT_TICK	actions_factory_prod.go	30s (DefaultActionsTimeoutTick)	Cadence of the background sweep that times out expired live Executions and frees their per-Domain live-execution slots. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_ACTIONS_LIVE_EXECUTIONS_CAP	actions_factory_prod.go	1000 (DefaultActionsLiveExecutionsCap)	Per-Domain concurrent-execution budget the dispatch service enforces before admitting a new Execution. Parsed as a positive integer.
PLEXSPHERE_ACTIONS_PRESIGN_EXPIRY	actions_factory_prod.go	1h (DefaultActionsPresignExpiry)	Lifetime of an over-ceiling output PUT URL the callback service mints. Parsed by time.ParseDuration; must be positive and below the blobstore.MaxPresignExpiry ceiling.
PLEXSPHERE_ACTIONS_INLINE_OUTPUT_MAX_BYTES	actions_factory_prod.go	16384 (actions.MaxInlineOutputBytes)	Inline-output ceiling above which the callback service mints a presigned object-store PUT URL. The ceiling is a domain invariant with no runtime override, so a value disagreeing with the domain constant is refused at boot; the knob documents the operator-visible default.
PLEXSPHERE_ARTIFACTS_CURSOR_HMAC_KEY	artifacts_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key (≥ 32 bytes) that binds the GET /v1/artifacts/plexd list cursor to the presenting caller. Only consulted once the registry switch PLEXSPHERE_ARTIFACTS_OCI_REGISTRY is set; when unset the per-version GETs still boot and only the paginated list falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_S3_ENDPOINT	audit_factory_prod.go	(empty — disables the archive uploader)	Endpoint URL the audit-archive S3 client targets. Setting this with the rest of the family below wires the blobstoreArchiveUploader so audit rows beyond the per-Domain retention horizon drain to object storage.
PLEXSPHERE_S3_REGION	audit_factory_prod.go	(empty)	AWS region for the audit-archive bucket. Pass-through to the S3 client Region.
PLEXSPHERE_S3_ACCESS_KEY	audit_factory_prod.go	(empty)	Access-key ID for the audit-archive bucket. Pass-through to the S3 client.
PLEXSPHERE_S3_SECRET_KEY	audit_factory_prod.go	(empty)	Secret access key for the audit-archive bucket. Should ride in a Secret, never a ConfigMap.
PLEXSPHERE_S3_USE_PATH_STYLE	audit_factory_prod.go	false	When "true", instructs the S3 client to use path-style addressing (https://endpoint/bucket/key) instead of virtual-host style. Required for SeaweedFS, MinIO, and other S3-compatible backends that do not host per-bucket subdomains.
PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT	audit_factory_prod.go	false	When "true", permits a plain http:// endpoint on PLEXSPHERE_S3_ENDPOINT. The blobstore client refuses http:// by default (blobstore.ErrInsecureEndpoint); enable this opt-in only for dev or air-gapped TLS-terminating proxies.

The audit-archive S3 family ships unset in the base manifests because the dev rig's SeaweedFS bucket is wired through a different PLEXSPHERE_AUDIT_ARCHIVE_BUCKET path documented under docs/contexts/audit/storage.md; production deployments using AWS S3, GCS-S3, or another vendor object store set the family above to point the audit archive at the real backend.

Forward link: deploy/local/base/plexsphere/.

OpenBao

The cluster-local secrets engine. OpenBao is the BSL-licensed fork of HashiCorp Vault and is the planned production secrets engine for plexsphere.

Aspect	Value
Image	openbao/openbao:2.0.0@sha256:5eedbca9922d85eca5e4bc68c11f968d245b4046641dd4173c1dcff7ae7091aa
In-cluster	openbao:8200 (ClusterIP Service)
Replicas	1
Storage	in-memory (-dev mode)
Root token	static dev-only-root-token

Dev-mode flags / configuration:

• server -dev — boots in development mode: storage is in-memory, the server is pre-unsealed, and a single root token is printed to stdout (matched against the literal in the openbao-secrets Secret).
• TLS listener disabled — clients connect over plaintext HTTP.
• no auth method binding, no policy attachment — the root token has full capability over every path.

Prod-delta: server -dev runs with in-memory storage, auto-unsealed root token, and TLS disabled; production requires durable HA storage, real seal, TLS listener, and policy-scoped tokens.

Forward link: deploy/local/base/openbao/.

SeaweedFS

The in-cluster S3-compatible object store. SeaweedFS is the lightweight single-binary alternative to running MinIO inside a kind cluster.

Aspect	Value
Image	chrislusf/seaweedfs:3.75@sha256:52d4955fa82e9edd426bf5d73467dfe5ad441ffa9c39aa31e96e1c2988e72755
In-cluster	seaweedfs:8333 (S3 API, ClusterIP Service)
Replicas	1 (StatefulSet)
Roles	master + volume + filer + s3 in one process
Storage	emptyDir

Dev-mode flags / configuration:

• a single SeaweedFS Pod runs weed server -master -volume -filer -s3 — every role inside one process so the dev cluster footprint stays small.
• emptyDir for the master / volume / filer data directories; every restart loses every blob.
• no IAM-style credentials or policies — any client that can reach seaweedfs:8333 can read or write any bucket.

Prod-delta: master, volume, filer, and S3 roles run inside one process with emptyDir storage and no IAM-style credentials; production splits the roles, uses durable PVCs/object storage, scoped credentials, and TLS termination.

Forward link: deploy/local/base/seaweedfs/.

Mimir

The in-cluster metrics backend. Mimir is Grafana's horizontally-scalable Prometheus long-term storage, run here in single-binary mode.

Aspect	Value
Image	grafana/mimir:2.14.3@sha256:046ec57d9776bd27143af22d20201d2c7806dca34254cc45673ced172ed76faf
In-cluster	mimir:9009 (ClusterIP Service)
Replicas	1
Target	all (single-binary)
Storage	emptyDir (filesystem-backed blocks)
Multi-tenancy	disabled

Dev-mode flags / configuration:

• --target=all — every Mimir component (distributor, ingester, querier, …) runs inside one process.
• filesystem-backed block storage on emptyDir; restarts lose every metric.
• --auth.multitenancy-enabled=false — every request maps to the anonymous tenant.
• ingest and query endpoints exposed unauthenticated.

Prod-delta: single-binary target=all with emptyDir storage and no auth; production uses object-store backed blocks (S3/GCS), the multi-target microservice topology, and authenticated multi-tenant ingest.

Forward link: deploy/local/base/mimir/.

Loki

The in-cluster logs backend. Loki is Grafana's log aggregation system, run here in monolithic mode.

Aspect	Value
Image	grafana/loki:3.3.2@sha256:8af2de1abbdd7aa92b27c9bcc96f0f4140c9096b507c77921ffddf1c6ad6c48f
In-cluster	loki:3100 (ClusterIP Service)
Replicas	1
Target	all (monolithic)
Storage	emptyDir (filesystem-backed chunks)
Ring	in-memory

Dev-mode flags / configuration:

• -target=all — every Loki component runs inside one process.
• filesystem-backed chunk storage on emptyDir; restarts lose every log line.
• in-memory ring (no Consul, no memberlist) since there is only one Pod.
• ingest and query endpoints exposed unauthenticated.

Prod-delta: monolithic target=all with emptyDir storage and no auth; production uses object-store backed chunks (S3/GCS), the read/write/backend microservice split, and authenticated multi-tenant ingest.

Forward link: deploy/local/base/loki/.

Crossplane

The management-fleet control plane. The dev manifests ship the core Crossplane v2 install without providers — the local cluster is the management fleet.

Aspect	Value
Image	crossplane/crossplane:v2.0.2@sha256:3a2a2569988aa49bb645ac219d99fb4bba0e3a2f15c39c4965f609acc55cf980
In-cluster	crossplane-webhooks:9443 (ClusterIP Service)
Replicas	1
RBAC	scoped crossplane ClusterRole (upstream chart)
Webhooks	enabled — self-signed TLS bootstrapped by core init

Dev-mode flags / configuration:

• the real crossplane core start controller. The crossplane-init init container runs crossplane core init, which installs the core CRD families (apiextensions / pkg / ops / protection.crossplane.io) and bootstraps the webhook TLS material — Crossplane v2 owns CRD installation, so no CRD bundle is vendored.
• no upstream provider packages are pre-installed; the local cluster is its own empty management fleet.
• the controller's ServiceAccount is bound to the scoped crossplane ClusterRole transcribed verbatim from the upstream chart — not cluster-admin.
• webhook serving certificates are self-signed and bootstrapped by core init into the crossplane-root-ca / crossplane-tls-server / crossplane-tls-client Secrets; production management clusters manage those certificates externally.
• no RBAC manager subprocess — the second upstream Deployment that auto-derives per-Provider ClusterRoles is left to the Helm-chart install path real management clusters use.

Prod-delta: single-replica core install with no RBAC manager and a self-signed webhook CA bootstrapped by core init; production management clusters run the upstream Helm chart with the RBAC manager and externally managed certificates.

Forward link: deploy/local/base/crossplane/.

External Secrets Operator

The ESO controller that pulls Secrets from cluster-external backends in production overlays. The dev cluster runs ESO with no SecretStore wiring so the controller is idle until a real backend is configured.

Aspect	Value
Image	ghcr.io/external-secrets/external-secrets:v0.18.2@sha256:87615c878c0528ea994538d2a6ed87931f8389b9e145f4422891b3ba06430cd7
In-cluster	external-secrets:8080 (ClusterIP Service)
Replicas	1
Leader election	disabled
Metrics	unauthenticated

Dev-mode flags / configuration:

• single replica with --enable-leader-election=false; one controller is enough for a kind cluster.
• metrics endpoint exposed unauthenticated on :8080/metrics so a laptop curl can sample without bearer-token plumbing.
• no SecretStore or ClusterSecretStore resources are shipped in the dev manifests; ESO sits idle until a contributor wires one up manually.

Prod-delta: leader election disabled, replicas=1, and metrics endpoint unauthenticated; production raises replicas above one, re-enables leader election, and gates the metrics endpoint behind authentication.

Forward link: deploy/local/base/external-secrets/.

plexd

The bootstrap-token registration agent — the sibling product the plexsphere API expects to register itself via POST /v1/register on startup. The image tag is templated from /PLEXD_VERSION; bumping plexd is a one-line edit to that file followed by make dev-up-plexd.

Aspect	Value
Image	plexd:$(cat PLEXD_VERSION) (tag-pinned in /PLEXD_VERSION, default v0.1.0)
In-cluster	(no Service — plexd is a client of the plexsphere API)
Replicas	1
Bootstrap	static token from plexd-bootstrap-token Secret
Probes	none (no /healthz in dev mode)

Dev-mode flags / configuration:

• the bootstrap token is a static value mounted from a Kubernetes Secret rather than rotated per-Pod.
• the plexsphere API is reached over plaintext HTTP (in-cluster Service); no mTLS, no SPIFFE identity.
• no liveness or readiness probes — the dev image does not yet expose /healthz.
• registers via POST /v1/register; the make dev-up-plexd target tails the plexd Pod logs and waits for the success line before returning.

Prod-delta: plexd registers with the plexsphere API over plaintext HTTP using a static bootstrap token mounted from a Secret and skips liveness/readiness probes; production uses mTLS with per-workload SPIFFE identities and exposes /healthz.

Forward link: deploy/local/base/plexd/.

Golden-flow chainsaw negative gate

Sibling to the happy-path tests/e2e/dev/golden-flow/chainsaw-test.yaml, the negative gate tests/e2e/dev/golden-flow/negative-bad-token.yaml asserts the dev stack rejects a malformed bootstrap token with the canonical problem-code response and never flips the plexd Deployment to Available=True. Both try: (the deliberate failure path) and the success-path assert: blocks carry an (REQ-009, PX-0021) traceability description so a chainsaw failure surfaces the originating requirement on stderr. The workspace-level companion gates live under tests/workspace/golden_flow_chainsaw_contract_test.go — the pendingGoldenFlowChainsawSteps allowlist + the wired-vs-pending count gate keep the chainsaw test honest about which bootstrap-token contract steps are actually wired.