Local dev stack

This is the per-dependency reference for the local development stack that make dev brings up. Each section describes one service: the SHA-pinned image the dev manifests apply, the dev-mode flags the container boots with, the prod-delta sentence the Pod announces at boot, and a forward link to the manifest source so the reader can diff a flag change against the operator-facing posture in one click .

The line operators tail to confirm a dev posture is active is the same one every Pod under deploy/local/base/ emits (either via a dev-warning init-container or via internal/platform/insecuredefaults.Emit for plexsphere-built Go binaries):

level=WARN msg="insecure default active in dev mode" env=dev dependency=<svc> prod_delta=<sentence>

The integration test tests/integration/dev_stack_insecure_defaults_test.go greps every Pod's logs for this line; a Pod that omits it fails the suite.

See also:

• docs/contributing/dev-stack.md — the entry-point runbook (make dev → seed → sign in with plexctl → golden flow → tear down).
• docs/tutorials/set-up-local-plexsphere.md — the narrower local-kind dev with the manifest-to-CI mapping table and the troubleshooting matrix.
• deploy/local/README.md — the kustomize base reference, including the SpiceDB wiring contract and per-service dev-only postures.

Postgres

The shared OLTP database — both plexsphere and SpiceDB store rows here, in separate schema namespaces. The kind dev introduced this dependency; the dev-stack reuses it unchanged.

Aspect	Value
Image	postgres:16-alpine@sha256:93d55776e04376e19adb2733e3ccebb4392ee7dd86d8ff238503b30fe719c84f
In-cluster	postgres:5432 (ClusterIP Service)
Replicas	1 (StatefulSet)
Storage	emptyDir — wiped on Pod restart
Auth	static password from postgres-credentials Secret

Dev-mode flags / configuration:

• single-replica StatefulSet with emptyDir storage rather than a persistent volume — the cluster lifecycle is bound to the kind container, so durability is not the goal.
• one Postgres instance hosts both the plexsphere and the SpiceDB schemas to keep the dev cluster footprint minimal.
• POSTGRES_PASSWORD ships as a static value in postgres-credentials; every workload reads the same value.

Prod-delta: production runs Postgres as a managed primary + replicas with a real durable volume, per-workload least-privilege roles, TLS-only listener, and rotating credentials sourced from the secrets manager. The SpiceDB schema lives in its own database instance.

Forward link: deploy/local/base/postgres/.

NATS

The messaging plane with JetStream enabled — plexsphere publishes domain events here and the testcontainers integration tier asserts the same JetStream contract. The kind dev introduced this dependency.

Aspect	Value
Image	nats:2.10.20@sha256:97f3bde5637e9cb75c09c15869c79c2c0d909f26c8dd371d169b175a58ff771a
In-cluster	nats:4222 (ClusterIP Service)
Replicas	1 (StatefulSet)
JetStream	enabled, emptyDir-backed
Auth	none — open NATS server

Dev-mode flags / configuration:

• --jetstream enabled; storage directory under emptyDir so streams are recreated on Pod restart.
• single replica, no clustering — dispatch is local to one Pod.
• no TLS, no NKey/JWT auth; any client that can reach nats:4222 can publish to any subject.

Prod-delta: production runs a NATS cluster with at least three replicas, durable JetStream storage on a persistent volume, TLS for both client and inter-node connections, and per-account NKey/JWT auth. Stream replication factor matches the cluster size.

Forward link: deploy/local/base/nats/.

Dex

The OIDC identity provider used by the dev sign-in flow. The kind dev introduced this dependency; the static plexsphere-test client and admin@example.com user mirror the testcontainers fixture internal/platform/testutil/containers/dex.go.

Aspect	Value
Image	ghcr.io/dexidp/dex:v2.41.1@sha256:bc7cfce7c17f52864e2bb2a4dc1d2f86a41e3019f6d42e81d92a301fad0c8a1d
In-cluster	dex:5556 (ClusterIP Service)
Replicas	1
Storage	in-memory (Dex storage.type: memory)
Static user	admin@example.com (password password)
Static client	plexsphere-test

Dev-mode flags / configuration:

• Dex storage.type: memory — every restart wipes the user / refresh token state.
• the static user and static client are baked into the Dex ConfigMap; there is no upstream IdP federation.
• the issuer URL hard-codes http://dex:5556/dex, mapped to the host via the kind extraPortMappings block; plexctl login sign-in flows go through the same URL.

Prod-delta: production wires Dex (or a managed IdP) with a real storage backend (Postgres or Kubernetes CRDs), federation against the corporate IdP, TLS-only issuer, and rotating client secrets managed by the secrets engine.

Forward link: deploy/local/base/dex/.

SpiceDB

The ReBAC engine that the plexsphere API consults on every identity-scoped HTTP endpoint. The wiring contract (preshared-key auth, in-Postgres datastore, gRPC-only on :50051) is documented at length in deploy/local/README.md.

Aspect	Value
Image	authzed/spicedb:v1.39.0@sha256:b98123b44d730cbdfef29494ff9b8ff603d64418affe8fd64d90406ee4e0d9c5
In-cluster	spicedb:50051 (gRPC, ClusterIP Service)
Replicas	1
Datastore	Postgres (shared with plexsphere)
Auth	preshared key (test-key)
Dispatch	disabled (--dispatch-cluster-enabled=false)

Dev-mode flags / configuration:

• --http-enabled=false — the only consumer is the plexsphere API's authzed gRPC client.
• --dispatch-cluster-enabled=false — single replica, no peers.
• --datastore-engine=postgres against the shared cluster Postgres.
• preshared-key auth via spicedb-credentials.SPICEDB_GRPC_PRESHARED_KEY.

Prod-delta: production replaces the preshared-key with mTLS + per-workload SPIFFE identities, runs SpiceDB as a multi-replica HA deployment with cluster dispatch enabled, and uses a dedicated Postgres instance (the production transition is tracked separately).

Forward link: deploy/local/base/spicedb/.

plexsphere

The plexsphere API itself — every /v1/* surface a contributor, an operator, or an API client talks to. The dev overlay boots the API with the full production factory chain so every /v1/* route resolves to its real handler instead of falling through to the earlier 501 Not Implemented stub. Without the env wiring documented below the cmd/plexsphere/*_factory_prod.go factories short-circuit to nil and the affected route returns 501.

Aspect	Value
Image	plexsphere:dev (kind-loaded by make docker-build)
In-cluster	plexsphere:8080 (HTTP, ClusterIP Service)
Replicas	1
Sidecar	dex-localhost-proxy (socat 127.0.0.1:5556 → dex.default.svc:5556)
Init	wait-for-migrate (gates on goose_db_version rows)
Probes	/livez + /readyz

The env-var contract every production factory in cmd/plexsphere/*_factory_prod.go reads at boot. Each row names the manifest source the dev overlay wires it through — Secret stringData, ConfigMap data, or a JSON6902 patch in the dev overlay — plus the factory file that consumes it and the /v1/* surface that returns 501 when it is unset:

Env var	Dev source	Factory	/v1/* surface required by
PLEXSPHERE_DSN	Secret postgres-credentials.DATABASE_URL	(every factory)	every /v1/* surface (datastore wiring)
PLEXSPHERE_PROJECTS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	projects_factory_prod.go	GET /v1/projects (cursor pagination)
PLEXSPHERE_CLOUDS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	clouds_factory_prod.go	GET /v1/clouds (cursor pagination)
PLEXSPHERE_PROVIDER_BUNDLES_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	providerbundles_factory_prod.go	GET /v1/provider-bundles (cursor pagination)
PLEXSPHERE_DOMAINS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	domains_factory_prod.go	GET /v1/domains (cursor pagination)
PLEXSPHERE_IDENTITIES_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	identities_factory_prod.go	GET /v1/domains/{id}/identities
PLEXSPHERE_INVITATIONS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	invitations_factory_prod.go	GET /v1/domains/{id}/invitations (cursor pagination)
PLEXSPHERE_AUDIT_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	audit_factory_prod.go	GET /v1/audit (cursor pagination)
PLEXSPHERE_AUTHZ_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	authz_factory_prod.go	GET /v1/authz/relations (cursor pagination)
PLEXSPHERE_CLOUD_CREDENTIALS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	cloudcredentials_factory_prod.go	GET /v1/clouds/{id}/credentials (cursor pagination)
PLEXSPHERE_CREDENTIALS_CURSOR_HMAC_KEY	(unset — operator-tunable; dev falls back to the identity cursor codec)	credentials_factory_prod.go	GET /v1/projects/{id}/credentials (cursor pagination)
PLEXSPHERE_CREDENTIAL_ASSIGNMENTS_CURSOR_HMAC_KEY	(unset — operator-tunable; dev falls back to the identity cursor codec)	credentialassignments_factory_prod.go	GET /v1/projects/{id}/credential-assignments (cursor pagination)
PLEXSPHERE_SINK_ENABLEMENTS_CURSOR_HMAC_KEY	(unset — operator-tunable; dev falls back to the identity cursor codec)	sinkenablements_factory_prod.go	GET /v1/projects/{id}/sink-enablements (cursor pagination)
PLEXSPHERE_NODES_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	nodes_factory_prod.go	GET /v1/nodes (cursor pagination)
PLEXSPHERE_CAPABILITIES_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	capability_inventory_factory_prod.go	GET /v1/projects/{project_id}/capabilities (cursor pagination)
PLEXSPHERE_HOOKS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	hook_catalog_factory_prod.go	GET /v1/hooks (cursor pagination)
PLEXSPHERE_INTEGRITY_VIOLATIONS_CURSOR_HMAC_KEY	Secret plexsphere-cursor-keys (overlay patch)	integrity_violations_list_factory_prod.go	GET /v1/integrity-violations (cursor pagination)
PLEXSPHERE_NSK_WRAP_KEY_B64	Secret plexsphere-cursor-keys (overlay patch)	registration_factory_prod.go, heartbeat_factory_prod.go	POST /v1/register (NSK wrap) and POST /v1/nodes/{id}/heartbeat (NSK unwrap / plaintext verification)
PLEXSPHERE_SIGNING_PUBLIC_KEY_B64	Secret plexsphere-cursor-keys (overlay patch)	registration_factory_prod.go	POST /v1/register (signing public key)
PLEXSPHERE_SIGNING_KEY_ID	ConfigMap plexsphere-config	registration_factory_prod.go	POST /v1/register (kid)
PLEXSPHERE_CSRF_ALLOWED_ORIGINS	dev overlay JSON6902 patch	auth_factory_prod.go	every state-changing /v1/* (CSRF allowlist)
PLEXSPHERE_SESSION_IDLE_TIMEOUT	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (rolling idle window on the cookie)
PLEXSPHERE_SESSION_MAX_LIFETIME	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (absolute session cap)
PLEXSPHERE_AUTH_STATE_TTL	ConfigMap plexsphere-config	auth_factory_prod.go	/v1/auth/* (OIDC state lifetime)
PLEXSPHERE_PLATFORM_IDP_ISSUER	ConfigMap plexsphere-config (dev Dex issuer)	platform_operator_idp_seed_factory_prod.go	the platform-operator sign-in (seeds the platform-shared binding; the four PLEXSPHERE_PLATFORM_IDP_{ISSUER,CLIENT_ID,CLIENT_SECRET_REF,DISCOVERY_URL} keys travel together — all unset disables the seed, a partial set fails the boot)
PLEXSPHERE_PLATFORM_IDP_CLIENT_ID	ConfigMap plexsphere-config (plexsphere-test)	platform_operator_idp_seed_factory_prod.go	the platform-operator sign-in (OIDC client id on the seeded binding)
PLEXSPHERE_PLATFORM_IDP_CLIENT_SECRET_REF	ConfigMap plexsphere-config (symbolic secret://… reference — never a raw secret)	platform_operator_idp_seed_factory_prod.go	the platform-operator sign-in (secret-store reference on the seeded binding)
PLEXSPHERE_PLATFORM_IDP_DISCOVERY_URL	ConfigMap plexsphere-config (dev Dex discovery URL)	platform_operator_idp_seed_factory_prod.go	the platform-operator sign-in (OIDC discovery URL on the seeded binding)
PLEXSPHERE_PLATFORM_IDP_ALIAS	ConfigMap plexsphere-config (dex-platform)	platform_operator_idp_seed_factory_prod.go	optional chooser label on the seeded binding (PLEXSPHERE_PLATFORM_IDP_JIT_POLICY is the second optional knob; defaults to allow)
PLEXSPHERE_PLATFORM_OPERATOR_SUBJECT	ConfigMap plexsphere-config (dev Dex sub of operator@example.com)	platform_operator_grant_seed_factory_prod.go	the first Platform Operator's authority (seeds the operator user row under the reserved platform-operators Domain plus the platform:plexsphere#admin and managementfleet:fleet#admin grants; unset disables the seed)
PLEXSPHERE_PLATFORM_OPERATOR_EMAIL	ConfigMap plexsphere-config (operator@example.com)	platform_operator_grant_seed_factory_prod.go	optional display email on the seeded operator row (requires PLEXSPHERE_PLATFORM_OPERATOR_SUBJECT)
PLEXSPHERE_PLATFORM_OPERATOR_USER_ID	ConfigMap plexsphere-config (pinned so cloudcredentials.SeedOwnedBy can name the operator as the demo-credential owner)	platform_operator_grant_seed_factory_prod.go	optional user-id pin on the seeded operator row (requires PLEXSPHERE_PLATFORM_OPERATOR_SUBJECT; must stay in lockstep with the demo CloudCredential owner constant)
PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER	ConfigMap plexsphere-config ("true" in dev)	audit_factory_prod.go	/v1/audit, /v1/domains/{id}/identities
PLEXSPHERE_ARTIFACTS_FULCIO_SAN	ConfigMap plexsphere-config (dev fixture SAN)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (release-signing SAN pin)
PLEXSPHERE_ARTIFACTS_OIDC_ISSUER	ConfigMap plexsphere-config (token.actions.githubusercontent.com)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (Fulcio OIDC issuer pin)
PLEXSPHERE_ARTIFACTS_OCI_REGISTRY	ConfigMap plexsphere-config (dev fixture registry path)	artifacts_factory_prod.go	GET /v1/artifacts/plexd/{version} + its {version}/sigstore read surface (upstream OCI source)
PLEXSPHERE_BRIDGE_ACME_DIRECTORY_URL	ConfigMap plexsphere-config (acme-staging-v02.api.letsencrypt.org)	bridge_factory_prod.go	bridge ingress create/update surfaces (ACME directory the certificate-feasibility validator probes before persisting a rule with an ACME account reference)
PLEXSPHERE_BRIDGE_ACME_PROBER_TIMEOUT	(unset — factory default 10s)	bridge_factory_prod.go	bridge ingress create/update surfaces (optional duration bounding each outbound ACME directory probe the certificate-feasibility validator runs; a non-positive or unparseable value is rejected at boot)
PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET	ConfigMap plexsphere-config (plexsphere-action-output)	actions_factory_prod.go	POST /v1/.../actions dispatch + POST /v1/nodes/{id}/executions/{exec_id} callback (object-store bucket the callback service mints over-ceiling output PUT URLs against)
PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	actions_factory_prod.go	POST /v1/.../actions dispatch (absolute base URL the dispatch service stamps onto the per-target callback a Node reports its result to)
PLEXSPHERE_S3_ENDPOINT	ConfigMap plexsphere-config (http://seaweedfs:8333)	actions_factory_prod.go, audit_factory_prod.go	POST /v1/nodes/{id}/executions/{exec_id} callback over-ceiling output PUT (shared S3 object-store endpoint the Action Orchestrator builds its client against at boot; the audit archiver reuses the family)
PLEXSPHERE_S3_REGION	ConfigMap plexsphere-config (us-east-1)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (a missing region fails the dev boot with blobstore: Config.Region is required)
PLEXSPHERE_S3_ACCESS_KEY	ConfigMap plexsphere-config (any — SeaweedFS runs unauthenticated)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (non-secret dev placeholder)
PLEXSPHERE_S3_SECRET_KEY	ConfigMap plexsphere-config (any — SeaweedFS runs unauthenticated)	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (non-secret dev placeholder)
PLEXSPHERE_S3_USE_PATH_STYLE	ConfigMap plexsphere-config ("true")	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (SeaweedFS requires path-style addressing)
PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT	ConfigMap plexsphere-config ("true")	actions_factory_prod.go, audit_factory_prod.go	the Action Orchestrator object-store client (opts in to the plaintext in-cluster http:// SeaweedFS endpoint)
PLEXSPHERE_ACCESS_SIGNER_ENDPOINT	ConfigMap plexsphere-config (plexsphere-signer:8443)	access_factory_prod.go	session POST /v1/projects/{id}/sessions issue + POST /v1/projects/{id}/sessions/{sid}/revoke (host:port of the signer gRPC surface the issuance service signs every session JWT through)
PLEXSPHERE_ACCESS_CALLBACK_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	access_factory_prod.go	session POST /v1/projects/{id}/sessions issue (absolute base URL the issuance service stamps onto the per-session callback a target plexd reports activity to)
PLEXSPHERE_ACCESS_SESSION_SIGNING_SCOPE	Deployment literal plexsphere (platform)	access_factory_prod.go	session POST /v1/projects/{id}/sessions issue (platform signs every session JWT under the single platform key the selfhosted-single signer holds, matching PLEXSPHERE_SIGNER_SCOPE; any other value or unset keeps the per-Domain domain:<id> scope)
PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT	(unset — operator-tunable; dev reaches the in-cluster signer without client material)	access_factory_prod.go	session issue (optional mTLS client cert path for the signer gRPC dial; the unset triple defers the TLS-required error to the signer client)
PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY	(unset — operator-tunable; pairs with the client cert above)	access_factory_prod.go	session issue (optional mTLS client key path for the signer gRPC dial)
PLEXSPHERE_ACCESS_SIGNER_SERVER_CA	(unset — operator-tunable; the signer server CA the dial validates against)	access_factory_prod.go	session issue (optional server CA path for the signer gRPC dial)
PLEXSPHERE_ACCESS_CURSOR_HMAC_KEY	(unset — operator-tunable; dev falls back to the identity cursor codec)	access_factory_prod.go	GET /v1/projects/{id}/sessions (cursor pagination; a >=32-byte hex key enables the HMAC-signed cursor)
PLEXSPHERE_ACCESS_REVOCATION_TTL_FLOOR	(unset — factory default; the repo derives the revocation-list retention from the access domain floor)	access_factory_prod.go	session revoke (optional override documenting the revocation-list retention floor)
PLEXSPHERE_ACCESS_SWEEPER_TICK	(unset — factory default 60s)	access_factory_prod.go	the access-session idle/expiry reconcile / steady-state sweep cadence
PLEXSPHERE_ARTIFACTS_REFRESH_INTERVAL_SECS	(unset — factory default)	artifacts_factory_prod.go	the artifact-refresh reconcile / steady-state sweep cadence
PLEXSPHERE_PROVISIONING_BROKER_ENROL_BASE_URL	ConfigMap plexsphere-config (http://localhost:8080)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (absolute enrol base URL a booting node registers against during cloud-init first boot; REQUIRED when PLEXSPHERE_DSN is set or the broker refuses construction)
PLEXSPHERE_PROVISIONING_BROKER_PLEXD_DOWNLOAD_URL	ConfigMap plexsphere-config (http://localhost:8080/downloads/plexd)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (absolute URL the cloud-init first-boot runcmd fetches the plexd binary from; REQUIRED when PLEXSPHERE_DSN is set)
PLEXSPHERE_PROVISIONING_BROKER_PLEXD_IMAGE	ConfigMap plexsphere-config (ghcr.io/plexsphere/plexd:v0.7.0)	provisioning_broker_factory_prod.go	the provisioning broker reconcile (plexd container image a helm-values blueprint deploys, pinned to the dev PLEXD_VERSION; REQUIRED when PLEXSPHERE_DSN is set)
PLEXSPHERE_CLOUD_CREDENTIALS_KV_MOUNT	ConfigMap plexsphere-config (secret)	cloudcredentials_factory_prod.go	the cloud-credentials Custodian (OpenBao KV-v2 mount prefix; its presence opts the Custodian in so the /v1/cloud-credentials issuance surface can write KV-v2 material; effective only on the dev image, built with the cloudcredentials_openbao + secretstore_dev build tags)
PLEXSPHERE_CLOUD_CREDENTIALS_OPENBAO_ADDRESS	ConfigMap plexsphere-config (http://openbao:8200)	cloudcredentials_factory_prod.go	the cloud-credentials Custodian (in-cluster dev OpenBao endpoint the secretstore client dials)
PLEXSPHERE_CLOUD_CREDENTIALS_ALLOW_INSECURE_MATERIALISER	ConfigMap plexsphere-config (true)	cloudcredentials_factory_prod.go	the cloud-credentials Custodian (dev posture toggle; permits the plaintext / skip-verify connection to the dev OpenBao)
PLEXSPHERE_CLOUD_CREDENTIALS_OPENBAO_TOKEN	Secret openbao-credentials.OPENBAO_DEV_ROOT_TOKEN_ID	cloudcredentials_factory_prod.go	the cloud-credentials Custodian (dev static root token the secretstore_dev AuthDevToken strategy logs in with; rides the Secret, never the ConfigMap)
PLEXSPHERE_MANAGED_PUSH_WRAP_KEY_B64	(unset — opt-in; the managed-push surface stays 501 until set, so the dev overlay does not wire it)	managed_push_factory_prod.go	the /v1/domains/{id}/managed-push attach/push/rollback surface (base64 of the 32-byte AES-256-GCM key the sealer binds the attached kubeconfig under at rest, with the owning Domain UUID as GCM AAD; the wrap key's presence is the managed-push opt-in, so an empty value leaves the six handlers on their 501 stub)
PLEXSPHERE_OBS_MIMIR_QUERY_URL	ConfigMap plexsphere-config (http://mimir:9009/prometheus)	observability_query_factory_prod.go	GET /v1/domains/{id}/metrics/query (Grafana Mimir query API base URL; setting it enables the PromQL instant / range metrics-query proxy, validated at boot, with the addressed Domain injected server-side as the upstream X-Scope-OrgID tenant — points at the in-cluster Mimir, whose Prometheus-compatible API lives under the /prometheus prefix)
PLEXSPHERE_OBS_LOKI_QUERY_URL	ConfigMap plexsphere-config (http://loki:3100)	observability_query_factory_prod.go	GET /v1/domains/{id}/logs/query (Grafana Loki query API base URL; setting it enables the LogQL range logs-query proxy, validated at boot, with the same server-side X-Scope-OrgID tenant injection — points at the in-cluster Loki, whose query API lives at the root)
PLEXSPHERE_OBS_NATS_URL	ConfigMap plexsphere-config (nats://nats:4222)	observability_ingest_factory_prod.go	POST /v1/nodes/{id}/metrics, .../logs, .../audit (the observability-ingest opt-in gate; setting it dispatches the three node-facing ingest routes into the JetStream-buffered production pipeline, and an empty value leaves all three on their 501 stub)
PLEXSPHERE_OBS_LOKI_URL	ConfigMap plexsphere-config (http://loki:3100)	observability_routing_factory_prod.go	the observability routing Loki sink (drains buffered logs/audit records from JetStream onward to the in-cluster Loki, which makes a log line an agent forwards readable back through plexctl logs query; the Mimir remote-write sink stays unset, so buffered metrics remain in JetStream and are not queryable)
PLEXSPHERE_OBS_ALLOW_INTERNAL_SINK_DESTINATIONS	ConfigMap plexsphere-config (true)	observability_routing_factory_prod.go + sinks_transport_factory_prod.go	tenant sink destinations (waives the rule that refuses a sink addressing the platform's own network, at both the authoring half and the dial; set here because every collector reachable from the reader's own machine resolves into a refused range, which is what the Route logs to your own collector lesson delivers to — a dev-stack setting a real installation leaves unset, since the platform dials tenant-stated destinations from inside its own network; only the literal true enables it)

The drift gate tests/workspace/dev_overlay_factory_env_drift_test.go walks every productionXxxConfigFromEnv FuncDecl in cmd/plexsphere/ and fails closed if a required env var is read but not wired into one of the four manifest surfaces above.

Dev-mode flags / configuration:

• the four cursor HMAC keys are deterministic 32-byte hex fixtures (dec0deNN… repeating) — sufficient to clear the per-factory minXxxCursorHMACKeyLen floor and to make the dev cluster's cursor pagination round-trip without rotation. The substring dec0de is the "dev fixture" marker the prod-reference overlay's drift gate forbids.
• NSK_WRAP_KEY_B64 is base64 of the 32-byte ASCII fixture dev-nsk-wrap-key-do-not-use-32by; SIGNING_PUBLIC_KEY_B64 is the Ed25519 public half paired with the plexsphere-signer-seed fixture in the same overlay. The same 32-byte wrap key drives BOTH the registration factory's seal path (internal/identity/nodes/nsk/software.Provider.Issue) AND the heartbeat factory's verify path (internal/identity/nodes/nsk/software.Unwrapper.Unwrap). When the binary boots with PLEXSPHERE_DSN set but PLEXSPHERE_NSK_WRAP_KEY_B64 empty, the heartbeat factory refuses construction with ErrHeartbeatNSKWrapKeyRequired so the misconfiguration surfaces before /readyz lights up green.
• PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER=true opts the audit and identities factories into the deterministic staticPepper fallback because the dev overlay does not run a real OpenBao pepper. The per-Domain pseudonymisation still runs — the fallback is keyed off a fixed seed instead of a rotated material.
• PLEXSPHERE_CSRF_ALLOWED_ORIGINS=http://localhost:8080 is the closed allowlist the CSRF middleware compares incoming Origin headers against on every state-changing cookie-authenticated /v1/* request. With the in-tree dashboard removed, the only browser-facing origin in the dev cluster is the API behind the Gateway at host port 8080. Production overlays MUST supply their own absolute http(s) origin via the same patch shape.
• PLEXSPHERE_AUTH_PLATFORM_DEVICE_LOGIN=true opts the anonymous device-code surface into platform-operator logins so plexctl login --platform works against the platform IdP binding the base deployment seeds via PLEXSPHERE_PLATFORM_IDP_*. The knob defaults to false because device-code initiation is anonymous and remotely phishable (RFC 8628 §5.4); the dev overlay is the deliberate opt-in.
• PLEXSPHERE_AUTH_CALLBACK_URL and PLEXSPHERE_AUTH_VERIFICATION_URL are intentionally left unset in the dev overlay. The auth factory falls back to a request-Host-derived callback and the relative /v1/device default, which the server now renders itself: the control plane hosts the device-verification + approval page in-tree, so interactive device-grant sign-in completes without an external dashboard. Operators who front the stack with their own UI can still point PLEXSPHERE_AUTH_VERIFICATION_URL at it.
• the dex-localhost-proxy socat sidecar exists so the API can call Dex through the same http://localhost:5556/dex issuer the browser uses — see the inline rationale in deploy/local/base/plexsphere/deployment.yaml. The chainsaw bootstrap-seed suite (tests/e2e/bootstrap-seed/chainsaw-test.yaml::signin-against-bootstrap-bindings) mirrors this sidecar pattern in CI to assert that the bootstrap-seeded IdP bindings answer /v1/auth/sign-in in-cluster.

Prod-delta: every Secret value above ships empty in the base manifests at deploy/local/base/plexsphere/secret.yaml; only the dev overlay's JSON6902 patches inject the deterministic dec0de… / dev-nsk-wrap-key-… / Ed25519 fixtures. PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER is "true" only in the dev ConfigMap. Production overlays must inject each cursor HMAC key, the NSK wrap key, and the signing public key through SealedSecrets / External Secrets (rotation surface), and counter-patch PLEXSPHERE_AUDIT_ALLOW_INSECURE_PEPPER to "" so the staticPepper fallback is unreachable. For the Artifact Registry, production points PLEXSPHERE_ARTIFACTS_OCI_REGISTRY at the real signed upstream plexd registry and pins the real release-signing Fulcio SAN (PLEXSPHERE_ARTIFACTS_FULCIO_SAN) and OIDC issuer (PLEXSPHERE_ARTIFACTS_OIDC_ISSUER) the release pipeline signs with; the dev ConfigMap ships fixture values instead, so there is no signed plexd release to verify and the refresh sweep stays inert — Refresh logs and skips, the genesis seed remains unverified, and the binary still boots green. PLEXSPHERE_ARTIFACTS_REFRESH_INTERVAL_SECS is left unset in dev so the factory default cadence applies; operators tune it in production when the upstream release cadence warrants. For the bridge validation pipeline, production points PLEXSPHERE_BRIDGE_ACME_DIRECTORY_URL at the operator's real ACME directory (the production CA the tenant issues certificates from), not the Let's Encrypt staging endpoint the dev ConfigMap ships; the dev value is harmless because the certificate-feasibility probe only fires for an ingress rule that carries an ACME account reference, which the dev overlay never creates. For the Action Orchestrator, production points PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL at the operator's externally reachable API origin (the absolute base a target Node resolves the per-target result callback against) rather than the dev http://localhost:8080, and PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET at the real object-store bucket the over-ceiling output uploads land in; the dev ConfigMap ships a localhost base URL and the plexsphere-action-output bucket name, both of which round-trip against the dev rig's SeaweedFS via the same PLEXSPHERE_S3_* family the audit archive uses. For the Access Orchestrator, production points PLEXSPHERE_ACCESS_CALLBACK_BASE_URL at the operator's externally reachable API origin (the absolute base a target plexd resolves the per-session activity callback against) rather than the dev http://localhost:8080, and mounts the full signer mTLS triple (PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT, PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY, PLEXSPHERE_ACCESS_SIGNER_SERVER_CA) the dev overlay leaves unset because the dev cluster reaches the in-cluster plexsphere-signer:8443 Service over a trusted in-cluster path; production also sets PLEXSPHERE_ACCESS_CURSOR_HMAC_KEY so session-list cursors are HMAC-signed rather than passed through the dev identity codec. Production overlays inherit PLEXSPHERE_AUTH_PLATFORM_DEVICE_LOGIN unset — the fail-closed default that keeps the anonymous device-code surface tenant-only — and enable it only as a deliberate decision to offer plexctl login --platform to their operators. The deploy/local/overlays/prod-reference/ overlay is the canonical reference shape — the workspace gate TestProdReferenceOverlay_DoesNotInheritDevSecrets asserts none of the dev fixture material leaks into it.

Operator-tunable knobs (optional)

The env vars in this section are read with the if raw := strings.TrimSpace(getenv("…")); raw != "" guard, so an unset value falls back to the in-binary default named in the table. The dev overlay does NOT wire them — set them only when the default does not fit the deployment. The workspace gate tests/workspace/dev_overlay_factory_env_drift_test.go's OPTIONAL allowlist pins this set so a new optional knob cannot land without being declared here.

Env var	Factory	Default	Effect when set
PLEXSPHERE_PEERS_ENDPOINT_SWEEP_INTERVAL	peers_factory_prod.go	1m	Steady-state cadence for the Peer endpoint-stale sweeper. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_PEERS_RELAY_ASSIGNER_INTERVAL	peers_factory_prod.go	30s	Heartbeat cadence for the relay-fallback assigner reconcile loop. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_SPIFFE_BUNDLE_TTL	auth_factory_prod.go	15m (spiffe.DefaultBundleCacheTTL)	Cache lifetime for the SPIFFE trust-bundle the JWT-SVID verifier consults on every POST /v1/auth/service/token. Parsed by time.ParseDuration; zero or negative falls back to the package default. Lower values shorten the window between an IdP-side bundle rotation and the API picking it up; higher values reduce upstream load.
PLEXSPHERE_APPROVALS_EXPIRE_TICK	approvals_factory_prod.go	60s (DefaultApprovalsExpireTick)	Cadence of the background sweep that expires stale pending-approval proposals. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_APPROVALS_CURSOR_HMAC_KEY	approvals_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded key that seals the GET /v1/approvals list cursor and binds it to the presenting caller. The cursor is encrypted, not just signed, because the token is minted before the per-row visibility filter runs. When unset the list cursor falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_ACTIONS_CURSOR_HMAC_KEY	actions_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key (≥ 32 bytes) that binds the GET /v1/.../actions list cursor to the presenting caller. When unset the list cursor falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_ACTIONS_TIMEOUT_TICK	actions_factory_prod.go	30s (DefaultActionsTimeoutTick)	Cadence of the background sweep that times out expired live Executions and frees their per-Domain live-execution slots. Parsed by time.ParseDuration; must be positive.
PLEXSPHERE_ACTIONS_LIVE_EXECUTIONS_CAP	actions_factory_prod.go	1000 (DefaultActionsLiveExecutionsCap)	Per-Domain concurrent-execution budget the dispatch service enforces before admitting a new Execution. Parsed as a positive integer.
PLEXSPHERE_ACTIONS_PRESIGN_EXPIRY	actions_factory_prod.go	1h (DefaultActionsPresignExpiry)	Lifetime of an over-ceiling output PUT URL the callback service mints. Parsed by time.ParseDuration; must be positive and below the blobstore.MaxPresignExpiry ceiling.
PLEXSPHERE_ACTIONS_INLINE_OUTPUT_MAX_BYTES	actions_factory_prod.go	16384 (actions.MaxInlineOutputBytes)	Inline-output ceiling above which the callback service mints a presigned object-store PUT URL. The ceiling is a domain invariant with no runtime override, so a value disagreeing with the domain constant is refused at boot; the knob documents the operator-visible default.
PLEXSPHERE_ARTIFACTS_CURSOR_HMAC_KEY	artifacts_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key (≥ 32 bytes) that binds the GET /v1/artifacts/plexd list cursor to the presenting caller. Only consulted once the registry switch PLEXSPHERE_ARTIFACTS_OCI_REGISTRY is set; when unset the per-version GETs still boot and only the paginated list falls back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_BLUEPRINTS_TRACK_INTERVAL_SECS	blueprint_catalogs_factory_prod.go	900 (15m)	Steady-state cadence, in seconds, of the background sweep that re-imports every active, tag-tracking Blueprint catalog source. Parsed as a positive integer; the per-source import is degrade-safe (a down upstream registry is logged, never reds /readyz). Only consulted once the catalog-source surface is enabled via PLEXSPHERE_DSN.
PLEXSPHERE_BLUEPRINTS_OFFICIAL_CATALOG_REF	blueprint_catalogs_factory_prod.go	(empty — no catalog auto-registered)	Fully-qualified OCI reference (registry/repository:tag or registry/repository@sha256:…) of the official upstream Blueprint catalog. When set, the binary registers that source once on first boot, pinned to the official cosign identity and tracking its tag when tag-pinned. A malformed reference is rejected at build time. Only consulted once the catalog-source surface is enabled via PLEXSPHERE_DSN.
PLEXSPHERE_CLOUD_ASSIGNMENTS_CURSOR_HMAC_KEY	cloudassignments_factory_prod.go	(empty — unsigned identity cursor codec)	Hex-encoded HMAC key (≥ 32 decoded bytes) that binds the Cloud Assignment list cursors (GET /v1/projects/{id}/cloud-assignments, GET /v1/clouds/{id}/cloud-assignments) to the presenting caller. When unset the list cursors fall back to the unsigned identity codec. Secret material — should ride in a Secret, never a ConfigMap.
PLEXSPHERE_S3_ENDPOINT	audit_factory_prod.go	(empty — disables the archive uploader)	Endpoint URL the audit-archive S3 client targets. Setting this with the rest of the family below wires the blobstoreArchiveUploader so audit rows beyond the per-Domain retention horizon drain to object storage.
PLEXSPHERE_S3_REGION	audit_factory_prod.go	(empty)	AWS region for the audit-archive bucket. Pass-through to the S3 client Region.
PLEXSPHERE_S3_ACCESS_KEY	audit_factory_prod.go	(empty)	Access-key ID for the audit-archive bucket. Pass-through to the S3 client.
PLEXSPHERE_S3_SECRET_KEY	audit_factory_prod.go	(empty)	Secret access key for the audit-archive bucket. Should ride in a Secret, never a ConfigMap.
PLEXSPHERE_S3_USE_PATH_STYLE	audit_factory_prod.go	false	When "true", instructs the S3 client to use path-style addressing (https://endpoint/bucket/key) instead of virtual-host style. Required for SeaweedFS, MinIO, and other S3-compatible backends that do not host per-bucket subdomains.
PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT	audit_factory_prod.go	false	When "true", permits a plain http:// endpoint on PLEXSPHERE_S3_ENDPOINT. The blobstore client refuses http:// by default (blobstore.ErrInsecureEndpoint); enable this opt-in only for dev or air-gapped TLS-terminating proxies.

The audit-archive S3 family ships unset in the base manifests because the dev rig's SeaweedFS bucket is wired through a different PLEXSPHERE_AUDIT_ARCHIVE_BUCKET path documented under docs/contexts/audit/storage.md; production deployments using AWS S3, GCS-S3, or another vendor object store set the family above to point the audit archive at the real backend.

Forward link: deploy/local/base/plexsphere/.

OpenBao

The cluster-local secrets engine. OpenBao is the BSL-licensed fork of HashiCorp Vault and is the planned production secrets engine for plexsphere.

Aspect	Value
Image	openbao/openbao:2.0.0@sha256:5eedbca9922d85eca5e4bc68c11f968d245b4046641dd4173c1dcff7ae7091aa
In-cluster	openbao:8200 (ClusterIP Service)
Replicas	1
Storage	in-memory (-dev mode)
Root token	static dev-only-root-token

Dev-mode flags / configuration:

• server -dev — boots in development mode: storage is in-memory, the server is pre-unsealed, and a single root token is printed to stdout (matched against the literal in the openbao-secrets Secret).
• TLS listener disabled — clients connect over plaintext HTTP.
• no auth method binding, no policy attachment — the root token has full capability over every path.

Prod-delta: server -dev runs with in-memory storage, auto-unsealed root token, and TLS disabled; production requires durable HA storage, real seal, TLS listener, and policy-scoped tokens.

Forward link: deploy/local/base/openbao/.

SeaweedFS

The in-cluster S3-compatible object store. SeaweedFS is the lightweight single-binary alternative to running MinIO inside a kind cluster.

Aspect	Value
Image	chrislusf/seaweedfs:3.75@sha256:52d4955fa82e9edd426bf5d73467dfe5ad441ffa9c39aa31e96e1c2988e72755
In-cluster	seaweedfs:8333 (S3 API, ClusterIP Service)
Replicas	1 (StatefulSet)
Roles	master + volume + filer + s3 in one process
Storage	emptyDir

Dev-mode flags / configuration:

• a single SeaweedFS Pod runs weed server -master -volume -filer -s3 — every role inside one process so the dev cluster footprint stays small.
• emptyDir for the master / volume / filer data directories; every restart loses every blob.
• no IAM-style credentials or policies — any client that can reach seaweedfs:8333 can read or write any bucket.

Prod-delta: master, volume, filer, and S3 roles run inside one process with emptyDir storage and no IAM-style credentials; production splits the roles, uses durable PVCs/object storage, scoped credentials, and TLS termination.

Forward link: deploy/local/base/seaweedfs/.

Mimir

The in-cluster metrics backend. Mimir is Grafana's horizontally-scalable Prometheus long-term storage, run here in single-binary mode.

Aspect	Value
Image	grafana/mimir:2.14.3@sha256:046ec57d9776bd27143af22d20201d2c7806dca34254cc45673ced172ed76faf
In-cluster	mimir:9009 (ClusterIP Service)
Replicas	1
Target	all (single-binary)
Storage	emptyDir (filesystem-backed blocks)
Multi-tenancy	disabled

Dev-mode flags / configuration:

• --target=all — every Mimir component (distributor, ingester, querier, …) runs inside one process.
• filesystem-backed block storage on emptyDir; restarts lose every metric.
• --auth.multitenancy-enabled=false — every request maps to the anonymous tenant.
• ingest and query endpoints exposed unauthenticated.
• ingester and store-gateway rings pinned to replication_factor: 1. Mimir defaults both rings to 3; a single target=all replica can never form that quorum, so every query would otherwise fail with expanding series: too many unhealthy instances in the ring. The plexsphere query proxy (GET /v1/domains/{id}/metrics/query) reaches Mimir under the /prometheus API prefix.

Prod-delta: single-binary target=all with emptyDir storage and no auth; production uses object-store backed blocks (S3/GCS), the multi-target microservice topology, and authenticated multi-tenant ingest.

Forward link: deploy/local/base/mimir/.

Loki

The in-cluster logs backend. Loki is Grafana's log aggregation system, run here in monolithic mode.

Aspect	Value
Image	grafana/loki:3.3.2@sha256:8af2de1abbdd7aa92b27c9bcc96f0f4140c9096b507c77921ffddf1c6ad6c48f
In-cluster	loki:3100 (ClusterIP Service)
Replicas	1
Target	all (monolithic)
Storage	emptyDir (filesystem-backed chunks)
Ring	in-memory

Dev-mode flags / configuration:

• -target=all — every Loki component runs inside one process.
• filesystem-backed chunk storage on emptyDir; restarts lose every log line.
• in-memory ring (no Consul, no memberlist) since there is only one Pod.
• ingest and query endpoints exposed unauthenticated.

Prod-delta: monolithic target=all with emptyDir storage and no auth; production uses object-store backed chunks (S3/GCS), the read/write/backend microservice split, and authenticated multi-tenant ingest.

Forward link: deploy/local/base/loki/.

Crossplane

The management-fleet control plane. The dev manifests ship the core Crossplane v2 install without providers — the local cluster is the management fleet.

Aspect	Value
Image	crossplane/crossplane:v2.0.2@sha256:3a2a2569988aa49bb645ac219d99fb4bba0e3a2f15c39c4965f609acc55cf980
In-cluster	crossplane-webhooks:9443 (ClusterIP Service)
Replicas	1
RBAC	scoped crossplane ClusterRole (upstream chart)
Webhooks	enabled — self-signed TLS bootstrapped by core init

Dev-mode flags / configuration:

• the real crossplane core start controller. The crossplane-init init container runs crossplane core init, which installs the core CRD families (apiextensions / pkg / ops / protection.crossplane.io) and bootstraps the webhook TLS material — Crossplane v2 owns CRD installation, so no CRD bundle is vendored.
• no provider packages ride the overlay's one-shot apply; the make dev substrate step installs them after core is Healthy (their pkg.crossplane.io CRDs are runtime-served). Today that is provider-kubernetes (renders the cloudless blueprint's in-cluster substrate) and the upjet provider-aws-s3 v2.x family (serves the namespaced s3.aws.m.upbound.io Bucket the floci-backed aws-s3-bucket blueprint composes against the floci AWS emulator). See deploy/local/base/provider-aws/ and deploy/local/base/floci/.
• the controller's ServiceAccount is bound to the scoped crossplane ClusterRole transcribed verbatim from the upstream chart — not cluster-admin.
• webhook serving certificates are self-signed and bootstrapped by core init into the crossplane-root-ca / crossplane-tls-server / crossplane-tls-client Secrets; production management clusters manage those certificates externally.
• no RBAC manager subprocess — the second upstream Deployment that auto-derives per-Provider ClusterRoles is left to the Helm-chart install path real management clusters use.

Prod-delta: single-replica core install with no RBAC manager and a self-signed webhook CA bootstrapped by core init; production management clusters run the upstream Helm chart with the RBAC manager and externally managed certificates.

Forward link: deploy/local/base/crossplane/.

External Secrets Operator

The ESO controller that pulls Secrets from cluster-external backends in production overlays. The dev cluster runs ESO with no SecretStore wiring so the controller is idle until a real backend is configured.

Aspect	Value
Image	ghcr.io/external-secrets/external-secrets:v0.18.2@sha256:87615c878c0528ea994538d2a6ed87931f8389b9e145f4422891b3ba06430cd7
In-cluster	external-secrets:8080 (ClusterIP Service)
Replicas	1
Leader election	disabled
Metrics	unauthenticated

Dev-mode flags / configuration:

• single replica with --enable-leader-election=false; one controller is enough for a kind cluster.
• metrics endpoint exposed unauthenticated on :8080/metrics so a laptop curl can sample without bearer-token plumbing.
• no SecretStore or ClusterSecretStore resources are shipped in the dev manifests; ESO sits idle until a contributor wires one up manually.

Prod-delta: leader election disabled, replicas=1, and metrics endpoint unauthenticated; production raises replicas above one, re-enables leader election, and gates the metrics endpoint behind authentication.

Forward link: deploy/local/base/external-secrets/.

plexd

The bootstrap-token registration agent — the sibling product that registers itself against the plexsphere API via POST /v1/register and then stays resident on the event stream. plexd is not built from this repository: it ships as a released multi-arch image at ghcr.io/plexsphere/plexd, and the dev stack pulls the release by the content digest pinned in /PLEXD_DIGEST, retags it to the version in /PLEXD_VERSION, and side-loads it into the kind node. Bumping plexd edits both files (plus the dev overlay's newTag: and the broker's PLEXSPHERE_PROVISIONING_BROKER_PLEXD_IMAGE, which workspace gates hold equal to the version) followed by make dev-up-plexd.

The stack runs two of these agents, plexd and plexd-2, from the same image and the same environment. Each registers against its own seeded Resource inside the one seeded Project, so each agent's peer projection carries the other and the mesh legs of the golden-flow e2e suite have something to assert against. A single agent's projection would be empty.

Aspect	Value
Image	plexd:v0.7.0 — retagged locally from the linux/<arch> manifest of ghcr.io/plexsphere/plexd@<PLEXD_DIGEST>; the local tag is the /PLEXD_VERSION pin verbatim (currently v0.7.0)
Pull policy	Never — the kind node carries the side-loaded tag, the kubelet never reaches a registry
Command	plexd up (registers, then holds the event stream open)
In-cluster	(no Service — plexd is a client of the plexsphere API at plexsphere:8080)
Replicas	two Deployments, plexd and plexd-2, one replica each
Bootstrap	a one-shot token per Deployment, mounted from the plexd-bootstrap-token and plexd-bootstrap-token-2 Secrets
Probes	liveness on :9101/healthz; deliberately no readiness probe
Volumes	emptyDir on /var/lib/plexd (node identity) and /var/run/plexd (node-API socket), pod-level fsGroup: 65532

Image pipeline

make docker-build-plexd resolves the linux/<daemon-arch> child manifest from ghcr.io/plexsphere/plexd@$(PLEXD_DIGEST), pulls that manifest, and retags it as plexd:$(PLEXD_VERSION); make kind-load-plexd side-loads that tag into the kind node. Nothing is compiled — there is no cmd/plexd in this tree. make dev runs both before it applies the dev overlay (an image that is not on the node before the apply wedges both Pods in ErrImageNeverPull), and make dev-up-plexd re-runs them on every roll.

The pull is by digest, not by tag. plexd release tags are mutable (see below), the pulled content goes straight into a kind node under imagePullPolicy: Never, and dev-stack-smoke runs nightly on GitHub-hosted runners — so a moved tag would substitute what executes in CI and on every contributor's laptop with nothing in the path to notice. A content digest cannot be substituted that way. It does not, however, announce that the tag moved — the registry keeps serving the pinned manifest — so docker-build-plexd resolves the tag and compares it against /PLEXD_DIGEST before pulling, and aborts with exit 65 when the two disagree. plexd releases are not signed today, so there is no cosign verify step; add one in the same change that starts publishing signatures. Resolve a new digest with:

docker buildx imagetools inspect ghcr.io/plexsphere/plexd:v0.7.0

and take the top-level Digest: — the image index, not a per-platform manifest — so the recipe can resolve the right platform on arm64 and amd64.

The pull itself is per-platform: the recipe resolves the pinned index down to its linux/<daemon-arch> child manifest and pulls that, so the local image is rooted at a single-platform manifest. Pulling the index digest directly breaks the side-load on Docker daemons backed by the containerd image store (GitHub-hosted runners, current Docker Desktop defaults): such a pull records the full multi-arch index while fetching only one platform's content, docker save — which kind load docker-image runs under the hood — then emits an archive whose index references children it does not contain, and the kind node's ctr images import --all-platforms rejects it with content digest …: not found. The child digest is read from the pinned index, so the content pin is unchanged.

The second, independent load site is tests/e2e/dev/kind-load.sh, which the nightly e2e-full CI job invokes directly rather than through make dev. It reads the same pin and shells out to the same docker-build-plexd target — also under --skip-build, because that flag only skips compilation and plexd is pulled, not built.

docker-build-plexd pulls on every run, and a failed pull is fatal (exit 69). Falling back to the host Docker cache with a warning was rejected: the warning scrolls past among thousands of CI log lines, the step stays green, and the stale image is loaded and scheduled — exactly the failure mode the pull-every-run posture exists to prevent. On an offline or throttled machine, opt into the cached copy explicitly:

PLEXD_ALLOW_STALE=1 make docker-build-plexd

The escape hatch is refused when CI is set.

Upstream serves v0.7.0, 0.7.0, and latest for the same multi-arch digest — the release workflow publishes the v-prefixed spelling alongside bare semver as of v0.2.0. Releases published before v0.2.0 carry no v-prefixed tag, so moving the pin backwards makes the pull fail with a 404.

Release tags are mutable in practice: v0.2.0 was republished three times in a single day. This is why the pull is digest-pinned: after a republish, the tag resolves to content /PLEXD_DIGEST does not name, and the pre-flight comparison in docker-build-plexd aborts rather than letting the two files drift apart unnoticed. Confirm what the registry currently serves for the tag with:

docker buildx imagetools inspect ghcr.io/plexsphere/plexd:v0.7.0

If the reported index digest differs from /PLEXD_DIGEST, the release was republished: update the file and re-run make dev-up-plexd, which re-pulls, re-arms the seeded bootstrap tokens, and re-rolls the plexd Deployment. The tag does not change in that case, so plexd-2 stays on the Pod it is already running; put both agents on the re-pulled image with make dev-down && make dev.

ghcr.io is throttled from some networks (measured at ~150-300 KB/s on links that pull docker.io at 10-16 MB/s). The host Docker cache survives make dev-down, so the slow path is crossed once per digest rather than once per cluster; PLEXD_ALLOW_STALE=1 reuses that cache on a disconnected machine.

Configuration

Dev-mode flags / configuration:

• configured entirely from the environment — no config file is mounted. A missing config file is non-fatal: the agent logs config file not found, continuing with defaults and overrides at startup and proceeds. That warning is expected in this Pod, not a symptom.
• plexd up, not plexd join. Both register the node, but join exits afterwards, which under a Deployment is a restart loop that never reaches Available=True; up registers and then holds the event stream open.
• the bootstrap token is a static, committed value mounted from a Kubernetes Secret rather than minted per-Pod through the issuer API.
• the plexsphere API is reached over plaintext HTTP (in-cluster Service); no mTLS, no SPIFFE identity.
• a liveness probe on /healthz:9101 and deliberately no readiness probe: /readyz answers 200 only once the WireGuard data plane is up, and a Pod that drops every capability never gets there — it stays at 503 not ready: data plane not configured, so a readiness probe would hold the Pod NotReady forever and break the Available waits make dev and the golden-flow chainsaw scenario depend on.
• the node identity lives in an emptyDir on /var/lib/plexd. data_dir has no environment override, so that mount path must equal the agent's compiled default.

The environment the plexd Pod boots with (deploy/local/base/plexd/deployment.yaml):

Env var	Value	Why
PLEXD_API	http://plexsphere:8080	in-cluster Service DNS of the plexsphere API
PLEXD_MODE	node	stated in the manifest rather than inherited from the default
PLEXD_PROJECT_ID	019100c0-0d0c-7000-8000-000000000d01	the seeded Plexd Dev Project
PLEXD_RESOURCE_HANDLE	urn:dev:node:plexd-1	resolves against the seeded Resource's external_ref
PLEXD_BOOTSTRAP_TOKEN	Secret plexd-bootstrap-token	the one-shot registration credential
PLEXD_POLICY_ENABLED	"false"	the nftables pre-flight runs before registration and aborts in a Pod without CAP_NET_ADMIN
PLEXD_HEALTH_LISTEN	0.0.0.0:9101	the upstream loopback default is unreachable for the kubelet, which probes the Pod IP
PLEXD_ACTIONS_ENABLED	"false"	no action execution in the dev Pod
PLEXD_LOG_LEVEL	debug	log verbosity of the structured agent log. The agent writes its metrics-ingest receipt at debug level, and the golden-flow assert-plexd-ingest step greps that line, so info would leave the ingest leg unobservable

plexd-2 (deploy/local/base/plexd/deployment-2.yaml) boots with the same environment except for two values: PLEXD_RESOURCE_HANDLE is urn:dev:node:plexd-2, and PLEXD_BOOTSTRAP_TOKEN comes from the plexd-bootstrap-token-2 Secret. Both Pods carry the same Project id, which is what puts each of them in the other's peer projection.

Registration prerequisites and recovery

The plexsphere-bootstrap Job (deploy/local/base/plexsphere-bootstrap/configmap.yaml) seeds everything registration needs, so make dev converges without an operator minting anything by hand:

• Project Plexd Dev (slug plexd-dev, fixed id 019100c0-0d0c-7000-8000-000000000d01) under Domain acme-corp. Both agents register into it.
• Resources urn:dev:node:plexd-1 and urn:dev:node:plexd-2, kind node, origin: Adopted — one per Deployment, because plexsphere.nodes.resource_id is UNIQUE and a Resource therefore holds at most one Node.
• two BootstrapToken rows, kind node, 24h TTL, holding the Argon2id hashes of the bootstrap_tokens[0].plaintext and bootstrap_tokens[1].plaintext literals committed in that ConfigMap — the same literals the Pods mount from deploy/local/base/plexd/secret.yaml and deploy/local/base/plexd/secret-2.yaml. The plaintexts are deliberately not reproduced here: this page is published by make docs-publish, and a canonically-formatted bearer string in the rendered output reads as a credential to copy rather than a fixture to regenerate. An integration gate pins each Secret byte-identical to the manifest entry at the same index, and refuses two entries that carry the same plaintext.

The token is one-shot and the identity lives in an emptyDir. A container restart reuses the identity on disk, but a Pod recreation (a kubectl delete pod, a rollout restart, a re-created cluster) starts on an empty data directory and registers again — against a token the previous Pod already consumed, which crash-loops the new Pod.

make dev-up-plexd re-runs the bootstrap Job before it rolls the Deployment, and the Job's token pass walks the whole bootstrap_tokens list, re-planting any row that is no longer redeemable (consumed or expired). That covers both tokens, and it is enough whenever an agent has not yet completed a registration against this cluster — a bumped pin on a stack whose plexd never came up, or a Pod whose earlier registration failed.

The roll itself covers the first agent only: the target restarts deployment/plexd, waits on its msg=registered marker, and leaves plexd-2 running as it is. There is no targeted equivalent for the second agent, and recreating its Pod by hand re-registers it against a Resource that already holds a Node — the failure described next.

make dev-up-plexd is not a recovery after a successful registration. plexsphere.nodes declares resource_id uuid NOT NULL UNIQUE, so the seeded Resource already holds a Node and the re-registration is refused; the target fails at its 60-second msg=registered probe and says so. Reset the fixture with make dev-down && make dev, which is also the only recovery for plexd-2.

A revoked token is never re-armed either: the seed pass fails with a bootstrap_tokens manifest config error rather than reverse an operator revocation. Rotate the committed plaintext in the manifest and the Secret instead.

Operator markers

On every successful start each agent writes the structured line msg=registered, carrying the node_id and mesh_ip it was assigned, to its plexd container's stderr. Both Deployments name that container plexd:

kubectl logs deployment/plexd -c plexd | grep msg=registered
kubectl logs deployment/plexd-2 -c plexd | grep msg=registered

Read the plexd container by name, never --all-containers: the dev-warning init container prints a sentence that itself talks about plexd registering, and a looser match reports success while the agent has said nothing. make dev-up-plexd waits on the first Deployment's marker.

The golden-flow chainsaw suite asserts more than registration. Its assert-plexd-registered step requires both Deployments Available and both markers; five further steps then assert what the released agents do afterwards:

Step	What it asserts
assert-node-state	each Node's reconciliation pull carries exactly its sibling as a peer, with a mesh address and a public key, plus the agent-authored status.mesh state report
assert-heartbeat-healthy	both reachability projections reach healthy under the agents' own heartbeat cadence
assert-plexd-capabilities	one capability-manifest row per Node, carrying the binary version read out of the running image
assert-plexd-rotation	an operator-triggered mesh key rotation the agent completes, observed through the peer's snapshot, with the rotated Node healthy afterwards
assert-plexd-ingest	the metrics-ingest receipt the agent logs once the platform acknowledges a batch

Prod-delta: plexd registers with the plexsphere API over plaintext HTTP using a static bootstrap token mounted from a Secret, runs with network-policy enforcement disabled and unauthenticated health endpoints bound to the Pod network, and keeps its node identity in an ephemeral emptyDir; production registers over mTLS with per-workload SPIFFE identities, enforces policy through nftables and persists the identity on durable storage.

Forward link: deploy/local/base/plexd/.

Running a second stack instance in one cluster

make dev applies the dev overlay with no -n flag, so the whole stack lands in the default namespace. The dev-stack chainsaw scenarios do not: each shells kubectl apply -k … -n <its own namespace> so it owns the world it asserts against, and the dev-stack-smoke CI lane therefore holds two complete stack instances at once — the make dev one in default and the golden-flow one in plexsphere-golden-flow.

The -n flag reaches namespaced objects only, and the overlay's RBAC is cluster-scoped. Every ClusterRoleBinding under deploy/local/base/*/rbac.yaml names its subject ServiceAccount with a literal namespace: default. kustomize's namespace transformer would rewrite those subjects, but it runs only when a kustomization sets namespace:, and no overlay under deploy/local/overlays/ does — namespace selection is deliberately left to the caller.

A stack instance outside default consequently holds no cluster-scoped permission, and the workloads that need one do not log a denial — they crash-loop:

Container	Failure without the grant
crossplane-init	crossplane core init installs the core CRD families and exits Forbidden, so the whole Pod restarts forever.
plexsphere	The fail-closed management-fleet boot probe Gets the per-Project namespace; a Forbidden (unlike a NotFound) is an infrastructure error, so the process exits before it serves.

Editing the shipped bindings is not the fix: a ClusterRoleBinding is cluster-scoped, so pointing one at a second instance takes the permission away from the first. tests/e2e/dev/prepare-stack-instance.sh is what every dev-stack scenario runs right after its apply, and it resolves the two containers above differently.

The control plane gets a per-namespace binding. The script binds the shipped plexsphere-api ClusterRole — the rule set stays single-sourced in the manifests the suite is smoke-testing — under a name prefixed with the target namespace, and stamps it with an ownerReference on that Namespace so Kubernetes garbage-collects it when the scenario tears the namespace down.

The cluster-wide controllers are stood down. Crossplane core and External Secrets are one-per-cluster controllers whose Deployments ride the overlay into every instance. Granting a second Crossplane core its own binding would make it reconcile the same cluster-scoped Provider and Function objects make dev already installed and materialise a runtime Deployment per package in the scenario's namespace — pulling every provider and function image again, inside a readiness window measured in minutes. The script therefore scales both Deployments to zero. That costs the scenario nothing: the plexsphere reconcile's verify gate reads an unavailable substrate controller as "cluster unhealthy", which it degrades over — the Project namespace stays Provisioning — rather than failing the boot probe.

The lockstep is enforced by tests/workspace/dev_stack_instance_rbac_test.go. One gate derives from the manifests every ClusterRoleBinding whose subject ServiceAccount the overlay itself ships, and fails closed when the script neither grants it nor stands the workload holding that ServiceAccount down; a second fails closed when a scenario renders the overlay without preparing its instance at all.

The two provider bindings (plexsphere-provider-aws, plexsphere-provider-kubernetes) fall outside that derived set. Their subjects are the ServiceAccounts the Crossplane package manager creates at install time, and only make dev installs those packages — in default, where the shipped bindings already match.

Reading a failed dev-stack scenario

A chainsaw scenario deletes its namespace during cleanup, so a job-level log collection step in the workflow reaches the cluster after the evidence is gone. The scenario's own failure handler is the only place that still sees it, and describes plus events are not enough: a container in CrashLoopBackOff has no live log, and its exit reason lives in the previous instance's stream.

tests/e2e/dev/dump-namespace.sh is the collector that closes both gaps. It prints the Pod inventory, the namespace events, and — for every container of every Pod, init containers first — both the current and the --previous log. The golden-flow scenario runs it from a test-level catch:, so any step failure produces the same bundle. The script never fails: it runs on the failure path, where half-deleted namespaces and unreadable streams are normal, so it reports what it can and exits 0 rather than replacing the failure under investigation with its own.

Golden-flow chainsaw negative gate

Sibling to the happy-path tests/e2e/dev/golden-flow/chainsaw-test.yaml, the negative gate tests/e2e/dev/golden-flow/negative-bad-token.yaml asserts the dev stack rejects a malformed bootstrap token with the canonical problem-code response and never flips the plexd Deployment to Available=True. Both try: (the deliberate failure path) and the success-path assert: blocks carry an (REQ-009, PX-0021) traceability description so a chainsaw failure surfaces the originating requirement on stderr. The workspace-level companion gates live under tests/workspace/golden_flow_chainsaw_contract_test.go — the pendingGoldenFlowChainsawSteps allowlist + the wired-vs-pending count gate keep the chainsaw test honest about which bootstrap-token contract steps are actually wired.