plexsphere

Appearance

Testing — pyramid, build tags, and the shared harness ​

This document is the canonical entry point for contributors writing tests against plexsphere. It is the operational companion to the Tests and Documentation From the Start rule in CLAUDE.md: a task is only complete once unit, integration, and E2E coverage — plus documentation — have landed alongside the code. This file explains where each tier lives, how the Go build tags gate which tests run, which primitives the shared harness exposes, and why every failure message must carry a Feature-ID suffix.

For the public surface of each shared-harness sub-package, see docs/reference/platform/testutil.md.

The pyramid below has three runnable tiers (unit, integration, E2E). CLAUDE.md's fourth deliverable — documentation — is not a tier in the pyramid; it is enforced separately by the tests/docs/ drift gate described later in this document.

The test pyramid ​

plexsphere follows a strict three-tier test pyramid. Each tier lives in a well-defined location and runs under a known build tag so CI can schedule the tiers independently.

Unit tests ​

  • Where: co-located next to the code under test as *_test.go files, plus the workspace-level drift gates under tests/workspace/ and the documentation gates under tests/docs/.
  • Build tag: none — unit tests build and run under the default tag set.
  • Budget: the whole unit suite must finish in well under a minute on a developer workstation. An individual test should finish in under one second.
  • Scope: a unit test exercises a single function, method, or tight cluster of types without booting a container, an envtest API server, or a Kubernetes kind cluster. Domain builders (see internal/platform/testutil/builders) exist precisely so unit tests can assemble aggregates without network I/O.
  • Command: make test (runs go test ./... across every workspace module) or go test ./internal/... for a single module.

Integration tests ​

  • Where: module-local *_test.go files guarded by the integration build tag, plus the cross-module suites under tests/integration/.
  • Build tag: //go:build integration on the first source line (blank line separates the constraint from the package clause). Every container-backed fixture in internal/platform/testutil/containers and the envtest bootstrapper in internal/platform/testutil/envtest carry the tag so the unit-tag build never pulls in Docker or the envtest binaries.
  • Budget: each fixture individually must be reachable inside startupTimeout (60 s, see internal/platform/testutil/containers/common.go). The StartAll composite fans out concurrently and budgets 90 s end-to-end.
  • Scope: verifying the interaction between the code under test and a real dependency — Postgres, SpiceDB, NATS, OpenBao, SeaweedFS, or a kube-apiserver spun up by envtest. Never a mock.
  • Command: make test-integration (equivalently, go test -tags=integration ./...). The job requires a reachable Docker daemon and, for the envtest path, the setup-envtest binary plus KUBEBUILDER_ASSETS pointing at the extracted etcd / kube-apiserver binaries.

E2E tests ​

  • Where: cluster-level scenarios under tests/e2e/ (Chainsaw-driven Kubernetes acceptance suites).
  • Build tag: none — these tests are not Go code. Chainsaw suites are declared in chainsaw-test.yaml manifests; the plexsphere-owned manifest tests/e2e/chainsaw-config.yaml lists every bounded-context directory make e2e passes to chainsaw test as positional arguments (chainsaw v0.2.14 has no native way to enumerate test directories inside a config file). Playwright specs live in TypeScript under web/tests/e2e/.
  • Budget: individual E2E scenarios budget 120 s end-to-end. Suite-level budgets are the concern of the CI workflow.
  • Scope: one or more bounded contexts deployed together, exercised through their public API — the dashboard for Playwright, the Kubernetes API for Chainsaw. The tier proves the end-user flow without any knowledge of internal package boundaries.
  • Command: make e2e drives Chainsaw through the shared configuration file; npm --prefix web/tests/e2e test drives the Playwright smoke specs.
  • Reserved context placeholders: a few tests/e2e/<context>/ directories carry only a .gitkeep and a README.md while their actual chainsaw-test.yaml lives one level down in a suite sub-directory (e.g. dr/regional-failover/, observability/ingest-burst/, approvals/break-glass/). The context directory is still listed in chainsaw-config.yaml so Chainsaw recurses into it and discovers the nested suite; the placeholder set is pinned by reservedContextDirs in tests/workspace/chainsaw_config_test.go, which also asserts each one keeps its .gitkeep + README.md.

Server-deploying suites: shared core image vs. fixture images ​

Chainsaw suites pull the server binary from one of two image conventions, and the convention dictates how a container's command must be written:

  • The shared core image plexsphere:e2e is the distroless/static build the root Dockerfile produces with --build-arg COMPONENT=plexsphere. It copies the binary to /entrypoint and dispatches through ENTRYPOINT ["/entrypoint"]; it carries no shell and no /usr/local/bin/<name> binaries. A container that runs this image must therefore omit command (the ENTRYPOINT runs the server with the suite's env) or set command: ["/entrypoint", …] (see tests/e2e/ci/chainsaw-test.yaml, which runs /entrypoint --version). Pointing command at /usr/local/bin/plexsphere crash-loops the Pod with stat /usr/local/bin/plexsphere: no such file or directory, which only surfaces as a downstream step timeout.
  • Per-suite fixture images (plexsphere:e2e-signer, plexsphere:e2e-access-sessions, …) are built from a Dockerfile co-located with the suite that COPYs one or more binaries to /usr/local/bin/<name>. Those suites dispatch via an explicit command: ["/usr/local/bin/<name>"] because the fixture bakes that path.

tests/workspace/chainsaw_shared_core_image_command_test.go walks every tests/e2e/** manifest and fails the build if a live (non-skip: true) container running plexsphere:e2e overrides command with anything other than /entrypoint, so a stale override fails a fast go test ./tests/workspace/... instead of a long kind run.

Server-deploying suites: object-store wiring ​

Any Chainsaw suite that boots the production cmd/plexsphere binary with PLEXSPHERE_DSN set must also wire the object store. Since the Action Orchestrator landed, the composition root treats the object store as a hard dependency whenever a DSN is present: cmd/plexsphere/actions_factory_prod.go's productionActionsConfigFromEnv rejects a blank presign bucket or callback URL (returning ErrActionsObjectStoreBucketRequired / ErrActionsCallbackBaseURLRequired), and the boot path logs the rejection at ERROR and exits 1. A suite that sets the DSN without the object-store block therefore lands the pod in CrashLoopBackOff, so kubectl wait --for=condition=Available times out before any HTTP probe runs.

The eight env vars are required on the plexsphere container as a unit whenever PLEXSPHERE_DSN is set:

  • PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET — the bucket the callback service mints presigned PUT URLs against.
  • PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL — the base URL executors call back to with their results.
  • PLEXSPHERE_S3_ENDPOINT — the object-store S3 endpoint.
  • PLEXSPHERE_S3_REGION — the S3 region (any non-empty value for SeaweedFS).
  • PLEXSPHERE_S3_ACCESS_KEY and PLEXSPHERE_S3_SECRET_KEY — S3 credentials.
  • PLEXSPHERE_S3_USE_PATH_STYLE — true, since SeaweedFS serves path-style buckets.
  • PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT — true, since the in-cluster endpoint is plain HTTP.

Each server-deploying fixture supplies the object store with two steps inserted before the plexsphere deploy step:

  1. A deploy-seaweedfs step applies an inline SeaweedFS Deployment + Service. SeaweedFS runs server -master -volume -filer -s3 in a single process; the S3 gateway binds 8333 and --volume.port=8082 dodges the 8080 gateway clash, with /data on an emptyDir for a disposable cluster.
  2. A seed-object-store-bucket step runs a one-shot curl Pod that PUTs the bucket against the S3 gateway and asserts phase: Succeeded. SeaweedFS does not auto-create a bucket on a presigned PUT, so the bucket is created explicitly. The create is idempotent: a 409 against an existing bucket is tolerated.

The SeaweedFS topology and the seed Pod are inlined per fixture rather than shared, even though the bodies are near-identical across suites. Chainsaw v0.2.14 does not interpolate the per-test namespace into applied resource bodies, and every fixture pins an explicit namespace (plexsphere-<suite>), so a single shared resource fragment could not carry the right namespace into each suite. The reference copy lives in tests/e2e/actions/bulk-dispatch/chainsaw-test.yaml; a new suite copies its deploy-seaweedfs and seed-object-store-bucket steps plus the plexsphere container env block, swapping the namespace in the two service URLs.

The co-presence rule is a PR-blocking drift gate: TestChainsawActionsObjectStoreRequired in tests/workspace/chainsaw_actions_object_store_required_test.go walks every tests/e2e/**/chainsaw-test.yaml, and for each container that sets PLEXSPHERE_DSN it requires all eight object-store vars on that same container. A sidecar that sets no DSN is exempt, and a DSN named only in a YAML comment is ignored, so the rule scopes to exactly the containers that boot the server.

Server-deploying suites: access composition-root wiring ​

The object store is not the only composition root a DSN activates. The Access Orchestrator gates the same way, in two boot stages: cmd/plexsphere/access_factory_prod.go's productionAccessConfigFromEnv first rejects a blank signer endpoint or callback base URL (ErrAccessSignerEndpointRequired / ErrAccessCallbackBaseURLRequired); then — once those clear — the factory closure builds the mTLS signer client, where access.NewSignerClient rejects a nil TLS config and Run aborts. A suite wired for the object store but missing the access knobs still CrashLoopBackOffs; the signer is built at boot, so the client cert/key + server CA are mandatory even though this suite never dials it.

Five env vars are required on the plexsphere container as a unit whenever PLEXSPHERE_DSN is set:

  • PLEXSPHERE_ACCESS_SIGNER_ENDPOINT — the signer an issuance dials. The endpoint is only validated for presence at boot (the dial is lazy and the boot probe / /readyz only sweep Postgres, never the signer), so the fixtures pass an in-process loopback placeholder (127.0.0.1:8443).
  • PLEXSPHERE_ACCESS_CALLBACK_BASE_URL — the base URL a target plexd is handed; it reuses the in-cluster plexsphere service URL, matching PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL.
  • PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT, PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY, PLEXSPHERE_ACCESS_SIGNER_SERVER_CA — file paths under /etc/plexsphere/access-signer to the mTLS material buildAccessSignerTLS loads with tls.LoadX509KeyPair. The material only has to parse (the signer is never dialed), so the fixtures mount the deterministic dev-only client leaf + CA reused verbatim from deploy/local/overlays/dev/access-signer-client-secret.yaml (valid through 2036, never a production credential).

So unlike the object store, the access scaffold needs no extra cluster service — but it does need the cert material on disk. Each server-deploying fixture inlines, alongside the plexsphere container env block:

  1. A Secret named plexsphere-access-signer-client carrying client.crt / client.key / ca.crt, applied before the plexsphere Deployment.
  2. A pod volume projecting that Secret, mounted read-only on the plexsphere container at /etc/plexsphere/access-signer.

The Secret + volume are inlined per fixture (with the per-suite namespace) for the same reason the object-store topology is: chainsaw v0.2.14 does not interpolate the per-test namespace into applied resource bodies.

The co-presence rule is a second PR-blocking drift gate: TestChainsawAccessSignerRequired in tests/workspace/chainsaw_access_signer_required_test.go walks every tests/e2e/**/chainsaw-test.yaml, and for each container that sets PLEXSPHERE_DSN it requires the five access vars and the /etc/plexsphere/access-signer mount on that same container — an env-only check would pass a present-but-unmounted cert path that fails LoadX509KeyPair at boot. It is kept separate from the object-store gate because the two roots have distinct reference fixtures; the access reference copy lives in tests/e2e/access/sessions/chainsaw-test.yaml.

Build tags ​

plexsphere's Go test code uses one single build constraint beyond the default: integration. The rules are short and blocking:

  • A test file that depends on a container, envtest, or any external service must declare //go:build integration on line 1.
  • A production file that exports a helper only useful during integration (e.g. the container fixtures in internal/platform/testutil/containers) shares the same build tag so the unit-tag build never imports testcontainers-go.
  • Never invent a new tag without updating this document and the workspace drift gates in tests/workspace/. One tag keeps CI simple: unit runs without -tags, integration runs with -tags=integration, and no third axis exists.

The shared harness ​

The primitives every new bounded context should reach for live under internal/platform/testutil/. The module ships with its own go.mod so test-only dependencies (testcontainers-go, gomega, envtest) never contaminate the main graph.

The five sub-packages are:

  • containers — testcontainer fixtures for Postgres, SpiceDB, NATS, OpenBao, and SeaweedFS, plus the StartAll composite that boots all five concurrently. A standalone StartDex fixture brings up a Dex OIDC server for tests that need an identity provider; it is opt-in and deliberately excluded from StartAll. Pinned image tags are exported as constants so the workspace-level image-pin gate (testcontainers_testutil_image_pins_test.go) can verify drift from a single source of truth.
  • envtest — the controller-runtime envtest bootstrapper. Vendored Crossplane v2 + ESO CRD YAMLs live under envtest/testdata/crds/ and are integrity-checked against a sha256 manifest before the API server starts, so a corrupted stub aborts rather than silently registering a mutated schema.
  • builders — a generic Builder[T] primitive and concrete fluent builders for Domain, Project, Identity, Resource, Label, Policy, Cloud, and Credential. Every builder applies defaults → mutations → invariants in that order, so user-supplied With<Field> calls always win over defaulting.
  • sse — an ed25519-verifying SSE client (NewCapture(ctx, url, verifierKey)) returning a *Capture with Events(), Errors(), ExpectEnvelope, and a leak-free Close.
  • matchers — gomega matchers tailored to plexsphere: BeReady(), HaveCondition(type, status), and EventuallyEmit(src) for draining an SSE capture. Failure messages render the observed conditions (or envelope) as a readable diff and carry no traceability identifier, in line with the rendered-surface rule.

For the full exported surface of each sub-package, see docs/reference/platform/testutil.md.

Rendered-surface rule on every failure ​

Errors, panics, and test-assertion messages must NOT carry the traceability identifier (planwerk feature id, requirement id, story id, or review-item code). The previous (REQ-XXX, PX-YYYY) trailer convention is retired: a kubectl logs reader, a CI failure-log scraper, and a developer reading go test output have no access to the planwerk tracker, so the identifier is opaque to them.

Record the traceability of an assertion in the adjacent comment, not the surfaced string. The full convention and the per-language comment syntax for the comment-only home of an identifier are in traceability-conventions.md.

The rule is enforced by tests/workspace/no_identifiers_in_rendered_surfaces_test.go, which walks every tracked file (Markdown, OpenAPI, YAML, Go, TS / TSX, SQL, proto, Makefile, …) and flags any identifier match outside an allowed comment span. For Go files the gate parses the AST and scans every *ast.BasicLit string literal, in both production and *_test.go files.

A minimal example: the errInvariant helper used by builders carries its requirement linkage in the function's doc-comment, not the error message:

go
// errInvariant returns a builder-invariant error. The originating
// requirement is REQ-003, PX-0004; the error string stays free of
// the identifier so it reads cleanly in a CI failure log.
func errInvariant(field, reason string) error {
    return fmt.Errorf("builder invariant: %s %s", field, reason)
}

The corresponding t.Fatalf shape:

go
if err != nil {
    // Surfaces in CI logs operators read — keep the message clean.
    t.Fatalf("start postgres container: %v", err)
}

Worked example ​

The example below exercises one fixture (Postgres), one builder (DomainBuilder), and one matcher (BeReady()) inside a single integration test. It is deliberately small so you can copy-paste it into a new bounded context's test package and wire up the surrounding logic without hunting for boilerplate. The same shape scales up to the full-stack sample at tests/integration/testutil_sample_test.go, which drives StartAll, every builder, and the SSE capture end-to-end.

go
//go:build integration

package mypackage_test

import (
    "context"
    "testing"

    . "github.com/onsi/gomega"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    "github.com/plexsphere/plexsphere/internal/platform/testutil/builders"
    "github.com/plexsphere/plexsphere/internal/platform/testutil/containers"
    "github.com/plexsphere/plexsphere/internal/platform/testutil/matchers"
)

func TestMyPackage_HappyPath(t *testing.T) {
    RegisterTestingT(t)

    // Fixture — a live Postgres container. Cleanup is registered for
    // us; logs dump on failure automatically.
    pg := containers.StartPostgres(t)

    // Builder — a valid Domain aggregate with default Name and Slug.
    domain, err := builders.NewDomainBuilder().
        WithName("plexsphere-dev").
        WithSlug("plexsphere-dev").
        Build()
    if err != nil {
        // The failure string stays free of any traceability
        // identifier; the requirement this assertion guards is
        // recorded in this comment instead (REQ-003, PX-0004).
        t.Fatalf("build domain: %v", err)
    }

    // Exercise — the system-under-test uses pg.DSN and the domain value.
    conds := applyAndObserveConditions(t, context.Background(), pg.DSN, domain)

    // Matcher — assert the aggregate reached Ready=True. The failure
    // message renders every observed condition as a readable diff.
    Expect([]metav1.Condition(conds)).To(matchers.BeReady())
}

Each of the three pieces stands on its own:

  • containers.StartPostgres(t) returns PostgresFixture{Container, DSN}. The matching helpers for SpiceDB, NATS, OpenBao, and SeaweedFS follow the same shape; containers.StartAll(t) returns a Fixtures bundle when you need more than one.
  • builders.NewDomainBuilder() seeds defaults and invariants. The builder exposes fluent WithName, WithSlug, WithDescription, and WithProjects helpers; every Build() error is a plain, readable invariant message with no traceability identifier in the string — record the requirement an assertion guards in the adjacent comment, never in the surfaced text.
  • matchers.BeReady() accepts []metav1.Condition directly and falls back to reflection for any duck-typed slice with Type and Status fields. Pair it with gomega.Eventually — or the shorthand matchers.EventuallyEmit — when the aggregate settles asynchronously.

Cross-references ​