plexsphere

Appearance

Testing — pyramid, build tags, and the shared harness ​

This document is the canonical entry point for contributors writing tests against plexsphere. It is the operational companion to the Tests and Documentation From the Start rule in CLAUDE.md: a task is only complete once unit, integration, and E2E coverage — plus documentation — have landed alongside the code. This file explains where each tier lives, how the Go build tags gate which tests run, which primitives the shared harness exposes, and why every failure message must carry a Feature-ID suffix.

For the public surface of each shared-harness sub-package, see docs/reference/platform/testutil.md.

The pyramid below has three runnable tiers (unit, integration, E2E). CLAUDE.md's fourth deliverable — documentation — is not a tier in the pyramid; it is enforced separately by the tests/docs/ drift gate described later in this document.

The test pyramid ​

plexsphere follows a strict three-tier test pyramid. Each tier lives in a well-defined location and runs under a known build tag so CI can schedule the tiers independently.

Unit tests ​

  • Where: co-located next to the code under test as *_test.go files, plus the workspace-level drift gates under tests/workspace/ and the documentation gates under tests/docs/.
  • Build tag: none — unit tests build and run under the default tag set.
  • Budget: the whole unit suite must finish in well under a minute on a developer workstation. An individual test should finish in under one second.
  • Scope: a unit test exercises a single function, method, or tight cluster of types without booting a container, an envtest API server, or a Kubernetes kind cluster. Domain builders (see internal/platform/testutil/builders) exist precisely so unit tests can assemble aggregates without network I/O.
  • Command: make test (runs go test ./... across every workspace module) or go test ./internal/... for a single module. make test fans the per-module runs out across the host CPUs — UNIT_PARALLEL bounds the concurrency and defaults to the CPU count, so make test UNIT_PARALLEL=2 throttles it on a shared runner. Each module still writes its own coverage profile and they are merged into one coverage/coverage.out, so the merged profile is identical regardless of the parallelism. Throughput-sensitive modules listed in UNIT_SERIAL_MODULES (currently tests/load, whose driver asserts a sustained request rate) run last and alone on an unsaturated CPU so the parallel batch's contention cannot false-fail them. make test-race fans out the same way under RACE_PARALLEL (half the CPU count by default, because the race detector multiplies each binary's memory footprint).
  • One invocation per checkout: do not run two unit-tier invocations (make test plus a second go test ./tests/workspace/...) in the same checkout at the same time. The workspace gates are not concurrency-safe within one tree: the OpenAPI lint gates run npm ci into the shared tools/openapi/node_modules, and the codegen drift gates regenerate into the real tree and diff against git — two overlapping runs corrupt the npm dir and race the regeneration snapshots into false failures.

Integration tests ​

  • Where: module-local *_test.go files guarded by the integration build tag, plus the cross-module suites under tests/integration/.

  • Build tag: //go:build integration on the first source line (blank line separates the constraint from the package clause). Every container-backed fixture in internal/platform/testutil/containers and the envtest bootstrapper in internal/platform/testutil/envtest carry the tag so the unit-tag build never pulls in Docker or the envtest binaries.

  • Budget: each fixture individually must be reachable inside startupTimeout (60 s, see internal/platform/testutil/containers/common.go). The telemetry backends (StartMimir, StartLoki) get a longer backendStartupTimeout (180 s) because their single-binary boot only flips /ready to 200 once the ingester ring has formed. The StartAll composite fans out concurrently and budgets 90 s end-to-end.

  • Scope: verifying the interaction between the code under test and a real dependency — Postgres, SpiceDB, NATS, OpenBao, SeaweedFS, or a kube-apiserver spun up by envtest. Never a mock.

  • Command: make test-integration (equivalently, go test -tags=integration ./...). The job requires a reachable Docker daemon and, for the envtest path, the setup-envtest binary plus KUBEBUILDER_ASSETS pointing at the extracted etcd / kube-apiserver binaries.

  • Postgres: clone a database, don't boot a container. The suite runs ONE Postgres container per test binary and gives each test a private database cloned off a pre-migrated template (CREATE DATABASE … TEMPLATE). A clone is a file-copy of an already-initialised data directory inside a running server — milliseconds — where a container boot costs the image start plus the double "ready to accept connections" wait. Pick the helper by what the test actually needs:

    HelperGives youUse it when
    containers.ClonePostgres(t)a private database at the embedded migration headthe default — the test needs a schema, not a server
    containers.ClonePostgresEmpty(t)a private, unmigrated databasethe test drives migrations.Up / Down itself and must walk from version zero
    containers.StartPostgres(t, opts…)a private CONTAINERthe test needs a server: it stops and restarts it, dumps the whole instance with pg_dumpall, creates cluster-level roles, or creates fixed-name databases

    A private container is also what a test needs when its verdict is a wall-clock comparison: on the shared container a neighbouring test's burst lands on the same postmaster, and the timing it inflates is reported as a failure of the code under test. The cross-Domain registration concurrency gate is the one test in the suite that does this.

    A clone is fully isolated (its own catalog), so nothing about test authoring changes: calling migrations.Up on a clone is still valid and is simply a no-op. Any binary that clones must terminate the shared container from a TestMain (containers.TerminateSharedPostgres()) — the container's lifetime is the binary, so no t.Cleanup can own it. See tests/integration/main_test.go.

    The primitive is not exempt from the tier it serves. Its own suite in internal/platform/testutil/containers carries the integration tag — it boots a container — so the untagged unit fan-out never compiles it. make test-integration runs it as its first leg; that is the only place it executes.

    When a change alters the suite's shape, measure it: record the before/after make test-integration wall-clock in the pull-request description. That number is the whole reason the pattern exists.

E2E tests ​

  • Where: cluster-level scenarios under tests/e2e/ (Chainsaw-driven Kubernetes acceptance suites).

  • Build tag: none — these tests are not Go code. Chainsaw suites are declared in chainsaw-test.yaml manifests; the plexsphere-owned manifest tests/e2e/chainsaw-config.yaml lists every bounded-context directory make e2e passes to chainsaw test as positional arguments (chainsaw v0.2.14 has no native way to enumerate test directories inside a config file).

  • Budget: individual E2E scenarios budget 120 s end-to-end. Suite-level budgets are the concern of the CI workflow.

  • Scope: one or more bounded contexts deployed together, exercised through their public API — the Kubernetes API, via Chainsaw. The tier proves the end-user flow without any knowledge of internal package boundaries.

  • Command: make e2e drives Chainsaw through the shared configuration file.

  • Tiers: the catalog carries two lists, and the two CI lanes read one each.

    ListTargetCI jobContents
    testDirsmake e2ee2e-full (nightly + manual dispatch)every bounded-context area — the full matrix
    coreTestDirsmake e2e-coree2e (every pull request)the core journeys whose breakage is release-blocking

    coreTestDirs entries are individual scenario directories, not whole areas, so a pull request's e2e wall-clock stays at the core journeys — sign-in roundtrip, provisioning lifecycle, backup/restore/regional failover, and signer rotation — instead of the full matrix. make e2e-core side-loads only the images those journeys reference. Both lists are drift-gated: TestChainsawConfig_ListsAllE2EDirs pins testDirs to the on-disk layout, and TestChainsawConfig_CoreDirsAreValidSubset pins every coreTestDirs entry to a real scenario inside a catalogued area — so the PR tier can never exercise a scenario the nightly matrix misses.

    Adding a scenario to coreTestDirs puts it on every pull request's critical path. Add it only when a regression there must not reach main; everything else rides the nightly matrix.

  • Fixture-image fan-out: a scenario that ships its own image pins it with imagePullPolicy: Never, so make e2e must side-load that image into the kind cluster first. The set of images to build and load is a third catalog list, kindLoadDirs (and coreKindLoadDirs for the PR tier) — deliberately curated, not every kind-load.sh on disk, since skip-gated scaffolds and heavy substrates stay out of the matrix. The make e2e recipe loops over that list rather than carrying a per-suite stanza, so wiring a new fixture image is a one-line catalog edit plus a ≤5-line kind-load.sh shim. Each shim execs the shared build-and-load body at tests/e2e/testutil/build-and-load.sh, which carries the docker build + kind load every suite once copied verbatim; DEV_STACK=1 skips the whole fan-out and reuses the running dev cluster.

  • Reserved context placeholders: a few tests/e2e/<context>/ directories carry only a .gitkeep and a README.md while their actual chainsaw-test.yaml lives one level down in a suite sub-directory (e.g. dr/regional-failover/, observability/ingest-burst/, approvals/break-glass/). The context directory is still listed in chainsaw-config.yaml so Chainsaw recurses into it and discovers the nested suite; the placeholder set is pinned by reservedContextDirs in tests/workspace/chainsaw_config_test.go, which also asserts each one keeps its .gitkeep + README.md.

  • Probe helpers: a probe imports tests/e2e/internal/probekit and writes only its scenario. The kit owns the parts every probe would otherwise write identically — Main (subcommand dispatch plus the signal-cancelled context), EnvOr / RequireEnv, EmitJSON, RunMigrate, WaitForLive, ResolveProjectAndUser, and IssueBootstrapToken (through the production issuer, not a hand-rolled row insert). The probe owns its subcommands, its seed SQL, and its assertions.

    Two things chainsaw asserts on must not drift, so the kit preserves both exactly. EmitJSON always terminates its line — the assert steps read kubectl logs one event per line, and a missing newline merges two events into an unparseable one. And exit codes are passed IN via probekit.ExitCodes rather than exported as a canonical set: the probes genuinely disagree beyond ExitOK and ExitUsage, and each number is asserted in that scenario's chainsaw-test.yaml.

    Because each probe is its own module, the kit reaches it through a relative replace directive. A new probe module must be registered in go.work; its COPY … go.mod line in the root Dockerfile and in every e2e Dockerfile is then produced by make generate-dockerfiles, which regenerates the go.mod COPY preamble in all Go-builder Dockerfiles from the go.work module list, so the preamble never drifts by hand. TestDockerfileCopyListCoversEveryModule, TestE2EDockerfilesCoverEveryWorkspaceModule, and TestGoBuilderDockerfilePreamblesAreGenerated fail the build if the preamble falls out of lockstep, and their failure message points at make generate-dockerfiles.

Console front-end tests ​

The in-tree Console (console/) is a Vite + React app, so its tests run under vitest rather than go test. Component and helper tests live next to the code as *.test.ts / *.test.tsx, exercise the unit and integration tiers (a CSRF helper as a unit, the sign-in view and the no-flicker auth gate as integration), and run in a jsdom environment with React Testing Library. The suite is path-filtered in CI as the console-unit job and mirrored locally by make console-unit (equivalently npm --prefix console run test:run). The Console's end-to-end tier — a Dex-fronted Playwright sign-in flow — is tracked as a follow-up; there is no in-tree Playwright harness to extend yet, so this slice ships the unit and integration tiers only.

Component tests share one render harness, renderWithProviders from console/src/test/render.tsx, rather than each file hand-building a QueryClient and its provider wrapper. renderWithProviders(ui, options) mounts ui under a QueryClient with retries disabled (so a mocked error surfaces on the first attempt instead of stalling the test), and composes the optional layers a test asks for: activity: true adds the ActivityProvider, activity: { toaster: true } also mounts the ActivityToaster, and route mounts the component inside a TanStack memory router whose siblings are the navigation targets a link resolves to. A test that seeds the cache passes its own client; a renderHook test reaches for the createTestQueryClient and TestProviders building blocks the same module exports. This sits alongside the permission-aware withAuthz helper in console/src/test/authz.ts, which answers the whoami / authz reads a gated control makes.

Server-deploying suites: shared core image vs. fixture images ​

Chainsaw suites pull the server binary from one of two image conventions, and the convention dictates how a container's command must be written:

  • The shared core image plexsphere:e2e is the distroless/static build the root Dockerfile produces with --build-arg COMPONENT=plexsphere. It copies the binary to /entrypoint and dispatches through ENTRYPOINT ["/entrypoint"]; it carries no shell and no /usr/local/bin/<name> binaries. A container that runs this image must therefore omit command (the ENTRYPOINT runs the server with the suite's env) or set command: ["/entrypoint", …] (see tests/e2e/ci/chainsaw-test.yaml, which runs /entrypoint --version). Pointing command at /usr/local/bin/plexsphere crash-loops the Pod with stat /usr/local/bin/plexsphere: no such file or directory, which only surfaces as a downstream step timeout.
  • Per-suite fixture images (plexsphere:e2e-signer, plexsphere:e2e-access-sessions, …) are built from a Dockerfile co-located with the suite that COPYs one or more binaries to /usr/local/bin/<name>. Those suites dispatch via an explicit command: ["/usr/local/bin/<name>"] because the fixture bakes that path.

tests/workspace/chainsaw_shared_core_image_command_test.go walks every tests/e2e/** manifest and fails the build if a live (non-skip: true) container running plexsphere:e2e overrides command with anything other than /entrypoint, so a stale override fails a fast go test ./tests/workspace/... instead of a long kind run.

Server-deploying suites: object-store wiring ​

Any Chainsaw suite that boots the production cmd/plexsphere binary with PLEXSPHERE_DSN set must also wire the object store. Since the Action Orchestrator landed, the composition root treats the object store as a hard dependency whenever a DSN is present: cmd/plexsphere/actions_factory_prod.go's productionActionsConfigFromEnv rejects a blank presign bucket or callback URL (returning ErrActionsObjectStoreBucketRequired / ErrActionsCallbackBaseURLRequired), and the boot path logs the rejection at ERROR and exits 1. A suite that sets the DSN without the object-store block therefore lands the pod in CrashLoopBackOff, so kubectl wait --for=condition=Available times out before any HTTP probe runs.

The eight env vars are required on the plexsphere container as a unit whenever PLEXSPHERE_DSN is set:

  • PLEXSPHERE_ACTIONS_OBJECT_STORE_BUCKET — the bucket the callback service mints presigned PUT URLs against.
  • PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL — the base URL executors call back to with their results.
  • PLEXSPHERE_S3_ENDPOINT — the object-store S3 endpoint.
  • PLEXSPHERE_S3_REGION — the S3 region (any non-empty value for SeaweedFS).
  • PLEXSPHERE_S3_ACCESS_KEY and PLEXSPHERE_S3_SECRET_KEY — S3 credentials.
  • PLEXSPHERE_S3_USE_PATH_STYLE — true, since SeaweedFS serves path-style buckets.
  • PLEXSPHERE_S3_ALLOW_INSECURE_ENDPOINT — true, since the in-cluster endpoint is plain HTTP.

Each server-deploying fixture supplies the object store with two steps inserted before the plexsphere deploy step:

  1. A deploy-seaweedfs step applies an inline SeaweedFS Deployment + Service. SeaweedFS runs server -master -volume -filer -s3 in a single process; the S3 gateway binds 8333 and --volume.port=8082 dodges the 8080 gateway clash, with /data on an emptyDir for a disposable cluster.
  2. A seed-object-store-bucket step runs a one-shot curl Pod that PUTs the bucket against the S3 gateway and asserts phase: Succeeded. SeaweedFS does not auto-create a bucket on a presigned PUT, so the bucket is created explicitly. The create is idempotent: a 409 against an existing bucket is tolerated.

The SeaweedFS topology and the seed Pod are inlined per fixture rather than shared, even though the bodies are near-identical across suites. Chainsaw v0.2.14 does not interpolate the per-test namespace into applied resource bodies, and every fixture pins an explicit namespace (plexsphere-<suite>), so a single shared resource fragment could not carry the right namespace into each suite. The reference copy lives in tests/e2e/actions/bulk-dispatch/chainsaw-test.yaml; a new suite copies its deploy-seaweedfs and seed-object-store-bucket steps plus the plexsphere container env block, swapping the namespace in the two service URLs.

The co-presence rule is a PR-blocking drift gate: TestChainsawActionsObjectStoreRequired in tests/workspace/chainsaw_actions_object_store_required_test.go walks every tests/e2e/**/chainsaw-test.yaml, and for each container that sets PLEXSPHERE_DSN it requires all eight object-store vars on that same container. A sidecar that sets no DSN is exempt, and a DSN named only in a YAML comment is ignored, so the rule scopes to exactly the containers that boot the server.

Server-deploying suites: access composition-root wiring ​

The object store is not the only composition root a DSN activates. The Access Orchestrator gates the same way, in two boot stages: cmd/plexsphere/access_factory_prod.go's productionAccessConfigFromEnv first rejects a blank signer endpoint or callback base URL (ErrAccessSignerEndpointRequired / ErrAccessCallbackBaseURLRequired); then — once those clear — the factory closure builds the mTLS signer client, where access.NewSignerClient rejects a nil TLS config and Run aborts. A suite wired for the object store but missing the access knobs still CrashLoopBackOffs; the signer is built at boot, so the client cert/key + server CA are mandatory even though this suite never dials it.

Five env vars are required on the plexsphere container as a unit whenever PLEXSPHERE_DSN is set:

  • PLEXSPHERE_ACCESS_SIGNER_ENDPOINT — the signer an issuance dials. The endpoint is only validated for presence at boot (the dial is lazy and the boot probe / /readyz only sweep Postgres, never the signer), so the fixtures pass an in-process loopback placeholder (127.0.0.1:8443).
  • PLEXSPHERE_ACCESS_CALLBACK_BASE_URL — the base URL a target plexd is handed; it reuses the in-cluster plexsphere service URL, matching PLEXSPHERE_ACTIONS_CALLBACK_BASE_URL.
  • PLEXSPHERE_ACCESS_SIGNER_CLIENT_CERT, PLEXSPHERE_ACCESS_SIGNER_CLIENT_KEY, PLEXSPHERE_ACCESS_SIGNER_SERVER_CA — file paths under /etc/plexsphere/access-signer to the mTLS material buildAccessSignerTLS loads with tls.LoadX509KeyPair. The material only has to parse (the signer is never dialed), so the fixtures mount the deterministic dev-only client leaf + CA reused verbatim from deploy/local/overlays/dev/access-signer-client-secret.yaml (valid through 2036, never a production credential).

So unlike the object store, the access scaffold needs no extra cluster service — but it does need the cert material on disk. Each server-deploying fixture inlines, alongside the plexsphere container env block:

  1. A Secret named plexsphere-access-signer-client carrying client.crt / client.key / ca.crt, applied before the plexsphere Deployment.
  2. A pod volume projecting that Secret, mounted read-only on the plexsphere container at /etc/plexsphere/access-signer.

The Secret + volume are inlined per fixture (with the per-suite namespace) for the same reason the object-store topology is: chainsaw v0.2.14 does not interpolate the per-test namespace into applied resource bodies.

The co-presence rule is a second PR-blocking drift gate: TestChainsawAccessSignerRequired in tests/workspace/chainsaw_access_signer_required_test.go walks every tests/e2e/**/chainsaw-test.yaml, and for each container that sets PLEXSPHERE_DSN it requires the five access vars and the /etc/plexsphere/access-signer mount on that same container — an env-only check would pass a present-but-unmounted cert path that fails LoadX509KeyPair at boot. It is kept separate from the object-store gate because the two roots have distinct reference fixtures; the access reference copy lives in tests/e2e/access/sessions/chainsaw-test.yaml.

Build tags ​

plexsphere's Go test code uses one build constraint on the test axis: integration. The rules are short and blocking:

  • A test file that depends on a container, envtest, or any external service must declare //go:build integration on line 1.
  • A production file that exports a helper only useful during integration (e.g. the container fixtures in internal/platform/testutil/containers) shares the same build tag so the unit-tag build never imports testcontainers-go.
  • Never invent a new tag on this axis without updating this document and the workspace drift gates in tests/workspace/. One tag keeps CI simple: unit runs without -tags, integration runs with -tags=integration.

Separately, three tags gate optional production code rather than tests: aws_kms, gcp_kms, and azure_kv compile the cloud KMS providers under internal/signing/providers/kms/, so the default build never pulls three cloud SDKs into the binary.

That has a consequence worth stating plainly: their packages do not compile under the untagged build, so the untagged go test sees nothing — not a failure, not zero coverage, nothing at all. A tagged pass is the only way their tests run. make test appends one after the module fan-out, so the contract tests execute in the blocking unit job and their profile is merged into the coverage report like any other. To run them alone:

bash
cd internal/signing && go test -tags 'aws_kms gcp_kms azure_kv' ./providers/kms/...

Coverage measurement ​

The Codecov signal is the unit tier only. make test writes one profile per workspace module into coverage/, merges them into coverage/coverage.out, and the unit CI job uploads that. The integration and e2e tiers deliberately produce no profiles.

That split is a decision, not an oversight, so read a low coverage number on a package with the split in mind before acting on it.

internal/platform/db is the worked example: it reports around 4%. The package is not untested — its pure surfaces (config parsing, error classification, DSN redaction) carry unit tests, and everything else about it (the pool's success path, migrations actually executing, row-level-security reads, and the whole generated gen/ sqlc layer) requires a live Postgres and is exercised thoroughly from tests/integration/. None of that exercise reaches the number, because the integration tier emits no profile.

Why not merge the integration profiles in? It would make the number larger without making it more useful: the merged figure would move whenever a container failed to boot, and the tiers answer different questions — "is this logic specified" (unit) versus "does this integrate with a real dependency" (integration). Documenting the split costs one section; wiring profile collection through the sharded, container-backed integration job costs a permanent piece of CI machinery whose output nobody would act on differently. If a package's unit coverage looks alarming, check whether its behaviour is integration-exercised before concluding it is untested.

The shared harness ​

The primitives every new bounded context should reach for live under internal/platform/testutil/. The module ships with its own go.mod so test-only dependencies (testcontainers-go, gomega, envtest) never contaminate the main graph.

The six sub-packages are:

  • containers — testcontainer fixtures for Postgres, SpiceDB, NATS, OpenBao, and SeaweedFS, plus the StartAll composite that boots all five concurrently. A standalone StartDex fixture brings up a Dex OIDC server for tests that need an identity provider; StartMimir and StartLoki bring up the single-binary telemetry backends the observability routing suite exports metrics and log streams into. All three are opt-in and deliberately excluded from StartAll; the telemetry backends boot against the relaxed backendStartupTimeout rather than the tight per-fixture startupTimeout. WithStableHostPort pins a fixture's published port to a fixed host port so its DSN or URL survives a container stop/start round-trip — the failure-mode suites that restart a backend in place pass it to StartPostgres, StartNATS, and (via WithSpiceDBCustomizer) StartSpiceDB. Pinned image tags are exported as constants so the workspace-level image-pin gate (testcontainers_testutil_image_pins_test.go) can verify drift from a single source of truth: tests/integration consumes these fixtures directly and no longer maintains its own duplicated copy.
  • envtest — the controller-runtime envtest bootstrapper. Vendored Crossplane v2 + ESO CRD YAMLs live under envtest/testdata/crds/ and are integrity-checked against a sha256 manifest before the API server starts, so a corrupted stub aborts rather than silently registering a mutated schema.
  • builders — a generic Builder[T] primitive and concrete fluent builders for Domain, Project, Identity, Resource, Label, Policy, Cloud, and Credential. Every builder applies defaults → mutations → invariants in that order, so user-supplied With<Field> calls always win over defaulting.
  • sse — an ed25519-verifying SSE client (NewCapture(ctx, url, verifierKey)) returning a *Capture with Events(), Errors(), ExpectEnvelope, and a leak-free Close.
  • matchers — gomega matchers tailored to plexsphere: BeReady(), HaveCondition(type, status), and EventuallyEmit(src) for draining an SSE capture. Failure messages render the observed conditions (or envelope) as a readable diff and carry no traceability identifier, in line with the rendered-surface rule.
  • authztest — a concurrency-safe authorizer double for the uniform ReBAC Check(ctx, subject, relation, object, caveat) seam. authztest.AllowAll() grants every check, authztest.Deny(err) fails every check with err, and authztest.WithFn(fn) decides per tuple; the *Fake records each call for Calls() to return. Reach for it in unit tests instead of a per-package fakeAuthorizer. It is a unit-tier double only — integration tests exercise the real, container-backed SpiceDB, and a workspace gate bans importing authztest from tests/integration. A fake that must record which tuples were checked, selectively grant per (relation, object), or return a context-local denial sentinel keeps its own local type.

For the full exported surface of each sub-package, see docs/reference/platform/testutil.md.

Two shared test primitives live outside testutil because they mirror a production seam rather than a harness concern:

  • clocktest — a concurrency-safe fake clock in internal/platform/clock/clocktest. clocktest.NewFake(now) returns a *Fake whose Now() / Set(t) / Advance(d) are all mutex-guarded, so it satisfies the interface{ Now() time.Time } Clock port every bounded context injects and can be driven safely from several goroutines. Reach for it instead of hand-rolling a per-package fakeClock / fixedClock / stubClock. A fake that also records Sleep calls (the archiver and authz-sync back-off loops) keeps its own local type — clocktest.Fake covers only the Now seam.
  • dbtest — shared pgx test doubles for repository unit tests in internal/platform/db/dbtest. dbtest.FailingBeginner(err) returns a beginner whose BeginTx fails with err, so a repository unit test can drive a method past validation into its transaction wrapper without a live database. Reach for it instead of a per-package fakeBeginner. A double that must also return a usable transaction or record that BeginTx was reached keeps its own local type.
  • auditport — the canonical services-layer audit seam in internal/platform/db/auditport. A bounded context whose application service has adopted the kit emits the shared auditport.Entry (a closed Reason enum, not a free-form outcome string, and a names-only CaveatContext []string) through the auditport.Sink port. A unit test for such a service substitutes a recording sink that implements auditport.Sink — capturing []auditport.Entry — and asserts on Reason == auditport.ReasonGranted and the exact caveat NAMES, rather than on an outcome string or a map[string]any of caveat values. When you assert an audit row, check the service's Reason on the recording auditport.Sink; the transport read / denial rows keep their own outcome-string entry type and are observed on a separate transport sink. Contexts that have NOT adopted the kit (see the per-type DECISION blocks naming auditport) keep a local AuditEntry / AuditSink, so match the recording double to the port the service under test actually consumes.

Rendered-surface rule on every failure ​

Errors, panics, and test-assertion messages must NOT carry the traceability identifier (planwerk feature id, requirement id, story id, or review-item code). The previous (REQ-XXX, PX-YYYY) trailer convention is retired: a kubectl logs reader, a CI failure-log scraper, and a developer reading go test output have no access to the planwerk tracker, so the identifier is opaque to them.

Record the traceability of an assertion in the adjacent comment, not the surfaced string. The full convention and the per-language comment syntax for the comment-only home of an identifier are in traceability-conventions.md.

The rule is enforced by tests/workspace/no_identifiers_in_rendered_surfaces_test.go, which walks every tracked file (Markdown, OpenAPI, YAML, Go, TS / TSX, SQL, proto, Makefile, …) and flags any identifier match outside an allowed comment span. For Go files the gate parses the AST and scans every *ast.BasicLit string literal, in both production and *_test.go files.

A minimal example: the errInvariant helper used by builders carries its requirement linkage in the function's doc-comment, not the error message:

go
// errInvariant returns a builder-invariant error. The originating
// requirement is REQ-003, PX-0004; the error string stays free of
// the identifier so it reads cleanly in a CI failure log.
func errInvariant(field, reason string) error {
    return fmt.Errorf("builder invariant: %s %s", field, reason)
}

The corresponding t.Fatalf shape:

go
if err != nil {
    // Surfaces in CI logs operators read — keep the message clean.
    t.Fatalf("start postgres container: %v", err)
}

No raw error text in a problem detail ​

The detail of an RFC 9457 problem body is a sentence the handler writes. The error behind the refusal goes to the log, under the correlation_id the body carries, so an operator holding an id from a client report still reaches the chain. Interpolating the error instead hands a driver message, a constraint name, or an IdP endpoint to whoever made the request.

The rule is enforced by tests/workspace/no_raw_error_in_problem_details_test.go, which parses every non-test Go file under internal/transport/http/v1/ plus the middleware and recoverer files that emit problem bodies from outside that tree. It reports three things: a file that fails to parse, raw error text in the arguments of an inventoried problem writer, and a file outside the inventory that stamps application/problem+json itself. Run it alone with make check-problem-details; it also rides make lint and make test.

Writing a refusal, then, looks like this:

go
prob.ProblemErr(w, r, http.StatusInternalServerError, codeInternal,
    "Internal Server Error",
    "clouds ListClouds: the Clouds could not be read", err)

The Err-suffixed writers take the cause as their trailing argument and log it. A new problem writer has to be added to the problemEmitterNames inventory in the gate, otherwise its call sites are unchecked.

Worked example ​

The example below exercises one fixture (Postgres), one builder (DomainBuilder), and one matcher (BeReady()) inside a single integration test. It is deliberately small so you can copy-paste it into a new bounded context's test package and wire up the surrounding logic without hunting for boilerplate. The same shape scales up to the full-stack sample at tests/integration/testutil_sample_test.go, which drives StartAll, every builder, and the SSE capture end-to-end.

go
//go:build integration

package mypackage_test

import (
    "context"
    "testing"

    . "github.com/onsi/gomega"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    "github.com/plexsphere/plexsphere/internal/platform/testutil/builders"
    "github.com/plexsphere/plexsphere/internal/platform/testutil/containers"
    "github.com/plexsphere/plexsphere/internal/platform/testutil/matchers"
)

func TestMyPackage_HappyPath(t *testing.T) {
    RegisterTestingT(t)

    // Fixture — a live Postgres container. Cleanup is registered for
    // us; logs dump on failure automatically.
    pg := containers.StartPostgres(t)

    // Builder — a valid Domain aggregate with default Name and Slug.
    domain, err := builders.NewDomainBuilder().
        WithName("plexsphere").
        WithSlug("plexsphere").
        Build()
    if err != nil {
        // The failure string stays free of any traceability
        // identifier; the requirement this assertion guards is
        // recorded in this comment instead (REQ-003, PX-0004).
        t.Fatalf("build domain: %v", err)
    }

    // Exercise — the system-under-test uses pg.DSN and the domain value.
    conds := applyAndObserveConditions(t, context.Background(), pg.DSN, domain)

    // Matcher — assert the aggregate reached Ready=True. The failure
    // message renders every observed condition as a readable diff.
    Expect([]metav1.Condition(conds)).To(matchers.BeReady())
}

Each of the three pieces stands on its own:

  • containers.StartPostgres(t) returns PostgresFixture{Container, DSN}. The matching helpers for SpiceDB, NATS, OpenBao, and SeaweedFS follow the same shape; containers.StartAll(t) returns a Fixtures bundle when you need more than one.
  • builders.NewDomainBuilder() seeds defaults and invariants. The builder exposes fluent WithName, WithSlug, WithDescription, and WithProjects helpers; every Build() error is a plain, readable invariant message with no traceability identifier in the string — record the requirement an assertion guards in the adjacent comment, never in the surfaced text.
  • matchers.BeReady() accepts []metav1.Condition directly and falls back to reflection for any duck-typed slice with Type and Status fields. Pair it with gomega.Eventually — or the shorthand matchers.EventuallyEmit — when the aggregate settles asynchronously.

Cross-references ​