plexsphere

Appearance

Signed SSE Event Bus — partially un-descoped behind an opt-in gate ​

Status: partially un-descoped. A minimal signed slice is live behind the PLEXSPHERE_SSE_NATS_URL opt-in gate; the surface answers a conditional 501 when the bus is not configured.

Decision: partially un-descoped — a minimal slice ships behind an opt-in gate ​

The Signed SSE Event Bus was descoped from the first production release and has since been partially un-descoped: a minimal signed slice is live behind the PLEXSPHERE_SSE_NATS_URL opt-in gate. When the operator sets that env var, the composition root (cmd/plexsphere/sse_factory_prod.go) dials NATS JetStream and threads the full producer → relay → consumer pipeline; the relay filters the outbox onto a closed five-member wire-type set (node_state_updated, policy_updated, bridge_config_updated, action_request, session_setup) and the surface streams signer-signed envelopes. When the env var is unset — the default posture — the factory constructs only the in-memory NonceStore and GET /v1/nodes/{id}/events answers a conditional 501 with code: signed_event_bus_not_provisioned, so the mount stays fail-closed for any deployment that has not opted in.

The slice is an optimisation only. Reconciliation-pull (GET /v1/nodes/{id}/state) stays the correctness baseline for mesh state: node agents learn their desired state by polling, and every dispatch converges through the pull with the bus switched off, so the push stream is never a prerequisite. The slice also runs single-replica only — the handler-side NonceStore is process-local in-memory, so the composition root refuses to boot when PLEXSPHERE_SSE_STREAM_REPLICAS > 1 is set with that store wired.

The remaining workstream stays deferred and is inventoried below:

  • Head-of-line-blocking mitigation — per-subject FilterSubject consumption and the skip-and-advance schema work on no_signing_key rows (see "Production wiring — relay loop and stream provisioning" and "Operational hardening — relay HOL blocking").
  • A distributed nonce store — the prerequisite for relaxing the single-replica boot guard to replicas > 1 (see "NonceStore split / dual-store invariant").
  • Relaying session_revoked and node_secrets_updated over SSE — both fall outside the five-event set, so revocation converges only through the reconciliation-pull sessions block draining.
  • The replay-window 410 behaviour — the Last-Event-ID out-of-window arm stays unreachable until the stream (low, high) bounds are threaded through the EventStream port (see "EventStream Bounds seam").

This document is retained as the un-descope checklist: every item below names the file(s) that hold the placeholder or the wiring, the guarantee it enforces, and the follow-up that must close the remaining gaps.

Outbox accumulation ​

Whether or not the bus is opted in, plexsphere.outbox_events is not an unpublished-event backlog: the authz tuple-sync consumer (internal/authz/sync) reads every outbox row and marks it applied_at, acknowledging even event types it has no mapper for rather than leaving them to accumulate. So a deployment that has not opted into the SSE relay does not grow an ever-growing set of unpublished rows. What remains is ordinary applied-row retention — a general platform concern independent of the SSE surface, not an SSE-specific leak.

Security ports — wired on the opt-in path ​

SignatureVerifier ​

  • Port: internal/transport/http/v1/handlers/events_deps.go, type SignatureVerifier.
  • Backed by: per-Domain key resolver at internal/mesh/sse/signing_key_resolver.go.
  • Status: wired on the opt-in path — buildFullSSEWiring builds a signer-backed *sse.EnvelopeVerifier over the per-Domain resolver and threads it onto handlers.Deps.SignatureVerifier. Nil on the NonceStore-only path (NATS unset), where events_dispatch.go keeps the surface on its 501 + signed_event_bus_not_provisioned stub.

RelationChecker ​

  • Port: internal/transport/http/v1/handlers/events_deps.go, type RelationChecker.
  • Backed by: internal/authz via the mesh factory.
  • Status: wired — mount_v1.go threads the mesh wiring's RelationChecker onto handlers.Deps.RelationChecker (the same node-agent relation gate the reconciliation pull uses). The mount gate requires it non-nil, so the surface stays on its 501 stub for any build that does not wire it.

NodeRepo ​

  • Port: internal/transport/http/v1/handlers/events_deps.go, type NodeRepo.
  • Backed by: tenancy-tx Node aggregate.
  • Status: wired — mount_v1.go shares the reconciliation-pull / heartbeat NodeRepo onto handlers.Deps.NodeRepo. The mount gate requires it non-nil, so the surface stays on its 501 stub for any build that does not wire it.

AuditSink ​

  • Port: internal/transport/http/v1/handlers/events_deps.go, type AuditSink.
  • Optional today: a nil sink suppresses audit emission but the security gates still fire.
  • Wiring task: build an adapter around internal/audit.Sink that translates AuditEntry into the platform's audit.Entry shape; wire it onto handlers.Deps.AuditSink.

Production wiring — relay loop and stream provisioning ​

Relay loop ​

  • Module: internal/mesh/sse/relay_loop.go.
  • Status: wired on the opt-in path. buildFullSSEWiring constructs a Relay over the sqlc-backed RelayStore, calls EnsureNodeEventsStream once at startup, and Run spawns both the RunRelay loop (on the PLEXSPHERE_SSE_RELAY_INTERVAL cadence) and the RunSigningKeyRotationConsumer loop as managed background goroutines. The 0009_sse_relay.sql migration's cursor row in plexsphere.sse_relay_cursor is read and advanced per tick.
  • Deferred: the relay serves the closed five-member wire-type set only; per-subject FilterSubject consumption (below) and the no_signing_key head-of-line-blocking mitigation ("Operational hardening — relay HOL blocking" below) stay open.

sqlc-backed RelayStore ​

  • Module: internal/platform/db/gen/80_sse_relay_cursor.sql.go (generated by sqlc).
  • Status: wired. cmd/plexsphere/sse_relay_store.go wraps *db.Queries over the interactive tier pool and satisfies sse.RelayStore; buildFullSSEWiring passes newSSERelayStore(pool) to the relay so the per-stream cursor row is read and advanced against a real Postgres pool.

Per-subject FilterSubject ​

  • Module: internal/mesh/sse/event_stream.go.
  • Today: the JetStream replay is keyed off streamName only; the stream-level filter plexsphere.node.events.> admits every per-(Domain, Node) subject, so a consumer receives ALL events on the stream regardless of the addressed Node.
  • Wiring task: extend messaging.Client.Replay (or MessageReader.Replay) to accept a FilterSubject argument; bind the JetStream consumer with FilterSubjects.

StartFromTail in production messaging ​

  • Module: internal/platform/messaging/replay.go.
  • Today: rejects startSeq == 0; does not yet support DeliverPolicy=DeliverNew.
  • Wiring task: extend Replay to translate sse.StartFromTail (or a literal "tail" sentinel parameter) into JetStream's DeliverNew policy.

EventStream Bounds seam ​

  • Module: internal/transport/http/v1/handlers/events.go calls sse.ParseLastEventID(header, 0, ^uint64(0)).
  • Today: the OpenAPI 410 path (last_event_id_outside_replay_window) is unreachable because the bounds path is not wired; the 410 description is removed from the spec until then.
  • Wiring task: thread a Bounds(ctx, nodeID) (low, high uint64, err error) method through the EventStream port and pass real (low, high) to ParseLastEventID.

Storage layer — the 0009 migration ​

internal/platform/db/migrations/0009_sse_relay.sql creates the sse_relay_cursor table the relay now reads and advances on the opt-in path. A downgrade that drops it while a cluster has the bus opted in would lose the cursor the relay uses as its at-least-once boundary, so the operator-facing downgrade note lives in the migration file's header comment.

Operational hardening — relay HOL blocking ​

internal/mesh/sse/relay.go returns successes=0 on ErrSigningKeyNotProvisioned, so a Domain whose signing key is never provisioned blocks every subsequent event for that Domain forever. Today the sse_relay_skip_total{reason=no_signing_key} counter and the WarnContext line are the operator signal; the only remediation is to provision the missing key.

A future "skip and advance" fix requires either a sse_relay_skipped table or an outbox_events.status column — both are schema changes that sit outside this PR's scope. Track the schema choice and implementation in a follow-up story under the Signed Event Bus epic.

NonceStore split / dual-store invariant ​

SSEWiring exposes ONE NonceStore (the handler's). On the opt-in path buildFullSSEWiring honours the dual-store invariant: it wires this bundle's NonceStore onto the handler and constructs a SECOND, distinct in-memory NonceStore for the EventStreamAdapter pump, so each layer's symmetric replay gate fires in isolation and a published envelope is never dropped at the handler by a shared store. This holds only single-replica. Relaxing to replicas > 1 stays deferred and requires:

  1. Replace the in-memory NonceStore with a distributed store (e.g. Redis-backed) so inter-replica dedup holds.
  2. Split SSEWiring into AdapterNonceStore and HandlerNonceStore fields (or document that the same distributed store is wired onto both layers).
  3. Relax the PLEXSPHERE_SSE_STREAM_REPLICAS > 1 boot guard only once the distributed store is in use.

Test harness symmetry ​

internal/platform/testutil/sse/capture.go now computes envelope.CanonicalBytes(env_with_sig_cleared) to verify production envelopes; legacy producers that signed raw data: bytes still verify because signedPayload() falls back to the raw payload on parse failure.

Verification on every PR ​

  • The build gate refuses to ship if any of the security ports ends up wired on a production code path without a paired test.
  • go list -m -u all and govulncheck ./... should run on the new modules under internal/mesh, internal/transport/http, and cmd/sse-stub-plexd whenever this roadmap is touched. At time of writing, the locally-installed govulncheck@v1.1.4 and golangci-lint@v2.11.4 binaries are built with a pre-1.26 Go toolchain and refuse to scan a tree that requires Go 1.26 (file requires newer Go version go1.26). CI installs both tools via GOTOOLCHAIN=local go install against a Go 1.26 toolchain (see .github/workflows/ci.yaml); the dependency audit therefore runs in CI rather than at developer-side pre-commit. When the upstream releases catch up to Go 1.26, the audit can move pre-commit.

Producer-side closures already landed ​

Reachability evaluator outbox emitter ​

The per-Domain reachability evaluator's writer-side closure is live: every Healthy → Stale → Unreachable transition writes one tenancy.NodeReachabilityChanged row to plexsphere.outbox_events inside the same transaction as the UpdateReachabilityState SQL write (see internal/identity/tenancy/repo/node_heartbeat_repo.go::TransitionAndEmit and the writer adapter in cmd/plexsphere/evaluator_factory_prod.go). Two cross-cutting invariants are gated by integration tests at tests/integration/reachability_evaluator_emits_outbox_test.go:

  • One outbox row per state transition; idempotent re-runs do NOT duplicate the row (the SQL-side reachability_state <> $2 predicate is the load-bearing guard).
  • The outbox row and the SQL UPDATE share one transaction_id — the same-tx atomicity invariant a crash between the two writes would otherwise violate.

The companion chain-bound AuditSink adapter (*chainedReachabilityAuditSink) wraps the canonical hash-chained audit.Sink so every transition also lands one row on the per-Domain audit chain. The chain row's wire shape is pinned by tests/integration/reachability_evaluator_chained_audit_test.go: Relation = "node_reachability.transition", Object = "domain:<hex16>" (matching the audit factory's defaultDomainResolver), Reason = audit.ReasonGranted, RelationPath = [from_state, to_state, reason]; the per-Domain audit_log_chain_head.next_seq advances by exactly the number of transitions and chain.VerifyChain succeeds end-to-end.

The relay-loop closure that surfaces the outbox row on the wire as a node_state_updated envelope has now landed behind the opt-in gate (see "Production wiring" above). The tests/e2e/mesh/reachability-transitions/ chainsaw fixture stays skip: true until it is wired against the opt-in bus, at which point it asserts the cross-process producer→consumer convergence.

Policy compile-arm wire fan-out (policy_updated) ​

The policy compile arm's producer-side closure is live alongside the peer_* and node_state_updated per-source wire-type expansions. The publisher's wireTypeFor dispatch table at internal/mesh/sse/publisher.go maps the closed two-event policy outbox set (policy_revision_created and policy_deleted) onto the single wire literal policy_updated; the compile service emits one publish per matched Node on revision-created via the policy.WirePublisher port (production binding cmd/plexsphere/policy_wire_publisher_adapter.go) and one per previously compiled Node on delete via lookup-before-cascade through CompiledRulesetRepo.ListNodesByPolicy. Three cross-cutting invariants are gated by integration tests at tests/integration/policy_updated_wire_fanout_test.go:

  • Per-Node arity: one wire publish per (Node, Policy) row on revision-created; one wire publish per previously compiled Node on delete.
  • Payload byte-equality with the persisted rule_payload column on revision-created; canonical empty-rules payload []byte("[]") on delete.
  • JetStream Nats-Msg-Id dedup suppresses a re-driven outbox row as ack.Duplicate=true.

The wire envelope's signature chain is pinned by tests/integration/policy_updated_signature_chain_test.go against the production per-Domain SigningKeyResolver. The relay-loop side (the RelayStore adapter, the EnsureNodeEventsStream call, and the RunRelay goroutine) has now landed behind the opt-in gate (see "Production wiring — relay loop and stream provisioning" above), so the tests/e2e/policy/policy-updated/chainsaw-test.yaml chainsaw fixture's stub-binary producer/consumer pair can extend to the real plexsphere binary without a schema change.

Bridge wire fan-out (bridge_config_updated) ​

The bridge orchestrator's producer-side closure is live alongside the policy_updated and node_state_updated per-source wire-type expansions. The publisher's wireTypeFor dispatch table at internal/mesh/sse/publisher.go collapses the closed seven-member set of bridge.* outbox literals (bridge.RelayConfigured, the user-access / public-ingress / site-to-site *Configured and *Removed literals) onto the single wire literal bridge_config_updated. The four bridge application services each call the bridge.WirePublisher port (internal/bridge/ports.go::PublishBridgeConfigUpdated) after their RunInTx closure commits; the production adapter enumerates the Nodes the changed bridge Resource hosts via the bridge.BridgeNodeLister port, builds the effective config once through the effective.EffectiveConfigBuilder, and emits one signed envelope per hosted Node — the 1-bridge-row → N-Node fan-out. The wire effective_config bytes equal the reconciliation-pull bridge block for the same Node and bridge Resource; the parity is pinned by tests/integration/bridge_config_updated_pull_parity_test.go. A nil WirePublisher keeps a service on its single-arm posture (aggregate write + outbox row + audit row, no wire emission), and a publish error is logged and swallowed because the command has already committed and the pull fallback guarantees eventual convergence.

The consumer-side /v1/nodes/{id}/events HTTP plumbing — the relay-loop side (the RelayStore adapter, the EnsureNodeEventsStream call, and the RunRelay goroutine), the signature-verification, and the ReBAC node-agent gate — has now landed behind the opt-in gate (see "Production wiring — relay loop and stream provisioning" and "Security ports" above); a bridge-mode Node can receive bridge_config_updated over the opt-in bus while still converging through the reconciliation-pull bridge block. The publisher-side dispatch table, the per-Node fan-out algorithm, the BridgeConfigUpdatedPayload schema, and the depguard posture are documented at docs/contexts/bridge/events.md.

Downstream blocked items ​

Reconciliation-pull chainsaw equivalence E2E ​

tests/e2e/mesh/reconciliation-pull/chainsaw-test.yaml carries skip: true. The fixture's terminal convergence gate registers a Domain + N Nodes through the production POST /v1/domains/{id}/nodes surface, captures the resulting SSE peer_registered envelope from the running plexsphere binary, issues GET /v1/nodes/{id}/state against the same binary, and asserts that the (node_id, mesh_ip, public_key) tuple is byte-equal between the SSE projection and the pull peers array. The core relay wiring has now landed behind the opt-in gate (see above), but this equivalence fixture asserts a peer_registered envelope that falls outside the minimal five-event wire-type set the relay currently fans out, so it stays skipped until the peer-envelope taxonomy rides the bus — a chainsaw run must not post a green signal that misrepresents end-to-end coverage of the SSE↔pull equivalence story.

The integration-tier equivalent at tests/integration/state_pull_sse_equivalence_test.go exercises the same convergence story via in-process fakes and is GREEN today; the chainsaw fixture is the cross-process kind-cluster mirror that adds the production-binary regression layer once unblocked. The reconciliation-pull surface itself, its security-gate ordering, the four-block convergence with the SSE event taxonomy, the OpenAPI contract, and the wiring-receipt regression test against BuildProductionStateFactory are documented at docs/contexts/mesh/reconciliation-pull.md.

Cross-references ​

  • internal/mesh/sse/doc.go — package overview and the production wire-format contract.
  • cmd/plexsphere/sse_factory_prod.go — composition-root validation gates and the in-memory NonceStore wiring.
  • internal/transport/http/v1/handlers/events.go and events_dispatch.go — handler body and the fail-closed mount gate.
  • docs/contexts/mesh/reconciliation-pull.md — the pull-channel sibling of the SSE bus; pinned here so a reader chasing the deferred-wiring trail finds the reconciliation-pull surface that shares the signed_event_bus_not_provisioned problem code and unblocks together with the SSE peer endpoint when the deferred wiring above lands.
  • docs/contexts/mesh/peers.md — Key and Peer Manager bounded sub-context reference; pinned here because the three peer event_type strings (peer_registered, peer_psk_assigned, peer_deregistered) ride the SSE bus once the peer-envelope taxonomy is added to the minimal five-event wire-type set.