plexsphere

Appearance

Signing-key rotation workflow ​

This page explains the end-to-end signing-key rotation workflow that ties the Signing Service to the signed event bus. It is the companion to the narrower deployment-scoping and rotation page, which focuses on the lifecycle state machine and the signing_key.valid_until retention semantics; this page covers the control-plane RPCs, the platform-wide fan-out contract, and the operator-facing CLI.

The workflow opens a finite overlap window during which both the retiring and the incoming signing key verify, fans the resulting signing_key_rotated event out to every plexd Node in the scope so their per-Domain public-key cache invalidates, and then closes the window so the old key is retired. The Signing Service is the only process that ever touches Ed25519 private key material; every other participant in the workflow holds public halves only.

Ubiquitous language ​

The terms below extend the Signing Service ubiquitous-language table. Names are preserved verbatim across the Go code, the proto contract, the audit log, and the operator-facing CLI.

TermDefinitionCode anchor
OpenRotationThe control-plane RPC that begins a rotation for a given scope. Allocates the incoming key via the KeyProvider, persists it together with the open transition row in one transaction, publishes one signing_key_rotated event per Node in the scope, and returns the resulting RotationTransition.internal/signing/grpc_server.go, internal/signing/rotation_service.go
CloseRotationThe control-plane RPC that completes an open rotation. Flips the key pair (old → retired, new → active), stamps the transition row closed, and best-effort-retires the old key handle through the provider.internal/signing/grpc_server.go, internal/signing/rotation_service.go
RotationTransitionThe aggregate that binds an outgoing OldKey and incoming NewKey to a Window. Cross-scope transitions (platform → domain, domain A → domain B) are rejected at construction time.internal/signing/rotation/overlap.go
Window (OpenedAt, ClosesAt)The (OpenedAt, ClosesAt) instant pair during which both halves of a rotation are advertised. Strict OpenedAt < ClosesAt invariant; a zero-duration overlap is rejected by NewWindow.internal/signing/rotation/overlap.go
signing_key_rotated eventThe Domain Event the rotation publishes. Carries the (scope, old_key_id, new_key_id, new_public_key, valid_from, opened_at, closes_at) tuple and reaches every plexd Node in the scope via the signed event bus.internal/signing/events/events.go
OutboxEventPublisherThe production EventPublisher implementation in the plexsphere-signer binary. Resolves the fan-out target set (the Nodes in the scope) and writes one outbox row per Node inside a dedicated publisher-owned transaction (NOT the rotation transaction). The repo-side CreateKeyAndOpenRotation commits first; the publisher then opens a second pgx.Tx to fan rows out. The two-phase posture is documented in Fan-out narrative and the per-Node rows are all-or-nothing within the publisher tx (mid-fan-out crash leaves either every row or none).cmd/plexsphere-signer/event_publisher_outbox.go
--overlap-window flagOperator-facing CLI flag on plexsphere-signer (alias env PLEXSPHERE_SIGNER_OVERLAP_WINDOW) that overrides the default rotation overlap duration. Zero or negative values are rejected at startup.cmd/plexsphere-signer/main.go, internal/signing/rotation_service.go
RolloutThe platform-side orchestration wrapper that invokes OpenRotation over gRPC and writes one platform-level audit entry per call. Lives outside the signer so non-signer callers (the plexctl CLI, scheduled jobs, disaster-recovery scripts) reuse the same control-flow seam.internal/signing/orchestration/rollout.go

Lifecycle diagram ​

A rotation drives the SigningKey state machine from active through rotating and back to either active (on the incoming key) or retired (on the outgoing key). The transitions are pure functions in internal/signing/rotation/state.go; no code outside that package may produce a new state from an old one. The Window invariant lives in internal/signing/rotation/overlap.go and is rechecked at every construction site.

mermaid
stateDiagram-v2
    [*] --> active: Activate (zero -> active)
    active --> rotating: OpenRotation\n(pairs with a new key, opens Window)
    rotating --> active: CloseRotation\n(new key promoted at ClosesAt)
    rotating --> retired: CloseRotation\n(old key retired at ClosesAt)
    active --> retired: Retire\n(emergency retirement)
    retired --> [*]
    note right of rotating
      Window invariant:
      OpenedAt < ClosesAt (strict)
      See rotation/overlap.go
    end note

For the deeper state-machine treatment — including the boot-time Resume reconciliation that recovers an overlap window after a signer restart and the field-ownership split between signing_key_transition.(opened_at, closes_at) and signing_key.valid_until — see ./signing/deployment.md.

Fan-out narrative ​

RotationService.Open walks four steps in order. Steps 1 and 2 share the rotation transaction; steps 3 and 4 run AFTER that transaction has already committed, each in its own boundary. The atomicity contract is therefore scoped to the repo write: a crash, partition, or publisher error after step 2 leaves the rotation fully persisted but does NOT roll the repo write back. See the two-phase trade-off for the rationale.

  1. Provider mint. KeyProvider.NewSigningKey is invoked to materialise the incoming key. The provider returns the public half ([32]byte) plus an opaque provider_handle_id; the private bytes never enter the core binary's address space. A compensating provider.Retire is queued for every later failure path through compensateRetire.
  2. Atomic persist. KeyRepo.CreateKeyAndOpenRotation (internal/signing/migrations/repository_pg.go) runs the new-key INSERT, the old-key UPDATE to rotating, and the open transition-row INSERT inside a single pgx.Tx. This collapses the previously two-step NewSigningKey + OpenRotation flow into one transaction — the atomicity property the atomic-persist narrative below pins. The transaction commits before step 3 begins.
  3. Publish. EventPublisher.PublishKeyRotated broadcasts the signing_key_rotated Domain Event. In production the publisher is OutboxEventPublisher (cmd/plexsphere-signer/event_publisher_outbox.go), which resolves the fan-out target set (the Nodes in the scope) and writes one outbox row per Node inside its OWN pgx.Tx (opened via pgxOutboxTxRunner.InTx in cmd/plexsphere-signer/outbox_publisher_wiring.go). That publisher tx is independent of the step-2 rotation tx, which has already committed. An empty target set is a clean no-op; a single row-write failure rolls the publisher tx back (so the fan-out is all-or-nothing within step 3) and surfaces the error to RotationService.Open, which logs at WARN level without unwinding the already-committed rotation state.
  4. Audit. A single audit.Entry is appended with Relation = "rotate-open" by AuditMiddleware (internal/signing/audit_middleware.go), honouring the exactly-one-entry-per-RPC contract that the Sign and PublicKey RPCs already obey.

Two-phase fan-out trade-off ​

A literal single-transaction shape — repo write + outbox fan-out sharing one pgx.Tx — would require the EventPublisher port to take a caller-supplied pgx.Tx, which would (a) leak pgx into the internal/signing package boundary (today the package is pgx-free — the persistence adapter lives behind a port) and (b) couple the RotationService's transactional lifecycle to the publisher's implementation. The current shape keeps the boundary clean at the cost of a narrow crash window between step 2 and step 3 in which the rotation is persisted but no outbox rows have been written.

The crash window is small but not zero. RotationService.Resume at signer restart re-reads the open transition row and auto-closes it when the overlap window has already passed, but it deliberately does NOT republish the signing_key_rotated event — the contract is exactly-one-rotation = exactly-one-fan-out per logical rotation. Operators recovering from a crash in the step-2 / step-3 window therefore observe a persisted rotation with no consumer-side cache invalidation; the per-Node public-key caches drain naturally on the configured TTL, and a fresh OpenRotation on the same scope is refused with ErrRotationInProgress until the operator closes the half-fanned-out rotation. The runbook in ../how-to/platform/rotate-the-signing-key.md documents the recovery procedure.

On the consumer side, every API replica runs a SigningKeyRotationWatcher (internal/mesh/sse/signing_key_rotation_watcher.go) that pages forward through the signing_key_rotated outbox rows on the relay's cadence and calls Reset(DomainID) once per rotated Domain. The reset evicts the per-Domain entry from the public-key cache, so the next PublicKey call re-fetches the live (KeyID, [32]byte) pair through the SigningKeyResolver seam. The watcher runs per replica, not on the leader, because a replica that still trusts a retired key refuses the envelopes a sibling signs with the new one. Its paging position is held in memory and is seeked to the tail of the rotation history when the store is built: a restarted replica watches for the rotations opened from that point, and does not re-walk the rows behind it, which name Domains a process that has just started has cached no key for. The replica holding the relay tick resets inline as it reads the row, so it cannot sign a later row of the same batch with the old key.

The verify path does not wait for a watcher tick at all. EnvelopeVerifier.Verify resolves the key the envelope NAMES in its KeyID field — a field inside the signed pre-image — and refuses only one the signer reports as retired. "Retired" there is the signer's sign-side verdict, not the raw row value: PublicKey reports a rotating key whose overlap window has elapsed as retired, because Sign already refuses to mint under it. The two ends therefore share one rule and one clock — the signer's — so the outgoing half stops being accepted when the overlap closes rather than when somebody eventually calls CloseRotation. During an overlap OpenRotation has moved BOTH halves of the pair to rotating, so the Domain has no active key and each replica signs with whichever half its own cache names; verifying against "the active key" would drop every envelope a replica that had not yet ticked put on the bus. Publisher.Publish resets once when the signer answers ErrKeyRetired and re-signs, on the exact failure a stale cache produces on the publish side.

The signer resolves an empty key id to the scope's active key, or, when the scope has no active key because a rotation overlap is open, to the incoming key that rotation names. OpenRotation moves both halves of a rotation to rotating, so for the whole overlap window no key is active and the incoming key is the one fresh signatures come from. The signer's reconcile probe reads a scope by the same rule: a scope with an open rotation is provisioned, so the probe verifies the incoming key against the provider instead of minting a fresh active key beside the pair.

CloseRotation runs the symmetric tail at end of overlap: it flips the persisted state (old → retired, new → active), stamps the transition row closed, best-effort-retires the old provider handle, and writes the matching rotate-close audit entry. The close path does not republish a signing_key_rotated event — the cache invalidation that landed at OpenRotation time already pinned the new public half on every Node.

OpenRotation / CloseRotation contract ​

The two control-plane RPCs are declared in api/proto/signing/v1/signing.proto and implemented by GRPCServer in internal/signing/grpc_server.go. Both delegate through the SignerPort interface so the production composition root wraps them with AuditMiddleware to emit exactly one audit.Entry per RPC.

OpenRotation ​

OpenRotation(scope, new_key_id) → (old_key_id, new_key_id, opened_at, closes_at)

FieldDirectionSemantics
scoperequestWire form platform or domain:<uuid>; parsed by ParseScope. Malformed input maps to InvalidArgument.
new_key_idrequestThe caller-chosen identifier of the incoming key (URL-safe, ≤ 128 bytes). Validated by NewKeyID; malformed input maps to InvalidArgument.
old_key_idresponseThe retiring key's ID, resolved server-side from the current active key.
opened_at, closes_atresponseThe persisted overlap window. closes_at = opened_at + overlapWindow.

CloseRotation ​

CloseRotation(scope, old_key_id, new_key_id) → ()

FieldDirectionSemantics
scoperequestWire form platform or domain:<uuid>; parsed by ParseScope. Malformed input maps to InvalidArgument.
old_key_id, new_key_idrequestThe matching pair from the open transition row. Validated by NewKeyID; malformed input maps to InvalidArgument.

Stable status-detail strings ​

Every error returned by the two RPCs is mapped to a gRPC status with a byte-for-byte stable detail string by mapSignErrorToStatus. Callers branch on the detail; the prose is the contract. The strings live as package-level constants at the top of internal/signing/grpc_server.go.

Domain sentinelgRPC codeDetail string
ErrRotationInProgressFailedPreconditionsigning: rotation in progress
ErrRotationHandleUnchangedFailedPreconditionsigning: rotation needs a new provider key
ErrKeyRetiredFailedPreconditionsigning: key retired
ErrScopeMismatchNotFoundsigning: scope mismatch
ErrKeyNotFoundNotFoundsigning: key not found
ErrInvariant (and every error wrapping it, including ErrScopeNotPermittedForProfile)InvalidArgumentsigning: invariant violation
ErrProviderUnavailableUnavailablesigning: key provider unavailable
ErrClientIdentityDeniedPermissionDeniedsigning: client identity denied
any other unmapped errorInternalsigning: internal error

Paraphrasing these strings between releases is a documented contract break: the operator runbook and the plexctl signing exit-code catalogue grep for them verbatim.

--overlap-window flag ​

The default rotation overlap duration is 24 hours, pinned by defaultOverlapWindow in internal/signing/rotation_service.go. Operators override it at signer bring-up through a single CLI flag on the plexsphere-signer binary:

shell
plexsphere-signer --overlap-window 12h

or via the environment alias:

shell
PLEXSPHERE_SIGNER_OVERLAP_WINDOW=12h plexsphere-signer

Both surfaces are parsed in cmd/plexsphere-signer/main.go. A zero or negative value is rejected at startup with --overlap-window must be positive; the signer never boots advertising a degenerate overlap. When the flag is omitted the signing.defaultOverlapWindow constant is used directly — the bring-up path threads WithOverlapWindow only when the operator set the flag explicitly so the package-level default remains the single source of truth.

Atomic-persist narrative ​

Earlier iterations of RotationService.Open ran the new-key INSERT and the open transition-row INSERT as two separate repository calls. A crash, network partition, or compensating-retire failure between the two left the database in an inconsistent state: the new signing_key row existed with state active while no transition row pointed at it. Subsequent OpenRotation attempts then collided with the orphan row at the unique-state predicate, masking the partial failure as ErrRotationInProgress and forcing a manual operator clean-up.

KeyRepo.CreateKeyAndOpenRotation (declared in internal/signing/ports.go and implemented in internal/signing/migrations/repository_pg.go) collapses the two INSERTs and the old-key UPDATE into a single pgx.Tx via the existing runInTx helper. The repository classifies errors through the existing per-statement classifier so the caller still sees ErrRotationInProgress on a concurrent open, but the failure now rolls every row back together — there is no observable half-open state.

The compensating-retire flow on the service side was simplified in lockstep: RotationService.Open calls compensateRetire only on the provider-side failures (the NewSigningKey mint and the post-persist provider.Retire path); a CreateKeyAndOpenRotation rollback is self-contained because the transaction either lands every row or none. The end-to-end pin lives in tests/integration/signer_rotation_atomicity_test.go, which exercises the atomic compound against the real testcontainers Postgres + sqlc adapter: a happy-path Open asserts exactly one transition row; a concurrent-Open race triggers ErrRotationInProgress with no orphan rows; and a direct repo.CreateKeyAndOpenRotation call against a pre-seeded colliding key trips the (scope, key_id) unique index and asserts every row rolled back (the colliding row's state, the old key's state, and the absence of any transition row).

See also ​