Provisioning Broker

Authoritative bounded-context reference for the Provisioning Broker (internal/provisioning/broker/). It is the orchestration context that turns a Project operator's declaration — "give me a node on this cloud from this blueprint" — into running substrate that auto-enrols into the mesh.

The broker is the join across the aggregates the earlier provisioning stories built. It resolves a published BlueprintVersion and a Project-assigned Cloud Credential into a Crossplane v2 Composite Resource plus a ProviderConfig, mints and injects a bootstrap token, applies the manifests into the Project's already-reconciled namespace, and runs a pure status state machine that advances a broker-owned ProvisionedResource aggregate to Ready as Crossplane reports the substrate up and the plexd agent completes registration.

The context has no HTTP, OpenAPI, CLI, or dashboard surface: callers reach it through the in-process Service facade, and the reconcile loop drives each ProvisionedResource off live Crossplane status. The closed port set keeps the domain, application, event, and render layers free of pgx and controller-runtime (ports.go). This milestone models the create-to-Ready-or-Failed pipeline; graceful deletion — the teardown machine that tears a ProvisionedResource back down — shipped as a follow-up and has its own reference in deletion.md. The remaining deliberate deferrals — parameter-value persistence and outbox-backed lifecycle events — are catalogued under Milestone limitations.

This reference is a single page: the surface is narrow (one aggregate, one pure state machine, eight create-pipeline ports, eight create-pipeline sentinels) and the pieces travel in lockstep. The Recovery runbook at the foot is the operator-facing companion.

Cross-references

• ../../contributing/layout.md — the bounded-context map row that locates internal/provisioning and its sub-contexts inside the codebase and enumerates the depguard rules that confine pgx to repo/, controller-runtime to reconcile/, the Kubernetes client libraries to reconcile/ and render/, and bar cross-context imports.
• ./management-fleet.md — the sibling sub-context the broker reads through its AssignmentReader port. The Management Fleet owns the durable record of which management cluster owns each Project and reconciles the per-Project namespace; the broker observes only whether that namespace is Ready and refuses to apply substrate until it is.
• ./blueprints.md — the Blueprint Catalog sub-context. The broker's BlueprintResolver port resolves a published BlueprintVersion view by its surrogate id; the view embeds the catalog's immutable ParameterSchema and InjectionStrategy value objects unchanged.
• ./credentials.md and ./credential-pool.md — the Cloud Credentials sub-contexts. The broker's CredentialResolver port resolves a credential view that carries the deterministic OpenBao KV-v2 coordinates the rendered ProviderConfig's Secret reference is derived from; the credential value never enters the broker process.
• ../identity/tenancy.md — the Domain → Project → Resource → Node aggregate model. The tenancy Resource and Project are owned by the identity context; the broker references each only by id across an anti-corruption boundary and never imports the tenancy package.
• ./label-propagation.md — the cloud-tag propagation reference. It owns the depth behind the LabelTagResolver port, the per-provider cloud-tag transformer, and the skip-and-log contract for labels that cannot be mapped onto a provider tag; the render-pipeline summary below is the brief view.
• ../../../internal/provisioning/broker/doc.go — the package-level pin of the ubiquitous language, the closed port set, and the reconcile-model DECISION block (why a boot-probe-plus-ticker closure rather than a watch-driven controller-runtime Manager, and why the context mounts no HTTP route).
• ../../../internal/platform/db/migrations/0026_provisioning_broker.sql — the persistence schema for plexsphere.provisioned_resources and plexsphere.provisioned_resource_outbox_token.
• ../../../cmd/plexsphere/provisioning_broker_factory_prod.go — the production composition root: the env loader, the in-cluster client, the anti-corruption resolver adapters, the bootstrap-token issuer adapter, the reconcile sweep, and the boot-probe / steady-state-ticker pair.
• ../../../tests/e2e/provisioning/provisioning-broker/chainsaw-test.yaml — the Chainsaw e2e suite that stands up a per-Project namespace, runs the reconcile APPLY step, and asserts the rendered Crossplane Composite Resource, the rendered ProviderConfig, and the External-Secrets-Operator-materialised credential Secret the ProviderConfig references by name.

Ubiquitous language

The terms below travel together across the Go code, the SQL migration, the domain-event payloads, the structured-log attributes, and the recovery runbook. Names are preserved verbatim so a reader chasing a string from a log line finds it in the source without translation.

Term	Definition	Code anchor
ProvisionedResource	The aggregate root modelling one declared piece of substrate the broker is driving toward Ready. Carries (id, resourceID, projectID, blueprintVersionID, cloudCredentialID, phase, xrName, bootstrapTokenID, createdAt, updatedAt). The application mints a UUIDv7 id at declaration; the fields are unexported and reached through accessors so the creation invariants hold only through the two constructors.	provisioned_resource.go
ProvisionedResourceID	The UUIDv7 identity of a ProvisionedResource, a named wrapper over uuid.UUID. The String() projection is the canonical hyphenated lowercase form; it is also the seed of the derived XR name. The zero value is rejected by every creation invariant.	ids.go
ProjectID / ResourceID	The UUID identities of the Project the substrate is provisioned for and the tenancy Resource it backs — each a named wrapper, distinct at compile time from the other id types. Both are observed across the anti-corruption boundary: the broker references them only by id and never imports the tenancy package.	ids.go
BlueprintVersionID / CloudCredentialID	The surrogate identities of the published BlueprintVersion and the Cloud Credential a declaration resolved against. The aggregate stores the credential id but not a cloud id — the Cloud is a pure function of the credential row's cloud id, so storing it separately would open a drift surface.	ids.go
BootstrapTokenID	The identity of the bootstrap token the broker minted and injected into a ProvisionedResource's Composite Resource. It is the zero value before issuance — exposed through HasBootstrapToken — and recorded once the reconcile loop has issued a token. The broker stores only the token id, never the plaintext.	ids.go
ProvisionedResourcePhase	The closed-set lifecycle of a ProvisionedResource: Pending, Provisioning, Enrolling, Ready, Failed. The phase is advanced exclusively by the pure transition function over observed Crossplane facts, never from process memory.	phase.go
ObservedXR	The snapshot of live facts the reconcile loop reads from a Crossplane Composite Resource on a single tick: whether the XR exists, whether Crossplane reports the substrate Ready, whether Crossplane reports a terminal failure, and whether the expected plexd Node has registered. Every field is a fact the cluster reports, not a desired state.	phase.go
Action	The reconcile instruction the transition function emits for one tick: Noop, Apply, DeregisterNode, or DeleteSubstrate. Apply drives the create pipeline; DeregisterNode and DeleteSubstrate drive the ordered teardown machine documented in deletion.md.	phase.go
InjectionStrategy	The closed enumeration the Blueprint Catalog owns — cloud-init-user-data, helm-values, provider-secret — naming how the broker threads a minted bootstrap-token plaintext into the rendered Composite Resource. The broker consumes the strategy; it does not define it.	render/injection.go
Service	The in-process application-service facade. Its single method Provision turns a declaration into a persisted, Pending ProvisionedResource. It resolves the declaration against the ports and owns the admission gate; the aggregate enforces its own creation invariants.	service.go
Reconciler	The controller-runtime adapter that reconciles exactly one ProvisionedResource per call: observe the XR, decide via the pure transition, apply the action, then emit the lifecycle event and persist the phase.	reconcile/reconcile.go
per-Project namespace	The Kubernetes Namespace the Management Fleet carves out for one Project on its management cluster, named plexsphere-project-<project-id>. The broker renders its Composite Resource and ProviderConfig into this namespace and refuses to apply until the Management Fleet has driven it Ready.	render/render.go

The ProvisionedResource aggregate

The context owns a single aggregate root, ProvisionedResource. It is modified one per transaction; the application service's Provision resolves every collaborator before the aggregate is built, and the reconcile loop advances its phase one tick at a time.

Constructors and invariants

NewProvisionedResource and HydrateProvisionedResource are the only paths that produce a valid aggregate. Both run the single private buildProvisionedResource, so a fresh aggregate and a row hydrated from storage are validated identically — a row that drifts from the invariants in storage is rejected at the hydration boundary rather than surfacing later in the reconcile loop.

Invariant	Layer	Failure mode
id, resourceID, projectID, blueprintVersionID, and cloudCredentialID are all non-zero.	Aggregate constructor.	ErrInvalidInput — a zero id has no semantic anchor and would collide with the "not assigned" sentinel.
phase is one of the five closed-set values.	Aggregate constructor; SQL CHECK constraint provisioned_resources_status_check.	ErrInvalidInput on construction; the CHECK is defence-in-depth against an out-of-set string reaching a persisted row.
xrName is non-empty after trimming.	Aggregate constructor.	ErrInvalidInput — the reconcile loop addresses the live Composite Resource by this name; an empty name would make the loop unable to observe its own applied object.
bootstrapTokenID is optional — a freshly declared aggregate has none.	Aggregate constructor accepts the zero value.	None — a zero BootstrapTokenID is the honest pre-issuance state, exposed through HasBootstrapToken.
createdAt and updatedAt are non-zero, with createdAt ≤ updatedAt.	Aggregate constructor.	ErrInvalidInput — an unordered or zero timestamp pair is a corrupt row.
id is the PRIMARY KEY; resource_id, project_id, blueprint_version_id, and cloud_credential_id each FOREIGN KEY … ON DELETE RESTRICT.	SQL.	A duplicate id, or a delete of a referenced aggregate while a ProvisionedResource still tracks running substrate against it, fails closed at the database.

The derived XR name

The Composite Resource's metadata.name is derived, never operator-supplied: the application service stamps pr- onto the canonical UUID text of the ProvisionedResourceID (deriveXRName). The id is the one value that is unique per aggregate, immutable for its lifetime, and already a DNS-1123-safe lowercase hyphenated string, so the rendered object, the observe step, and every later apply tick all agree on one name. An operator-supplied name would add a uniqueness-collision surface the broker would have to police; deriving it removes that class of failure.

Status state machine

Each ProvisionedResource walks an eight-phase lifecycle. The phase is advanced only by the pure, total transition function Next (phase.go), which reads no clock, no process memory, and no I/O: it derives the Action and the next ProvisionedResourcePhase solely from the live ObservedXR snapshot and the aggregate's current phase. The same inputs always yield the same output, so the machine is fully unit-testable without a cluster.

The eight phases split into two groups: the five-member create → Ready → Failed pipeline (Pending, Provisioning, Enrolling, Ready, Failed) and the three-member ordered teardown machine (Deregistering, Deprovisioning, Deleted). Next has one converge arm, one teardown arm, and two terminal sinks (Failed and Deleted, both sticky):

  Converge arm — current phase ∈ {Pending, Provisioning, Enrolling,
  Ready}, or an unrecognised phase string (treated as Pending)

    ┌──────────┐  XR does not exist yet
    │ Pending  │ ─────────────────────────┐  (Apply, Pending)
    └────┬─────┘ ◀────────────────────────┘
         │
         │  XR exists, Crossplane not yet Ready
         ▼
  ┌────────────────┐
  │ Provisioning   │ ─┐  (Apply, Provisioning) — keeps the
  │   (Apply)      │ ◀┘  manifests applied while substrate converges
  └───────┬────────┘
          │  Crossplane reports Ready, plexd Node not yet registered
          ▼
  ┌────────────────┐
  │  Enrolling     │ ─┐  (Apply, Enrolling) — waits for the
  │   (Apply)      │ ◀┘  bootstrap token to be consumed
  └───────┬────────┘
          │  Crossplane Ready AND plexd Node registered
          ▼
  ┌────────────────┐
  │    Ready       │  terminal-success — (Noop, Ready)
  └────────────────┘

  Terminal sink — current phase is Failed; sticky

  ┌────────────────┐
  │    Failed      │  always → (Noop, Failed)
  └────────────────┘

  A terminal Crossplane failure condition observed in ANY converge-arm
  phase routes straight to (Noop, Failed).

  Teardown arm — current phase ∈ {Deregistering, Deprovisioning,
  Deleted}; selected purely by the current phase and sticky, so a
  resource in a deletion phase never routes back into the converge arm

  ┌────────────────┐  plexd Node still observed registered
  │ Deregistering  │ ─┐  (DeregisterNode, Deregistering) — drain the
  │                │ ◀┘  Node out of the mesh first
  └───────┬────────┘
          │  Node deregistered, Crossplane XR still exists
          ▼
  ┌────────────────┐
  │ Deprovisioning │ ─┐  (DeleteSubstrate, Deprovisioning) — delete the
  │                │ ◀┘  Composite Resource only after the Node is gone
  └───────┬────────┘
          │  Node deregistered AND XR gone
          ▼
  ┌────────────────┐
  │    Deleted     │  terminal teardown sink — (Noop, Deleted); sticky
  └────────────────┘

  The teardown arm enforces the node-before-substrate ordering: the
  plexd Node MUST be deregistered out of the mesh strictly before the
  Crossplane Composite Resource is deleted, because deleting the
  substrate first would orphan a live mesh Peer that can no longer be
  drained cleanly.

Key properties the transition guarantees:

• The function is total. Every (ObservedXR, ProvisionedResourcePhase) pair maps to a defined result; an unrecognised phase string is treated as Pending, the safe converge-from-scratch entry point. The phase strings read for control flow are Failed and the three deletion phases (Deregistering, Deprovisioning, Deleted), because their stickiness cannot be re-derived from ObservedXR alone.
• Failed is a sticky sink. Once Crossplane reported a terminal failure and the aggregate was persisted Failed, the transition never thrashes it back toward Provisioning even if a later ObservedXR snapshot looks healthy. This is the anti-thrash guarantee.
• Terminal failure is checked first. The terminal-failure arm runs before the existence and readiness arms, so a terminally-failed XR that still reports XRExists or CrossplaneReady cannot be mistaken for a converging or Ready one.
• Enrolling is distinct from Ready. Crossplane reporting the substrate up is not the end of the pipeline: the broker holds the resource at Enrolling until the plexd agent consumes the injected bootstrap token and the expected Node registers.

Terminal failure is keyed off a deliberate marker

CrossplaneTerminalFailure is not Crossplane's built-in Synced condition being False. Synced=False is routinely transient — a Composition briefly not yet installed, a provider pod restart, RBAC propagation lag on a fresh namespace — and keying a sticky terminal sink off a transient signal would permanently brick a healthy resource on a momentary hiccup. The reconcile loop instead reads a dedicated, Composition-surfaced ProvisioningFailed status condition: only a deliberate, non-recoverable marker the Composition author sets — an invalid blueprint parameter the provider rejects, a quota the cloud will never grant — declares terminal failure. Until a catalog Composition surfaces that condition, a ProvisionedResource never reaches Failed and simply keeps converging, which is the safe default (reconcile/observe.go).

What the reconcile loop does with the result

Reconciler.Reconcile (reconcile/reconcile.go) runs one ProvisionedResource per call in four steps:

1. Observe — resolve the XR's GroupVersionKind from the blueprint XRD and its namespace from the Management Fleet assignment, Get the Composite Resource, and fold its Crossplane status into an ObservedXR. A NotFound marks the XR absent; any other API error is an infrastructure failure that propagates.
2. Decide — call the pure Next transition.
3. Apply — Apply runs the namespace gate, mints the bootstrap token exactly once, renders the Composite Resource and ProviderConfig, and Server-Side-Applies both; Noop does nothing. Every step is idempotent.
4. Emit + persist — when the phase changed, emit first then write the new phase through the Repository. A crossing into Ready or Failed publishes the matching lifecycle event; any other phase change carries no event. A transition that lands on the resource's current phase performs no write and emits no event.

Re-running Reconcile against an already-converged ProvisionedResource is success.

Why emit runs before persist

This is the one deliberate divergence from the Management Fleet reconcile's call order, which persists then emits. The no-write-when-unchanged guard returns early when the next phase equals the current phase, so a crash after a terminal phase write but before its emit would, on the next tick, observe the resource already terminal, let Next return (Noop, <same phase>), and hit the early return without ever emitting — a silently dropped terminal event. Emitting first closes that gap: a crash after the emit but before the persist leaves the phase non-terminal, so the next tick re-derives the terminal transition and re-emits. The contract emit-before-persist guarantees is therefore "a terminal event is never silently dropped", with at-least-once emission as the accepted failure mode. At this milestone that at-least-once emission is not yet collapsed to exactly-once: the production EventSink is a structured-slog sink, not an outbox-backed one (see Lifecycle events and Milestone limitations), so a re-emit after a crash-and-retry produces a duplicate slog breadcrumb line — harmless for an operator log line. The at-most-once provisioned_resource_outbox_token table is already in place for a future outbox-backed EventSink to key the exactly-once collapse off (reconcile/reconcile.go).

The render and apply pipeline

When the transition emits Apply, the reconcile loop's applyComposite step runs four sub-steps (reconcile/apply.go):

1. Gate — read the Management Fleet namespace observation through the AssignmentReader port. A not-yet-Ready namespace is a skip, not an error: the Management Fleet reconcile converges that namespace on its own ticker, and the broker's namespace is simply not ready yet — the expected steady state for a freshly declared ProvisionedResource. Returning an error would noise the caller's retry metrics for a non-fault. ErrNamespaceNotReady remains the sentinel for a caller that wants to assert the precondition explicitly; the steady-state loop does not.
2. Token — mint a bootstrap token through the TokenIssuer port exactly once per ProvisionedResource, gated by the aggregate's HasBootstrapToken. The token id is persisted before the apply through Repository.UpdatePhase carrying the resource's unchanged current phase, so a crash between the issue and the apply still leaves HasBootstrapToken true on the next tick — the issuer is never called twice for one resource.
3. Render — resolve the blueprint, credential, and cloud views and build the Composite Resource and ProviderConfig objects.
4. Apply — Server-Side-Apply both objects into the per-Project namespace under the stable field manager plexsphere-provisioning-broker. A Server-Side Apply against an already-present object is an idempotent no-op write, so re-applying every tick is the normal steady state; a field-ownership conflict is resolved in the broker's favour because the broker reconcile is the authoritative writer of the fields it renders.

What render/ produces

The render/ subpackage is a pure function — no cluster client — that builds two unstructured objects (render/):

Object	What render/ builds	Notes
Composite Resource (XR)	render.XR validates the operator parameter values against the blueprint's ParameterSchema first and fails closed on ErrParameterValuesInvalid before any object is constructed; it then derives the XR's apiVersion (from the XRD's spec.group plus the first served version) and kind (from spec.names.kind), namespaces the object into plexsphere-project-<project-id>, assigns the caller-supplied metadata.name, and threads each supplied parameter value under spec.parameters. The render.WithCloudTags option additionally stamps the resource's propagating labels onto spec.cloudTags as a per-provider cloud-tag map.	The minted bootstrap-token plaintext is threaded into the Composite Resource by render.InjectBootstrap at the site the blueprint's InjectionStrategy names (see below); the plaintext never reaches a log line, an error string, or an event payload.
ProviderConfig	render.ProviderConfig keys the object's apiVersion group off the cloud provider discriminator (<provider>.crossplane.io), inlines the non-secret endpoint JSONB blob under spec.endpoint, and points spec.credentials.secretRef.name at the External-Secrets-Operator-materialised Secret.	The Secret name is cloud-credentials-<16-hex-char-hash>, a stable hash of the credential's OpenBao KV mount and path — never of the credential value, which the broker process never reads.

render.InjectBootstrap threads the token through one of three sites, selected by the blueprint's InjectionStrategy:

• cloud-init-user-data renders a full #cloud-config enrolment document and writes it to spec.userData — the single field the Cloud-Init VM XRD exposes for the Composition to patch onto the provider's user-data input;
• helm-values threads the spec.parameters.helmValues.{bootstrapToken, apiUrl, plexdImage} triple a plexd Helm release consumes;
• provider-secret writes the bare token to spec.parameters.providerSecret.bootstrapToken, unchanged from the earlier broker milestone.

The two spec.parameters sites share the namespace the Composition patches operator parameters from and cannot collide with a Crossplane v2 reserved spec key; the cloud-init document instead occupies the dedicated spec.userData field. The full user-data document contract, the DaemonSet bundle and its two token-delivery modes, the per-strategy field map, and the force-apply preservation rule that keeps these fields alive across re-apply ticks are documented in cloud-init-daemonset.md.

The render step also stamps the resource's propagating labels onto the XR's spec.cloudTags field. The reconcile loop resolves the Resource's propagation-filtered effective label set through the LabelTagResolver port, a per-provider transformer turns it into a cloud-tag map, and render.WithCloudTags writes that map onto spec.cloudTags; an empty map leaves the field absent. A blueprint Composition copies spec.cloudTags onto the provider's own tag field. The transformer's per-provider constraints, the skip-and-log contract for labels that cannot be mapped, and the propagation gating live in ./label-propagation.md — broker.md is the create-pipeline reference and keeps only this summary.

Ports

The context reaches every collaborator through one of its ports declared in framework-free terms — context, time, the broker's own value types, and the immutable value objects of the sibling Blueprint Catalog sub-context — so the domain layer stays free of pgx, controller-runtime, and k8s.io (ports.go). Every port is consumer-defined: declared by the broker against the broker's own vocabulary, not re-exported from the adapter packages that satisfy them. The composition root wires concrete adapters; tests inject in-memory fakes. The eight create-pipeline ports are tabled below; the ordered teardown machine adds NodeDeregistrar and label propagation adds LabelTagResolver — documented in deletion.md and label-propagation.md — so ports.go declares ten in total.

Port	Methods	Adapter	Test seam
Repository	Create, GetByID, List, UpdatePhase, RunInTx	Postgres in repo/provisioned_resource_pg.go, a thin wrapper over the sqlc-generated queries. Create is the atomic persist-plus-emit boundary; UpdatePhase is phase-write-only. Constraint-name dispatch maps SQLSTATE collisions and FK violations onto the sentinels. depguard confines pgx to this subpackage.	In-memory fakes in the unit tests; the repository adapter has its own classification tests, and the integration suite drives the real adapter against a Postgres instance.
TokenIssuer	Issue	The seam onto the identity context's bootstrap-token issuer. The composition-root adapter supplies the Node kind, the live environment prefix, the operator-tuned TTL, and a fixed reserved actor id, translating the identity issuer's result into a broker-local IssuedToken.	A fake returns a stub token id and plaintext.
BlueprintResolver	Resolve	Resolves a broker-local BlueprintVersionView (XRD blob, Composition blob, ParameterSchema, InjectionStrategy) by surrogate blueprint version id. The production adapter reads the four columns directly and re-parses the schema and strategy through the Blueprint Catalog's own parsers. Returns ErrBlueprintNotFound.	A fake returns a fixed view.
CloudResolver	Resolve	Resolves a broker-local CloudView — the provider discriminator and the non-secret endpoint blob — by the Cloud id a credential view carries. Returns ErrCloudNotFound.	A fake returns a fixed view.
CredentialResolver	Resolve	Resolves a broker-local CredentialView — the residency Cloud id and the deterministic OpenBao KV-v2 coordinates, never the credential value — by cloud credential id. Returns ErrCredentialNotFound.	A fake returns a fixed view.
AssignmentReader	Read	The anti-corruption read onto the Management Fleet context. Returns the per-Project NamespaceObservation (the namespace name and a Ready flag); ErrAssignmentNotFound when the Project has not been placed. The broker does not re-implement namespace, RBAC, or quota convergence.	A fake returns a fixed observation.
AuditSink	Record	A composition-root shim translating the local AuditEntry value object onto internal/audit.Entry — keeping the module free of an internal/audit import the cross-context rule denies.	An in-memory recording sink.
Clock	Now	A wall-clock implementation at the composition root.	A frozen clock pins a deterministic now in unit tests.

The reconcile loop additionally declares a narrow EventSink port local to the reconcile/ package (reconcile/config.go): the lifecycle events it emits are an integration concern, not an aggregate write, so per the interface-segregation principle the adapter declares the smallest dependency it needs — a single Publish(eventType, payload) — rather than widening Repository with an outbox surface.

Lifecycle events

The create pipeline defines three typed domain events (events/events.go); the ordered teardown machine adds two more — broker.ProvisionedResourceDeleting and broker.ProvisionedResourceDeleted, documented in deletion.md. The EventType discriminator string is stable and becomes part of the wire contract once an event is emitted; the set is closed and the five discriminator strings match the event_type CHECK constraint in the migration character-for-character so the storage CHECK and the application discriminator cannot drift.

Event type (discriminator)	Trigger	Emitter
broker.ProvisionedResourceRequested	A Project operator's declaration is accepted and a Pending ProvisionedResource is minted.	The Repository.Create adapter, transactionally with the aggregate row.
broker.ProvisionedResourceReady	The ProvisionedResource crosses into the Ready phase — Crossplane reports the substrate up and the plexd agent has registered.	The reconcile loop, through its EventSink port.
broker.ProvisionedResourceFailed	The ProvisionedResource crosses into the terminal Failed phase.	The reconcile loop, through its EventSink port.

The reconcile loop is the sole emitter of the terminal Ready and Failed events. The ProvisionedResourceFailed payload carries a human-readable Reason derived from the live Crossplane terminal condition message — only the reconcile loop observes that condition, so the repository cannot stamp a meaningful Reason and UpdatePhase is deliberately phase-write-only. The Reason is a free-form summary, never a structured error code.

At this milestone the production composition root emits the reconcile events through a structured-slog sink — the broker Repository port is aggregate-shaped persistence with no free-standing outbox-append surface, so there is no public seam the EventSink could route the Ready / Failed events through without widening the port. The slog line is the operator-facing breadcrumb until a dedicated lifecycle-event relay lands; the rationale lives in the DECISION block on provisioningBrokerEventSink in provisioning_broker_factory_prod.go.

Error sentinels

Every create-pipeline operation funnels through one of eight package-local sentinels; the ordered teardown machine adds a ninth, ErrNodeDeregistrationFailed, documented in deletion.md. Callers branch on these via errors.Is — wrapping with fmt.Errorf("%w", …) is fine, identity must remain intact. The set is closed: adding another without updating the closed-set drift gate trips the build (errors.go).

Sentinel	Layer	Trigger	Remediation
ErrInvalidInput	Aggregate constructors / port boundaries	A zero id, a zero or unordered timestamp, an out-of-range phase string, an empty XR name, or a malformed XRD / manifest blob observed before any persistence call.	Programmer error at a boundary; surfaces in tests, not in steady-state production.
ErrBlueprintNotFound	Service / Repository	The BlueprintResolver cannot resolve a declaration's blueprint version id, or the blueprint_version_id FK has no parent row.	Re-check the published BlueprintVersion id; the catalog may not have published that version.
ErrCloudNotFound	Service	The Cloud resolved transitively from the credential's cloud id has no CloudResolver match.	Re-check the Cloud Inventory; the credential's residency Cloud may have been removed.
ErrCredentialNotFound	Service / Repository	The CredentialResolver cannot resolve a declaration's cloud credential id, or the cloud_credential_id FK has no parent row.	Re-check the Cloud Credential id; the credential may not exist or may have been deleted.
ErrAssignmentNotFound	Service / Reconcile (AssignmentReader)	The Project has not been placed onto a management cluster, so it has no Management Fleet namespace assignment.	Place the Project with the Management Fleet's AssignProjectToCluster before declaring substrate for it.
ErrParameterValuesInvalid	Service / render.XR	The declaration's parameter-value map fails the resolved BlueprintVersion's ParameterSchema validation.	Correct the parameter values against the blueprint's published schema. The offending-parameter detail is preserved in the wrapped message.
ErrProvisionedResourceNotFound	Repository	GetByID or UpdatePhase addresses a ProvisionedResourceID with no row.	Re-check the id; the ProvisionedResource may never have been declared.
ErrNamespaceNotReady	Reconcile precondition gate	A ProvisionedResource is observed but the owning Project's Management Fleet namespace has not reached Ready.	None for the steady-state loop — the apply skips this tick and retries; the namespace converges on the Management Fleet's own ticker. The sentinel exists for a caller that asserts the precondition explicitly.

Several further sentinels exist outside the closed domain set because they name wiring bugs, not observable domain failures, and a misconfigured composition root must fail fast at boot rather than on the first operation:

• ErrServiceRepositoryRequired / ErrServiceCollaboratorRequired — NewService was handed a nil Repository or a nil other collaborator (service.go).
• The per-collaborator ErrReconciler…Required sentinels — NewReconciler was handed a nil Client, Repository, Events, TokenIssuer, BlueprintResolver, CloudResolver, CredentialResolver, or AssignmentReader (reconcile/config.go).

Persistence

Migration 0026_provisioning_broker.sql introduces two tables in the plexsphere schema:

• plexsphere.provisioned_resources — one row per broker-owned ProvisionedResource. id is the application-minted UUIDv7 PRIMARY KEY; resource_id, project_id, blueprint_version_id, and cloud_credential_id each FOREIGN KEY … ON DELETE RESTRICT so none of the referenced aggregates can be deleted while a ProvisionedResource still tracks running substrate against them. status is held to the five-value closed set by provisioned_resources_status_check. xr_name records the rendered Composite Resource's metadata.name; bootstrap_token_id is nullable — a token is minted only when the resource reaches the enrolment stage, so a Pending / Provisioning row carries NULL.
• plexsphere.provisioned_resource_outbox_token — one row per (provisioned_resource_id, event_type) pair under a composite PRIMARY KEY. It is the structural guarantee that each broker lifecycle event reaches plexsphere.outbox_events at-most-once: a retried Provision call or a retried reconcile transition finds an existing token row and skips the outbox append. event_type is held to the closed set of five broker discriminators by a CHECK constraint.

Because neither table holds secret bytes or hash-chained forensic rows — the credential material lives in OpenBao KV-v2 referenced only by id, the bootstrap-token plaintext never lands in a broker column, and plexsphere.outbox_events carries the event history — the migration's Down block performs a real DROP in reverse-FK order.

Operational model

The provisioning broker is opt-in at the composition root. The single load-bearing knob is PLEXSPHERE_DSN: when it is empty the binary boots without a provisioning-broker reconcile probe (the early-boot posture for deployments that have not yet plumbed Postgres). The broker has no secret-material sink of its own — it resolves namespaces, blueprints, and non-secret cloud metadata — so the DSN is the only switch. The in-cluster Kubernetes access is ambient — the ServiceAccount the Pod runs under — and needs no env var.

Env var	Effect	Default
PLEXSPHERE_DSN	The Postgres connection string. Empty disables provisioning-broker wiring entirely.	"" (inert).
PLEXSPHERE_PROVISIONING_BROKER_RECONCILE_INTERVAL	The steady-state period between broker reconcile sweeps. Parsed with time.ParseDuration; a non-positive value is rejected at boot.	30s.
PLEXSPHERE_PROVISIONING_BROKER_TOKEN_TTL	The lifetime stamped onto every bootstrap token the broker mints. Validated at boot against the bootstrap-token issuer's [MinTTL, MaxTTL] window.	1h.

The reconcile runs on two cadences, driven by provisioning_broker_factory_prod.go and registered through internal/platform/bootstrap/provisioning_broker_reconcile.go:

1. Boot sweep — synchronous, before the listener binds. The reconcile sweep runs once; a failure here refuses startup, because a broker inventory that cannot be reconciled into its expected shape must not serve traffic.
2. /readyz probe + steady-state ticker — after the boot sweep the same closure is registered as a /readyz probe under the stable name provisioning-broker-reconcile, and a goroutine re-runs the sweep every reconcile interval. A failure on a later probe tick flips /readyz to HTTP 503 so Kubernetes (and operators) catch drift after the binary has already come up. The ticker logs but does not abort on a per-tick error, and exits cleanly on context cancellation.

The sweep lists every declared ProvisionedResource and reconciles each one, returning the first error it hits. It is idempotent because Reconcile is total and idempotent.

Milestone limitations

This milestone ships the create-to-Ready-or-Failed pipeline. Two capabilities are deliberately deferred to follow-up stories. Each is recorded here so an operator reading this reference — and a contributor extending the context — sees the boundary explicitly rather than discovering it from a stuck ProvisionedResource. A third capability, graceful deletion, was deferred when this reference was first written and has since shipped — see Graceful deletion below.

Operator-supplied parameter values are not persisted

Service.Provision accepts a ProvisionDeclaration.ParameterValues map and validates it against the resolved blueprint's ParameterSchema before it persists the aggregate — an invalid map fails closed with ErrParameterValuesInvalid and no row is written. The validated values are not, however, carried onto the ProvisionedResource aggregate or the provisioned_resources table: neither the aggregate nor migration 0026 has a parameters field.

The consequence is a limitation an operator must know:

• A blueprint whose ParameterSchema declares no required parameters provisions end-to-end normally.
• A blueprint whose ParameterSchema declares any required parameter cannot currently be provisioned end-to-end. The first reconcile apply has no live Composite Resource to read prior values back from, so it renders the XR with an empty parameter map; render.XR's ValidateValues then fails closed every tick and the ProvisionedResource stays at Pending indefinitely. The broker fails closed — it never provisions malformed substrate — but it also never advances.

Persisting ParameterValues on the aggregate (an aggregate field plus a provisioned_resources column) is a follow-up story. The within-context reconcile behaviour is recorded in the DECISION block on parameterValues in reconcile/apply.go.

Lifecycle events reach a slog sink, not the transactional outbox

The terminal broker.ProvisionedResourceReady and broker.ProvisionedResourceFailed events the reconcile loop emits go through a structured-slog EventSink at the production composition root — not through plexsphere.outbox_events. The broker.ProvisionedResourceRequested event is written transactionally, by Repository.Create; the two reconcile-emitted terminal events are operator-facing slog breadcrumbs at this milestone.

The provisioned_resource_outbox_token table is already in place (migration 0026, with the Ready and Failed discriminators in its event_type CHECK constraint) so a future outbox-backed EventSink can append the terminal events transactionally and key the at-most-once idempotency off the token — collapsing the reconcile loop's at-least-once emission (see Why emit runs before persist) to exactly-once. Wiring that sink is a follow-up story; the rationale lives in the DECISION block on provisioningBrokerEventSink in provisioning_broker_factory_prod.go.

Graceful deletion

Graceful deletion — tearing a ProvisionedResource back down — was out of scope when this reference was first written and has since shipped as a follow-up story. The status state machine now carries a second, sticky teardown arm alongside the converge arm documented above, and the Action set is the four-valued {Noop, Apply, DeregisterNode, DeleteSubstrate}. A deletion request drains the plexd Node out of the mesh, deletes the Crossplane Composite Resource and ProviderConfig in that order, and crosses the aggregate through Deregistering → Deprovisioning → Deleted.

The teardown machine has its own reference page: deletion.md documents the teardown state machine and its node-before-substrate ordering invariant, the NodeDeregistrar port, the broker.ProvisionedResourceDeleting and broker.ProvisionedResourceDeleted lifecycle events, the ErrNodeDeregistrationFailed sentinel, the deletion schema evolution in migration 0027_provisioning_broker_deletion.sql, and the teardown recovery runbook.

Recovery runbook

This section is the operator-facing companion to the reference above. Each entry follows the same shape — Symptom, Diagnostic, Remediation — and is scoped to a single failure mode. The reconcile loop is idempotent, so unless an entry says otherwise the safe baseline action is to let the next sweep retry and watch /readyz and the provisioning-broker-reconcile probe recover.

1. ProvisionedResource stuck in Pending

Symptom. A ProvisionedResource never leaves Pending across many sweeps. The structured log shows the broker reconcile tick succeeding, not failing — the resource simply does not advance.

Diagnostic.

• The transition holds a resource at Pending while ObservedXR.XRExists is false — the Composite Resource has not been applied. The most common cause is the namespace gate: the apply step skips (no error) while the owning Project's Management Fleet namespace is not yet Ready.
• Read the Management Fleet assignment for the Project: confirm the per-Project namespace plexsphere-project-<project-id> exists and the assignment's namespace_phase is Ready. A namespace stuck in Provisioning or Degraded keeps the broker apply skipping.
• Confirm the blueprint XRD resolves: a malformed XRD blob surfaces as an ErrInvalidInput-wrapped failure in the observe step, which does fail the tick — so a succeeding tick rules the XRD out.

Remediation. Drive the per-Project namespace to Ready — see the Management Fleet recovery runbook. Once it is Ready the next broker sweep applies the Composite Resource and the resource advances to Provisioning on its own. No ProvisionedResource row needs re-creating.

2. ProvisionedResource stuck in Provisioning

Symptom. The Composite Resource exists on the cluster but the ProvisionedResource never crosses into Enrolling. The broker reconcile tick succeeds.

Diagnostic.

• The transition holds Provisioning while Crossplane has not reported the substrate Ready and has reported no terminal failure. kubectl get <xr-kind> pr-<uuid> -n plexsphere-project-<project-id> and inspect .status.conditions — the Ready condition is not yet True.
• Inspect the Crossplane Composition and the provider: a provider pod crash-looping, a missing ProviderConfig, or a cloud-side back-pressure all hold Ready False without being terminal.
• Distinguish this from entry 3: a transient Synced=False or Ready=False is not terminal — the broker deliberately keeps converging. Only a ProvisioningFailed condition set True is terminal.

Remediation. Repair the Crossplane substrate (the provider install, the Composition, the cloud-side cause). The broker reconcile keeps the manifests applied every tick; once Crossplane reports Ready the resource advances to Enrolling automatically. No broker action is needed.

3. ProvisionedResource crossed into Failed

Symptom. A ProvisionedResource is in the terminal Failed phase. A broker.ProvisionedResourceFailed lifecycle event was emitted, its Reason field carrying the Crossplane terminal-condition message.

Diagnostic.

• Failed is reached only when a Composition surfaces a ProvisioningFailed status condition set True — a deliberate, non-recoverable marker (an invalid blueprint parameter the provider rejects, a quota the cloud will never grant). Read the event's Reason or the live XR's ProvisioningFailed condition message for the specific cause.
• Failed is sticky: the pure transition never moves a Failed resource back toward Provisioning, even if the underlying fault is later fixed. The broker will not self-heal a Failed resource.

Remediation. The Failed phase is the honest record of a non-recoverable fault — not a bug. Fix the root cause (correct the blueprint parameters, raise the cloud quota), then declare a new ProvisionedResource through Service.Provision. The old Failed row is retained as the audit record of the failed attempt; do not attempt to revive it.

4. Reconcile probe failing — broker cannot reach Postgres or the cluster

Symptom. /readyz reports the provisioning-broker-reconcile probe failing (HTTP 503), or the boot sweep refused startup. The structured log carries provisioning broker reconcile tick failed.

Diagnostic.

• A List failure points at Postgres: confirm the PLEXSPHERE_DSN target is reachable and the broker tables exist (migration 0026 applied).
• A failure inside Reconcile that wraps a controller-runtime transport error (connection refused, TLS handshake timeout, context deadline exceeded) points at the management cluster's apiserver, or at a rotated / expired ServiceAccount token.
• A resolve blueprint / resolve credential / resolve cloud / read namespace assignment failure points at one of the sibling sub-contexts' rows being absent — a blueprint version, credential, cloud, or assignment a declared ProvisionedResource still references.

Remediation. Restore the failing dependency — Postgres reachability, apiserver reachability, the ServiceAccount credentials, or the missing sibling-context row. No broker data is lost: the ProvisionedResource rows are durable in Postgres. Once the dependency is healthy the next sweep reconciles every resource and /readyz returns to HTTP 200.

5. Bootstrap token issued but the Node never registers

Symptom. A ProvisionedResource sits in Enrolling across many sweeps and never crosses into Ready. Crossplane reports the substrate Ready.

Diagnostic.

• The transition holds Enrolling while ObservedXR.NodeRegistered is false. NodeRegistered is read from a NodeRegistered status condition the Composition surfaces once the plexd agent has consumed the injected bootstrap token and enrolled.
• Confirm the token was injected: the broker mints a token exactly once and threads its plaintext into the Composite Resource's spec.parameters injection site for the blueprint's InjectionStrategy. A token minted but not threaded means the blueprint's strategy and the Composition's patch path disagree.
• Confirm the token has not expired: a token un-redeemed past its TTL cannot be consumed, and the broker does not re-issue — the at-most-once gate HasBootstrapToken keeps the original token id on the aggregate.

Remediation. If the token expired before the agent could redeem it, the substrate must be re-provisioned with a fresh token: declare a new ProvisionedResource. If the agent is reachable but not enrolling, diagnose the plexd agent and the mesh-registration path on the substrate; the broker's part — minting and injecting the token — is complete once Enrolling is reached. Raising PLEXSPHERE_PROVISIONING_BROKER_TOKEN_TTL widens the redemption window for substrate that is slow to come up.

6. Composite Resource deleted out of band

Symptom. An operator (or another controller) deleted a rendered Composite Resource directly on the cluster. The ProvisionedResource row still shows Provisioning, Enrolling, or Ready.

Diagnostic.

• The reconcile derives its action exclusively from live cluster facts every tick, never from cached desired state. On the next sweep the ObservedXR snapshot reports XRExists false.
• The pure transition's converge arm treats a missing XR in any non-Failed phase as (Apply, Pending) — the resource walks back to Pending and the apply re-renders and re-applies the Composite Resource.

Remediation. None required — this is the case the reconcile loop exists to handle. The next sweep re-applies the Composite Resource idempotently and the resource re-converges through Provisioning and Enrolling back to Ready. If the resource does not self-heal, fall through to entry 1 (the namespace gate may be skipping) or entry 4 (the cluster may be unreachable).