Graceful Deletion

This page is the graceful-deletion companion to ./broker.md. The broker reference documents the create-to-Ready-or-Failed pipeline: how a Project operator's declaration becomes a persisted ProvisionedResource, how the render and reconcile loop drives the Crossplane Composite Resource toward Ready, and how a terminal Crossplane condition routes it to Failed. This page documents the other half of the lifecycle — the teardown machine: how a graceful-deletion request drains the plexd Node out of the mesh, deletes the Crossplane substrate, and releases the aggregate.

The two pages cover one bounded context and one aggregate (../../../internal/provisioning/broker/). They share the ubiquitous language, the pure transition function, the ports, the error sentinels, and the persistence schema; the split is editorial, not architectural. Read ./broker.md first for the create-pipeline half — the converge arm of the state machine, the render step, the bootstrap-token issuance, and the namespace gate are all documented there and are not repeated here.

The teardown machine has one hard invariant the create pipeline does not carry: the plexd Node must be drained out of the mesh before the Crossplane Composite Resource is deleted, because deleting the substrate first orphans a live mesh Peer that can no longer be drained cleanly. Almost every design choice on this page — two distinct teardown actions instead of one, the ordering of the teardown arm, the NodeDeregistrar port keyed by bootstrap-token id — exists to encode that ordering as a pure, unit-testable property.

Cross-references

• ./broker.md — the create-pipeline half of this bounded-context reference. The converge arm of the state machine, the render step, the eight ports' create-side roles, and the milestone limitations live there; this page documents only the teardown additions.
• ../../../internal/provisioning/broker/phase.go — the closed ProvisionedResourcePhase and Action sets and the pure, total Next transition function whose teardown arm this page documents.
• ../../../internal/provisioning/broker/errors.go — the closed sentinel set, including ErrNodeDeregistrationFailed.
• ../../../internal/provisioning/broker/ports.go — the port set, including the NodeDeregistrar consumer-defined seam and Repository.RecordDeletionIntent.
• ../../../internal/provisioning/broker/events/events.go — the closed EventType set and the deletion event payload structs.
• ../../../internal/provisioning/broker/service.go — the application-service facade; Service.Deprovision is the caller-facing entry point of teardown.
• ../../../internal/provisioning/broker/provisioned_resource.go — the ProvisionedResource aggregate and its RequestDeletion transition.
• ../../../internal/provisioning/broker/reconcile/reconcile.go, ../../../internal/provisioning/broker/reconcile/observe.go, and ../../../internal/provisioning/broker/reconcile/apply.go — the reconcile loop's observe → decide → apply → emit+persist tick, with the deregisterNode and deleteSubstrate apply arms.
• ../../../internal/platform/db/migrations/0027_provisioning_broker_deletion.sql — the deletion schema evolution: the widened CHECK constraints and the nullable deletion_requested_at column.
• ../../../cmd/plexsphere/provisioning_broker_factory_prod.go — the production composition root; brokerNodeDeregistrar is the anti-corruption adapter that satisfies the NodeDeregistrar port.
• ../../../tests/e2e/provisioning/graceful-deletion/chainsaw-test.yaml — the Chainsaw e2e suite that drives a declared ProvisionedResource through the full teardown machine and asserts the node-before-substrate ordering on a live kind cluster.

Ubiquitous language

The teardown vocabulary below extends the create-pipeline terms documented in ./broker.md. Names are preserved verbatim across the Go code, the SQL migration, the event payloads, and this runbook so a reader chasing a string from a log line finds it in the source without translation.

Term	Definition	Code anchor
PhaseDeregistering	The first teardown phase. Deletion has been requested and the broker is draining the plexd Node out of the mesh before any substrate is torn down. The reconcile holds this phase while the Node is still observed registered.	phase.go
PhaseDeprovisioning	The second teardown phase. The plexd Node has been deregistered and the broker is deleting the Crossplane Composite Resource and its ProviderConfig. The reconcile holds this phase while the Composite Resource is still observed present.	phase.go
PhaseDeleted	The terminal teardown phase. The Node is deregistered and the substrate is gone. Deleted is sticky — the transition function never moves a Deleted resource back toward a teardown action.	phase.go
ActionDeregisterNode	The reconcile action that drains the plexd Node out of the mesh — the first teardown step. The transition function emits it while a deletion-phase resource is still observed with its Node registered.	phase.go
ActionDeleteSubstrate	The reconcile action that deletes the Crossplane Composite Resource and ProviderConfig — the second teardown step. It is emitted only once the Node is observed deregistered, so it can never run before ActionDeregisterNode.	phase.go
NodeDeregistrar	The consumer-defined anti-corruption port onto the mesh. Deregister drains the Node; NodeRegistered reports whether the Node is still live. Keyed by BootstrapTokenID.	ports.go
Service.Deprovision	The application-service method that records a graceful-deletion request against an existing aggregate, advancing it into the teardown machine. The caller-facing entry point of teardown.	service.go
ProvisionedResource.RequestDeletion	The aggregate transition that records deletion intent: it returns a copy advanced to PhaseDeregistering, or an idempotent no-op when the resource is already tearing down.	provisioned_resource.go
Repository.RecordDeletionIntent	The transactional-outbox write that commits the teardown-phase advance and the ProvisionedResourceDeleting outbox event in one transaction — the transactional commit point of an accepted deletion request.	ports.go
deletion_requested_at	The nullable timestamptz column on plexsphere.provisioned_resources stamped when a deletion request is accepted. NULL means the resource has never been asked to delete.	0027_provisioning_broker_deletion.sql

Deletion state machine

The ProvisionedResource status state machine is one pure, total transition function — Next (phase.go) — over a closed eight-phase set. The create-pipeline arm and its first five phases (Pending, Provisioning, Enrolling, Ready, Failed) are documented in ./broker.md. This page documents the teardown arm and its three phases (Deregistering, Deprovisioning, Deleted).

Next reads no clock, no process memory, and no I/O: it derives the reconcile Action and the next ProvisionedResourcePhase solely from the live ObservedXR snapshot and the aggregate's current phase, so the same inputs always yield the same output and the machine is fully unit-testable without a cluster.

  Teardown arm — current phase ∈ {Deregistering, Deprovisioning, Deleted}; sticky

      Service.Deprovision  (deletion requested — RequestDeletion advances the aggregate)
              │
              ▼
  ┌────────────────────┐  observed.NodeRegistered == true
  │   Deregistering    │ ──────────────────────────────────┐
  │  (DeregisterNode)  │ ◀──────────────────────────────────┘
  └─────────┬──────────┘
            │  observed.NodeRegistered == false AND observed.XRExists == true
            ▼
  ┌────────────────────┐  XR still present
  │   Deprovisioning   │ ──────────────────────────────────┐
  │  (DeleteSubstrate) │ ◀──────────────────────────────────┘
  └─────────┬──────────┘
            │  observed.NodeRegistered == false AND observed.XRExists == false
            ▼
  ┌────────────────────┐
  │     Deleted        │  terminal — (Noop)
  └────────────────────┘

Key properties the transition guarantees:

• Teardown is sticky. Once the phase is Deregistering, Deprovisioning, or Deleted, the resource is never routed into the converge arm again — even when ObservedXR reports the substrate healthy. The arm is selected by the isDeletionPhase predicate, which is checked before the converge arm; a deletion-phase resource can only drain toward Deleted.
• The arm is keyed purely on ObservedXR. Inside the teardown arm the current phase only selected the branch; the next action and phase derive entirely from the observed facts. NodeRegistered is checked before XRExists, so swapping the two action branches is a detectable regression.
• Node-before-substrate ordering. While the plexd Node is observed registered the transition emits (DeregisterNode, Deregistering) regardless of whether the Composite Resource still exists — the substrate is never touched while the Node is live. Only once NodeRegistered is false does the transition emit (DeleteSubstrate, Deprovisioning). This is the hard ordering invariant: deleting the substrate first would orphan a live mesh Peer that can no longer be drained cleanly.
• Deleted is the sticky terminal sink. When the Node is gone and the Composite Resource is gone the transition emits (Noop, Deleted), and a resource already at Deleted always re-derives (Noop, Deleted).
• Next stays pure and total. Every (ObservedXR, ProvisionedResourcePhase) pair — including an unrecognised phase string — maps to a defined result. An unrecognised phase is treated as Pending, the safe converge-from-scratch entry point; the phase strings read for control flow are PhaseFailed and the three deletion phases, because their stickiness cannot be re-derived from ObservedXR alone.

Teardown carries two distinct actions — ActionDeregisterNode and ActionDeleteSubstrate — rather than a single collapsed ActionDelete. The DECISION block on the Action set in phase.go spells out why: a single delete action cannot encode the node-before-substrate ordering, whereas two distinct actions make the ordering a pure (ObservedXR, phase) property the transition function — not an adapter-side sequencing concern — owns.

What the reconcile loop does

Reconciler.Reconcile (reconcile/reconcile.go) runs one ProvisionedResource per call in four steps; the steps are the same for the converge and the teardown arm, only the actions differ:

1. Observe — observe (reconcile/observe.go) reads NodeRegistered from the NodeDeregistrar seam up front, resolves the Composite Resource's identity, Gets it from the management cluster, and folds the result into a broker.ObservedXR. The node-registration fact is carried through into both the XR-present and the XR-absent result, so a resource mid-teardown with the substrate already gone but the Peer still draining is still routed to ActionDeregisterNode.
2. Decide — call the pure broker.Next transition.
3. Apply — apply (reconcile/apply.go) switches on the action. ActionDeregisterNode calls deregisterNode, which drains the Node through NodeDeregistrar.Deregister. ActionDeleteSubstrate calls deleteSubstrate, which issues a controller-runtime Delete of the Composite Resource and then the ProviderConfig — the reverse of the create-pipeline apply order — with NotFound swallowed via client.IgnoreNotFound so a re-reconcile of an already-torn-down resource is a no-op. ActionNoop does nothing.
4. Emit + persist — when the phase changed, emit publishes the lifecycle event and then UpdatePhase writes the new phase. The emit runs before the persist so a crash between the two cannot silently drop a terminal event; the DECISION block at that step in reconcile.go records why. On the teardown arm only the crossing into PhaseDeleted carries an event — the intermediate phases Deregistering and Deprovisioning are quiet.

Ports

The teardown machine adds one port to the broker's closed port set; the create-pipeline ports are documented in ./broker.md. All ports are declared in framework-free terms so the domain layer stays free of pgx, controller-runtime, and k8s.io (ports.go).

NodeDeregistrar

NodeDeregistrar is the anti-corruption seam onto the mesh through which the reconcile loop's teardown arm drains and observes the plexd Node bound to a ProvisionedResource. It declares two methods:

• Deregister(ctx, id BootstrapTokenID) error — drains the Node out of the mesh, the first teardown step. The reconcile loop calls it strictly before any Composite Resource is deleted. An already-deregistered Node is the adapter's success case, not an error, so a retried teardown tick converges.
• NodeRegistered(ctx, id BootstrapTokenID) (bool, error) — reports whether the Node is still registered (live) in the mesh. This is the read the observe step folds into ObservedXR.NodeRegistered every tick, so the node-before-substrate ordering stays a pure function of observed facts.

The port is keyed by BootstrapTokenID, not by a mesh node id. The bootstrap-token id is the broker's own durable handle: the ProvisionedResource aggregate persists the bootstrap-token id and the token's consumed_at is what records the Node as registered. A node id, by contrast, is the mesh context's identifier — one the broker does not own and cannot mint. Keying the port on the bootstrap-token id keeps it consumer-defined and the broker free of the mesh context's identifier vocabulary; the DECISION block on the port in ports.go records the rejected alternative of resolving a node id first.

The production anti-corruption adapter is brokerNodeDeregistrar in provisioning_broker_factory_prod.go: it chains three seams — a bootstrap-token reader recovers the consuming Node id, a peer-by-node resolver finds that Node's live mesh Peer, and a peer commander drains it. Unit tests inject an in-memory fake behind the same port.

Repository.RecordDeletionIntent

RecordDeletionIntent is the transactional commit point of an accepted deletion request. It is a transactional-outbox write — it advances the ProvisionedResource into its first teardown phase, stamps deletion_requested_at, and appends the ProvisionedResourceDeleting outbox event, all in the same transaction. This mirrors Repository.Create rather than Repository.UpdatePhase: UpdatePhase is phase-write-only and emits no outbox event, whereas RecordDeletionIntent — like Create — couples a phase write to an outbox event atomically. The caller passes the aggregate already advanced by RequestDeletion, so resource.Phase() is the first teardown phase and the outbox event is derived from that same aggregate. A missing row surfaces ErrProvisionedResourceNotFound; the at-most-once outbox token makes a retried call append no second event row. The Postgres adapter runs the new sqlc query RequestProvisionedResourceDeletion.

Lifecycle events

Teardown adds two typed domain events to the broker's closed EventType set, bringing it to five (events/events.go). Both deletion payloads mirror the field set of ProvisionedResourceReady (EventID, OccurredAt, ProvisionedResourceID, ProjectID, XRName) — a deletion event needs no blueprint or credential references.

Event type (discriminator)	Trigger	Emitted by
broker.ProvisionedResourceDeleting	A graceful-deletion request is accepted: the aggregate crosses into Deregistering.	The application service, transactionally — RecordDeletionIntent appends it in the same transaction as the teardown-phase advance, mirroring how Create appends ProvisionedResourceRequested.
broker.ProvisionedResourceDeleted	The aggregate crosses into the terminal Deleted phase: the Node is deregistered and the substrate is gone.	The reconcile loop's emit step, on the crossing into PhaseDeleted.

The intermediate teardown phases Deregistering and Deprovisioning are quiet: the reconcile loop's emit step writes the phase but publishes nothing for them — only the terminal Deleted crossing carries a teardown event. The two discriminator strings match the event_type CHECK constraint the 0027 migration installs character-for-character, so the storage CHECK and the application discriminator cannot drift.

Error sentinels

Teardown adds one sentinel to the broker's closed domain set, bringing it to nine (errors.go). Callers branch on it via errors.Is — wrapping with fmt.Errorf("%w", …) is fine, identity must remain intact.

• ErrNodeDeregistrationFailed — surfaces from the reconcile loop's teardown apply arm when NodeDeregistrar.Deregister returns an error. The deregister step runs strictly before the substrate is deleted, so the reconcile tick aborts on this error before any phase write or Composite Resource delete — the substrate is never torn down while the plexd Node is still live. Callers branch on it via errors.Is to prove the teardown stalled at the deregistration step rather than at the substrate-delete step.

One further sentinel names a teardown wiring bug rather than an observable domain failure, so — like the broker's other wiring sentinels — it lives outside the closed domain set, in the reconcile/ subpackage (reconcile/config.go):

• ErrReconcilerNodeDeregistrarRequired — returned by NewReconciler when it is handed a nil NodeDeregistrar. A misconfigured composition root fails fast at boot rather than panicking on the first teardown tick.

Persistence

Migration 0027_provisioning_broker_deletion.sql is the schema evolution graceful deletion needs. It does not create a new table — teardown reuses the create-pipeline tables — but evolves the two closed-set CHECK constraints 0026 installed and adds one nullable column:

• provisioned_resources_status_check is widened from the five create-pipeline phases to the eight-phase set: the three teardown-machine phases (Deregistering, Deprovisioning, Deleted) are added alongside the original five. The CHECK pins the closed phase set at the storage boundary as defense-in-depth behind the application's own closed-set validation.
• provisioned_resource_outbox_token_event_type_check is widened from three discriminators to the five-discriminator set: the two deletion lifecycle event types (broker.ProvisionedResourceDeleting, broker.ProvisionedResourceDeleted) are added alongside the original three. The literal strings match the broker events package's EventType() values character-for-character.
• deletion_requested_at timestamptz is a new nullable column on plexsphere.provisioned_resources, added with no DEFAULT. It is stamped when a graceful-deletion request is accepted and the aggregate advances into Deregistering; it stays NULL for the whole create-pipeline life of a resource that has never been asked to delete. NULL models the pre-request state honestly — a DEFAULT would conflate "never requested" with "requested at row-creation time".

The migration adds one sqlc query, RequestProvisionedResourceDeletion: it is the only query that writes deletion_requested_at, so the column moves exactly once, at the one transition that owns it. Reusing UpdateProvisionedResourcePhase was rejected because that query fires on every create-pipeline transition and would force every non-deletion write to pass a NULL it never means to touch.

Because the migration changes only CHECK constraints and adds one nullable column — no secret bytes, no hash-chained forensic rows, no retention material — its Down block is a real reversal: it drops the column and narrows both CHECK constraints back to the 0026 sets. A downgrade resurrects no compliance-sensitive plaintext on a subsequent Up.

Recovery runbook

This section is the operator-facing companion to the reference above. Each entry follows the same shape — Symptom, Diagnostic, Remediation — and is scoped to a single teardown failure mode. The reconcile loop is total and idempotent, so unless an entry says otherwise the safe baseline action is to let the next tick retry and watch the resource drain toward Deleted.

1. Deregistration failure

Symptom. A ProvisionedResource mid-teardown holds the Deregistering phase across many sweeps and never advances. The reconcile tick fails with an error that satisfies errors.Is(err, broker.ErrNodeDeregistrationFailed). The Crossplane Composite Resource is still present — the substrate has not been deleted.

Diagnostic.

• The teardown arm emits ActionDeregisterNode while the plexd Node is observed registered, and deregisterNode aborts the tick the moment NodeDeregistrar.Deregister returns an error — before any phase write and before any substrate delete. Deregistering plus ErrNodeDeregistrationFailed is therefore the fingerprint of a stalled drain, not a stalled substrate delete.
• Inspect the wrapped cause: the brokerNodeDeregistrar adapter chains a bootstrap-token reader, a peer-by-node resolver, and a peer commander. The wrapped error names which seam failed — the token could not be resolved to a consuming Node, the Node has no live mesh Peer, or the peer commander could not drain it.
• Confirm the mesh control plane is reachable: a transport failure to the peer commander surfaces here, distinct from a substrate fault.

Remediation. Restore the failing seam — mesh control-plane reachability, the peer commander, or the credentials the adapter drains the Peer with. No teardown state is lost: the resource holds Deregistering and the substrate is intact, so once the drain succeeds the next tick advances the resource to Deprovisioning and proceeds. Because the substrate was never touched, there is no orphaned mesh Peer to clean up. Do not delete the Composite Resource by hand to "move teardown along" — that orphans the still-live Peer the deregistration step exists to drain.

2. Substrate-delete failure

Symptom. A ProvisionedResource mid-teardown holds the Deprovisioning phase across many sweeps. The plexd Node is already deregistered, but the reconcile tick fails when deleting the Crossplane Composite Resource or the ProviderConfig.

Diagnostic.

• Deprovisioning means the teardown arm already observed NodeRegistered == false and emitted ActionDeleteSubstrate — the Node was drained cleanly and the failure is downstream of it. There is no orphaned-Peer risk in this state: the node-before-substrate ordering has already done its job.
• The reconcile's deleteSubstrate swallows NotFound via client.IgnoreNotFound, so a failing tick here is a real delete error, not an already-absent target — an apiserver transport failure, an RBAC denial on the delete verb, or a webhook rejecting the delete.
• Inspect the wrapped error: it names whether the Composite Resource or the ProviderConfig delete failed.

Remediation. Restore management-cluster reachability or the RBAC the broker's controller-runtime client deletes with. The teardown is idempotent: once the delete succeeds the next tick observes XRExists == false and drains the resource straight to Deleted, emitting the broker.ProvisionedResourceDeleted event. If a webhook is rejecting the delete, fix the webhook — do not bypass it by force.

3. Resource stuck mid-teardown

Symptom. A ProvisionedResource sits in Deregistering or Deprovisioning across many sweeps with no reconcile error in the log — the tick succeeds each time but the phase does not advance.

Diagnostic.

• A teardown phase that holds with no error means the observed facts have not changed: in Deregistering, NodeDeregistrar.NodeRegistered keeps reporting the Node live; in Deprovisioning, the Composite Resource Get keeps succeeding. The transition is correctly holding the phase — the blocker is on the cluster, not in the broker.
• The common Deprovisioning cause is a Crossplane managed-resource finalizer blocking the Composite Resource delete: the broker's Delete call returns success (it only requests deletion), but the XR lingers with a deletionTimestamp set while a managed resource awaits external-API teardown. kubectl get the Composite Resource in the per-Project namespace and inspect .metadata.finalizers and .metadata.deletionTimestamp.
• The common Deregistering cause is a mesh Peer that will not drain — in-flight connections or a peer commander that reports success without the Peer actually leaving the mesh.

Remediation. Clear the blocker on the cluster, not in the broker. For a Deprovisioning finalizer, let the Crossplane managed resource finish its external-API teardown, or — if it is genuinely stuck — clear the offending finalizer on the managed resource so Crossplane can complete the XR delete. Once the Composite Resource is gone the next tick drains the resource to Deleted. Do not edit the deletion_requested_at column or the status phase directly in Postgres to "unstick" the resource — the phase is derived from live cluster facts every tick, so a hand-edited phase is overwritten on the next sweep and only obscures the real blocker.

4. XR deleted out of band

Symptom. An operator (or another controller) deleted the Crossplane Composite Resource directly on the cluster while the ProvisionedResource was mid-teardown.

Diagnostic.

• The reconcile derives its action exclusively from live cluster facts every tick, never from cached desired state. On the next tick the observe step Gets the Composite Resource, hits NotFound, and folds XRExists == false into the ObservedXR snapshot.
• The outcome depends on the Node: if the plexd Node is already deregistered, the teardown arm observes NodeRegistered == false and XRExists == false and emits (Noop, Deleted) — the resource drains straight to Deleted on that tick. If the Node is still registered, the teardown arm still emits (DeregisterNode, Deregistering) first — an out-of-band XR delete does not skip the node-before-substrate drain.

Remediation. None required when the Node is already gone — this is the case the idempotent teardown machine exists to handle. The next tick records Deleted and emits the broker.ProvisionedResourceDeleted event with no operator action. If the Node was still registered when the XR was deleted out of band, the resource correctly returns to Deregistering until the drain completes; let the reconcile loop finish the teardown rather than forcing the phase.