Management Fleet

Authoritative bounded-context reference for the Management Fleet (internal/provisioning/managementfleet/). It owns the durable record of which Kubernetes clusters host the provisioning substrate (Crossplane v2 + the External Secrets Operator) and which cluster owns each Project, and reconciles a dedicated RBAC-scoped namespace per assigned Project.

The context has no HTTP surface: callers reach it through the in-process Service facade, and the reconcile loop drives the per-Project namespace off live cluster state. The closed port set keeps the domain and application layers free of pgx and controller-runtime (ports.go). This milestone registers clusters, places Projects, reconciles namespaces, and verifies a cluster carries an Available substrate before assignment — installing the substrate stays with the deploy/ Helm chart.

This reference is a single page: the surface is narrow (two aggregates plus the provider-install record, two pure state machines, six ports, eight sentinels) and the pieces travel in lockstep. The Recovery runbook at the foot is the operator-facing companion.

Cross-references

• ../../contributing/layout.md — the bounded-context map row that locates internal/provisioning/managementfleet inside the codebase and enumerates the depguard rules that confine pgx to repo/, controller-runtime to reconcile/, and bar cross-context imports.
• ./credentials.md and ./credential-pool.md — the sibling provisioning contexts. The management fleet mirrors their module shape (a domain package, a repo/ adapter, a port set, an events subpackage) but owns inventory and reconcile rather than secret material.
• ../identity/tenancy.md — the Domain → Project → Resource → Node aggregate model. A Project is owned by the tenancy context; the management fleet observes it only as the key of a ProjectClusterAssignment and as the seed of the per-Project namespace name.
• ../../../internal/provisioning/managementfleet/doc.go — the package-level pin of the ubiquitous language and the reconcile-model DECISION block (why a boot-probe-plus-ticker closure rather than a watch-driven controller-runtime Manager).
• ../../../internal/platform/db/migrations/0023_management_fleet.sql — the persistence schema for plexsphere.management_clusters and plexsphere.project_cluster_assignments.
• ../../../cmd/plexsphere/management_fleet_factory_prod.go — the production composition root: the env loader, the in-cluster client, the reconcile sweep, and the boot-probe / steady-state-ticker pair.
• ../../../tests/e2e/provisioning/management-fleet/chainsaw-test.yaml — the Chainsaw e2e suite that stands up a kind cluster with digest-pinned Crossplane v2 and the External Secrets Operator, runs the reconcile, and asserts the per-Project Namespace, Role, RoleBinding, ServiceAccount, ResourceQuota, and a passing verify gate.

Ubiquitous language

The terms below travel together across the Go code, the SQL migration, the domain-event payloads, the structured-log attributes, and the recovery runbook. Names are preserved verbatim so a reader chasing a string from a log line finds it in the source without translation.

Term	Definition	Code anchor
ManagementCluster	The aggregate root modelling one registered Kubernetes cluster in the fleet. Carries (id, name, slug, region, kubeconfigSecretRef, status, createdAt, updatedAt). The application mints a UUIDv7 id at registration; the slug is the unique human-facing key the assignment model shards on. The fields are unexported and reached through accessors so the creation invariants hold only through the two constructors.	cluster.go
ClusterID	The UUIDv7 identity of a ManagementCluster, a named wrapper over uuid.UUID. The String() projection is the canonical hyphenated lowercase form. The zero value is rejected by every creation invariant.	cluster.go
ProjectClusterAssignment	The aggregate root keyed by Project id, recording which management cluster owns a Project, the region the placement landed in, the per-Project namespace name, and that namespace's lifecycle phase. The (projectID, managementClusterID) pairing is immutable once the Project has provisioned resources.	assignment.go
ProjectID	The UUID identity of a Project as the fleet observes it — a named wrapper over uuid.UUID, distinct at compile time from ClusterID. It is the assignment key and the seed of the namespace name.	namespace.go
per-Project namespace	The Kubernetes Namespace carved out for one Project on its management cluster. Its name is plexsphere-project-<uuid> — the fixed prefix followed by the canonical hyphenated lowercase Project UUID. The name is a deterministic function of the Project id (a 55-character RFC 1123 label) so the persisted column and the live cluster name can never drift.	namespace.go
NamespacePhase	The closed-set lifecycle of the per-Project namespace: Pending, Provisioning, Ready, Degraded, Terminating, Deleted. The phase is advanced exclusively by the pure transition function over observed cluster facts, never from process memory.	phase.go
ObservedCluster	The snapshot of per-Project facts the reconcile loop reads live from a management cluster on a single tick: whether the Namespace, Role, RoleBinding, ServiceAccount, and ResourceQuota exist, and whether the verify gate passed. Every field is a fact the cluster reports, not a desired state.	phase.go
Action	The reconcile instruction the transition function emits for one tick: Noop, Converge, or Delete.	phase.go
Service	The in-process application-service facade. Methods: RegisterCluster, AssignProjectToCluster, LookupAssignment, SelectClusterForRegion, MarkTerminating, Unassign. It orchestrates the Repository and owns the transaction boundary; the aggregates enforce their own creation invariants.	service.go
Reconciler	The controller-runtime adapter that reconciles exactly one ProjectClusterAssignment per call: observe cluster facts, decide via the pure transition, apply the action, persist the phase, emit the lifecycle event.	reconcile/reconcile.go
ClusterProviderInstall	The aggregate root recording one Crossplane provider package on one management cluster: (managementClusterID, packageSource, packageVersion, phase, statusMessage, createdAt, updatedAt). The aggregate key is the (cluster, source) pair, because a Crossplane Provider object is cluster-scoped and carries exactly one spec.package — a cluster runs at most one version of a given source.	providerinstall.go
ProviderInstallPhase	The closed-set lifecycle of a provider package on a cluster: Pending, Installing, Serving, Failed, Conflict. Advanced exclusively by the pure DeriveProviderInstallPhase over facts observed on the cluster.	providerinstall.go
ObservedProviderPackage	The snapshot of live facts the installer reads from one Provider object on a single tick: whether it exists, whether its spec.package names the source the fleet expects, and the status and message of its Healthy condition.	providerinstall.go
ProviderInstaller	The controller-runtime adapter that converges one cluster's provider packages per call: resolve the desired set, apply one Provider per package, observe each Healthy condition, record the phase, and reap the records whose package is no longer desired.	reconcile/providers.go
verify gate	The read-only readiness check that confirms a management cluster serves the apiextensions.crossplane.io, pkg.crossplane.io, and external-secrets.io API groups and that the Crossplane and External Secrets Operator controller Deployments report Available.	reconcile/verify.go

Aggregates

The context owns two aggregate roots. They are modified one per transaction; the application service coordinates the cross-aggregate read (the resource count) inside the same RunInTx as the assignment write so the immutability invariant holds against one consistent snapshot.

ManagementCluster — invariants

NewManagementCluster and HydrateManagementCluster are the only paths that produce a valid aggregate. Both enforce the same creation invariants, so a row that drifts from them in storage is rejected at the hydration boundary rather than surfacing later in the reconcile loop.

Invariant	Layer	Failure mode
id is non-zero.	Aggregate constructor.	ErrInvalidInput — a zero ClusterID would collide with the "not yet assigned" sentinel.
slug is non-empty.	Aggregate constructor; SQL UNIQUE constraint management_clusters_slug_unique.	ErrInvalidInput on an empty slug; a duplicate slug surfaces from the repository adapter.
createdAt and updatedAt are non-zero, with createdAt ≤ updatedAt.	Aggregate constructor.	ErrInvalidInput — an unordered or zero timestamp pair is a corrupt row.

ProjectClusterAssignment — invariants

Invariant	Layer	Failure mode
projectID and managementClusterID are both non-zero; timestamps are non-zero and ordered.	Aggregate constructor.	ErrInvalidInput.
namespaceName equals NamespaceName(projectID).	Aggregate constructor — the field is derived, never caller-supplied, and re-derived on hydration.	A persisted name that drifts from the derivation cannot reach the reconcile loop: hydration recomputes it.
project_id is the PRIMARY KEY — a Project is assigned to at most one cluster at a time.	SQL PRIMARY KEY.	A second CreateAssignment for the same Project surfaces a constraint violation from the repository adapter.
The (projectID, managementClusterID) pairing is immutable once the Project owns at least one resource.	Application service — CountResourcesForProject inside the placement RunInTx.	ErrAssignmentImmutable — re-pointing would orphan the Composite Resources, ProviderConfigs, and Secrets in the old namespace.
namespace_phase is one of the six closed-set values.	SQL CHECK constraint project_cluster_assignments_phase_check; application validation.	The CHECK is defence-in-depth: an out-of-set phase string cannot reach a persisted row even on an application bug.
Neither a Project nor a management cluster can be deleted while a live assignment references it.	SQL FOREIGN KEY … ON DELETE RESTRICT on both project_id and management_cluster_id.	The delete fails closed at the database — a CASCADE would silently orphan a live namespace with no inventory record that it needs tearing down.

Namespace-phase state machine

The per-Project namespace walks a six-phase lifecycle. The phase is advanced only by the pure, total transition function Next (phase.go), which reads no clock, no process memory, and no I/O: it derives the Action and the next NamespacePhase solely from the live ObservedCluster snapshot and the assignment's current phase. The same inputs always yield the same output, so the machine is fully unit-testable without a cluster.

The machine has two arms, selected by teardown intent:

  Converge arm — current phase ∈ {Pending, Provisioning, Ready, Degraded}

    ┌──────────┐   resources missing OR verify failing
    │ Pending  │ ──────────────────────────────────────┐
    └────┬─────┘                                        │
         │                                              ▼
         │                                   ┌────────────────────┐
         │            all resources present  │   Provisioning     │
         │            AND verify passed       │  (Converge)        │
         ▼                                    └─────────┬──────────┘
  ┌──────────────┐ ◀───────────────────────────────────┘
  │    Ready     │  all resources present AND verify passed
  │  (Converge,  │
  │ then Noop)   │ ──┐ a required resource went missing
  └──────┬───────┘   │  OR verify started failing
         │           ▼
         │   ┌────────────────────┐
         │   │     Degraded       │  keeps converging to repair
         │   │    (Converge)      │
         │   └────────────────────┘
         │
         │  Service.MarkTerminating  (teardown requested)
         ▼
  Teardown arm — current phase ∈ {Terminating, Deleted}; sticky

  ┌────────────────────┐  resources still remain
  │   Terminating      │ ──────────────────────────┐
  │    (Delete)        │ ◀──────────────────────────┘
  └─────────┬──────────┘
            │  no per-Project resources remain on the cluster
            ▼
  ┌────────────────────┐
  │     Deleted        │  terminal — (Noop)
  └────────────────────┘

Key properties the transition guarantees:

• Teardown is sticky. Once the phase is Terminating or Deleted the namespace never converges again; it can only drain to Deleted.
• Degraded is distinct from Provisioning. A namespace that reached Ready and then lost a resource is marked Degraded, not Provisioning, so an operator can tell a first install apart from a regression. Both keep emitting Converge.
• Ready is event-worthy once. A namespace arriving at Ready from a lower phase emits Converge so the caller records the crossing; on the next tick it settles to Noop.
• The function is total. Every (ObservedCluster, NamespacePhase) pair maps to a defined result; an unrecognised phase string is treated as Pending, the safe converge-from-scratch entry point.

What the reconcile loop does with the result

Reconciler.Reconcile (reconcile/reconcile.go) runs one assignment per call in four steps:

1. Observe — Get the five per-Project resources from the cluster and run the verify gate, building an ObservedCluster. A NotFound marks a resource absent; any other API error is an infrastructure failure that propagates. An ErrClusterUnhealthy from the verify gate collapses into VerifyPassed = false — the namespace keeps converging — while a non-unhealthy gate error propagates.
2. Decide — call the pure Next transition.
3. Apply — Converge creates or repairs the Namespace, then the RBAC objects, then the ResourceQuota, in that order; Delete tears the namespace down; Noop does nothing. Every step is idempotent.
4. Persist + emit — when the phase changed, write it through the Repository and, on a crossing into Ready or Deleted, publish the matching lifecycle event.

Re-running Reconcile against an already-converged or already-torn-down cluster is success.

Region-assignment resolver

While Reconcile converges one assignment's namespace against live cluster facts, a RegionAssignmentResolver decides where a Project is placed in the first place and whether an existing placement still matches its tenant's region. It runs each management-fleet sweep and has two halves — scheduling and migration — both governed by one exact-match contract: a placement is correct iff ManagementCluster.Region() == Domain.Region(). The match is exact string equality; there is no nearest-region fallback and no normalisation beyond the kebab-case validation ParseRegion already applied at construction.

A guiding rule frames both halves: a per-Project condition — no cluster for the region, a blocked migration — is a skip-and-WARN, never a sweep failure. Only an infrastructure error (a repository fault, an apiserver outage) fails a sweep. So a single misplaceable Project never stalls the placement of every other Project in the fleet.

Scheduling — Schedule()

Schedule() places each schedulable Project. A Project is schedulable as soon as it exists and has no assignment yet — placement is eager, it does not wait for the Project to own a resource. This is deliberate: the provisioned resource-create path resolves the Management Fleet assignment synchronously as a fail-closed admission gate (the Provision DECISION in the Provisioning Broker), so a brand-new Project that owns no resource would have no assignment and its first resource create would fail with "project cluster assignment not found". Placing the Project eagerly closes that chicken-and-egg; the trade-off — a per-Project namespace for Projects that may never provision — is accepted because the namespace converge is idempotent and an unplaced Project is the more surprising state to debug.

For each schedulable Project, the resolver looks for a registered management cluster whose region exactly matches the Project's Domain region:

• A cluster matches. The resolver creates a Pending assignment on it and emits a ProjectClusterAssigned event.
• No cluster matches. The resolver logs a WARN and skips the Project — no assignment row is created. The Project stays unplaced until a cluster for its region is registered; the next sweep retries. (An unpinned Domain — empty region — is placed under the same free-placement path that predates region pinning.)

Boot-time demo cluster seed

A fresh stack starts with an empty fleet inventory, so the resolver has nowhere to place a schedulable Project and logs the WARN above on every tick. To make the provisioned-resource flow reachable out of the box, the platform seeds a single demo ManagementCluster at boot — an idempotent, existence-based reconcile. It runs before the boot-time reconcile sweep, so the first sweep already has a cluster to place Projects onto.

One cluster is deliberate, not incidental. The demo tenants the bootstrap Job seeds carry no Domain region pin, and an empty-region placement resolves only against a single-cluster fleet: a second seeded cluster would make every empty-region placement ambiguous (ErrNoClusterForRegion) and the resolver would skip the Project. The single demo cluster is therefore the exact shape the empty-region placement rule requires. To exercise a multi-cluster fleet on a dev stack, region-pin the demo Domains so each Project resolves to a named-region cluster instead of the single-cluster free-placement path.

The seed gates /readyz under the probe name management-fleet-seeds: a missing demo cluster flips readiness to 503 and self-heals on the next idempotent tick.

Migration — MigrateOutOfRegion()

Re-pinning a Domain's region (a PATCH, since the region is mutable unlike the immutable slug) re-targets every Project beneath it. The Project region is derived from the Domain, not stored on the Project, so one Domain re-pin can leave many assignments pointing at a cluster whose region no longer matches. MigrateOutOfRegion() reconciles each such assignment one step per sweep, and the path forks on whether the Project owns resources.

Zero-resource Project — migrated automatically. The resolver drives the move itself, advancing one step per sweep:

1. It advances the current assignment's namespace phase into the teardown arm — from Pending, Provisioning, Ready, or Degraded to Terminating.
2. While the namespace is Terminating, it waits; the reconcile loop drains the per-Project namespace, RBAC, and quota.
3. Once the old assignment reaches Deleted, it re-points the Project to a fresh Pending assignment on a correct-region cluster, emitting ProjectClusterAssigned.

Because the state machine advances by a single step each sweep, a migration spans several sweeps; this is expected.

Resource-owning Project — blocked. When the Project owns resources, the (projectID, managementClusterID) pairing is immutable: the resolver does not move it. It logs a WARN, leaves the assignment untouched, and the sweep still returns success. Re-pointing a Project with live resources would orphan its Composite Resources, ProviderConfigs, and Secrets in the old region's namespace — the same fail-closed invariant ErrAssignmentImmutable guards in the application service. To migrate such a Project deliberately, drain its resources on the old cluster first (drive the namespace through Terminating → Deleted and Unassign), then let the next sweep place it in the new region.

Operators inspect the resulting placement — the owning cluster, the region the assignment landed in, and the namespace phase — through GET /v1/projects/{project_id}/management-cluster-assignment (see the Management Fleet HTTP API).

Per-Project resources

For every assignment the converge step provisions five objects on the management cluster. All five carry the Kubernetes recommended labels app.kubernetes.io/managed-by=plexsphere, app.kubernetes.io/part-of=management-fleet, app.kubernetes.io/component=<role>, and app.kubernetes.io/instance=<namespace name>, so one app.kubernetes.io/instance selector scopes to a single Project's full resource set (reconcile/labels.go).

When the assignment is region-pinned, the same five objects additionally carry the well-known Kubernetes topology label topology.kubernetes.io/region=<region>, set to the assignment's region. This label is distinct from the app.kubernetes.io/* recommended set: it locates a Project's resources in a placement region rather than describing the application. It is omitted for a region-less (unpinned) Domain, so a label selector on topology.kubernetes.io/region matches only region-pinned Projects.

Resource	Name	Purpose
Namespace	plexsphere-project-<uuid>	The RBAC and quota boundary for the Project's resources. Carries a kubernetes.io/description annotation.
Role	plexsphere-project	A namespace-scoped, least-privilege Role. Its single rule grants get, list, watch on configmaps, secrets, events, serviceaccounts in the core API group — no create/update/delete, no cluster-admin, and never the * wildcard in apiGroups, resources, or verbs.
RoleBinding	plexsphere-project	Binds the Role to the per-Project ServiceAccount.
ServiceAccount	plexsphere-project	The dedicated identity for the Project's in-namespace provisioning workload.
ResourceQuota	plexsphere-project-quota	Caps the Project's footprint on the shared cluster — object counts (ConfigMaps, Secrets, Pods, Services) and compute (CPU/memory requests and limits) — so one Project cannot starve another.

The RBAC grant is deliberately read-only at this milestone: no requirement asks the in-namespace workload to write, and an explicit, narrow grant keeps the bootstrap RBAC at the minimum the reconcile loop can audit. A future task that needs write verbs widens projectRoleRules deliberately and records the justification in the DECISION block on reconcile/rbac.go.

Provider packages

A Cloud declares a set of Crossplane provider packages that serve it. Assigning that Cloud to a Project is what causes every declared package to be installed on the management cluster hosting the Project: the fleet sweep converges each cluster's packages before it walks that cluster's namespaces, so a Project reaching Ready finds its controllers already coming up.

What is desired, and who decides

A package is desired on a cluster when some Cloud that declares it holds an approved Cloud Assignment to a Project the cluster hosts. Every package the Cloud declares counts, not just one: a Cloud naming an object-store provider and a compute provider puts both on the cluster. The materialised marker is deliberately not consulted: it records that the ReBAC cloud#uses tuple landed, which is an authorization-graph concern, while the substrate requirement follows from the operator's decision to approve. Gating the install on tuple convergence would leave a Project waiting for a controller because of an unrelated SpiceDB lag.

The join spans three bounded contexts, so it lives behind the DesiredProviderPackagesResolver port whose adapter is built at the composition root — the one seat where contexts may meet.

Which packages a Cloud declares is answered per mode. A Cloud that declares its packages inline answers from its own set; a Cloud that references a ProviderBundle answers from the one version of that bundle it pins, resolved on every sweep, cached nowhere and recorded on no Cloud. Editing a bundle publishes a new version and moves no pin, so it changes nothing any cluster converges with: the sweep after the edit re-reads the same pinned declaration. What changes a cluster is a promotion: a write on one Cloud's pin. The next sweep after it converges the Clouds that moved. Two Clouds on one bundle at different pins are two independent answers, so a staged rollout reaches the clusters hosting the promoted Clouds and no others. A resolution that fails fails the sweep: reporting an empty desired set instead would reap every install record on the cluster.

A referencing Cloud may deviate from the version it pins, and the same resolution carries the deviation: the pinned version's packages are merged with that Cloud's own override set before the desired-package join sees them, an override replacing the version of the member naming the same source and joining the set where the pinned version carries no such source. Nothing is removed. The fleet holds no code for this — it asks what each Cloud converges with and reads back the effective set — so two Clouds on one pin whose overrides differ are two independent answers the way two pins are. An override alone can therefore put two versions of one source on a shared cluster, which is the competing-versions arm of the Conflict phase.

What the installer applies

One pkg.crossplane.io/v1 Provider per package, Server-Side-Applied under the field manager plexsphere-management-fleet. The object name is derived from the package source: its final path segment, with underscores folded to hyphens. For the dev stack's xpkg.upbound.io/upbound/provider-aws-s3 that is provider-aws-s3 — exactly the object deploy/local/base/provider-aws/provider.yaml already declares, so the installer converges the operator's own object rather than standing a second controller beside it.

The apply names only spec.package and two labels (app.kubernetes.io/managed-by=plexsphere, app.kubernetes.io/part-of=management-fleet). Every other field stays with whoever owns it, which is what lets the dev provider keep the runtimeConfigRef pinning its ServiceAccount.

A member of a maintained provider family needs a second object. The families are a closed set — aws, azure, azuread, gcp — and a source is a member when its final path segment reads provider-<family>-<service> for a family in that set and a non-empty service. Such a member carries no ProviderConfig CRD of its own; it declares a dependency on the family package that holds them, so the installer converges that family package alongside every member it applies.

The family source is the member's with the final path segment replaced by provider-family-<family>, registry and namespace preserved, so a member pulled from a mirror derives its family from that same mirror. The family version is the member's version with everything from the first @ stripped — a tag, never a digest pin. Crossplane records a digest-pinned dependency target in its package Lock in a shape the member's own semantic-version constraint cannot match, which leaves the member unresolvable; the tag-only form is the one the dependency resolves. That is also why a Cloud may not declare a family package with a digest at all: the write is refused with the reason on the field, so the pin is never accepted and then quietly dropped further down.

That rule is younger than the column it guards, so a row written before it still reads back — refusing to hydrate it would take the Cloud out of every read at once and stop the converge for every cluster. The installer does not repair such a row by stripping the pin. It records the family package Conflict, naming the declared version, and applies nothing: converging it at the tag alone would run whatever that tag resolves to today, which is the substitution a digest pin exists to prevent. The operator repairs the declaration through the write path.

The family Provider takes the name Crossplane gives that object when it resolves a member's dependency, which is not the member rule. It is Crossplane's rule reproduced: the repository path — the source minus a registry host, where a first segment counts as a host only when it is localhost or carries a . or a : — run through xpkg.ToDNSLabel, which keeps [a-z0-9], turns ., /, : and - into hyphens, drops every other byte, and stops after 63. For xpkg.upbound.io/upbound/provider-family-aws that is upbound-provider-family-aws. The reproduction is byte-exact on purpose: a name one byte off Crossplane's would stand a second Provider for one package, which is the outcome the whole rule exists to avoid.

A derived family version is an inference, not a declaration — no Cloud names it — so the family apply carries no Force, and a declared package's apply does. Where another field manager already owns spec.package on the family object at a different value, the apiserver refuses the apply and the fleet records Conflict instead of taking the field. The owner that matters is Crossplane's own package manager, which writes that same field from its dependency resolution against the registry's tag list; two writers with independent opinions and Force on the fleet's side would mean one rewrite per reconcile tick and one ProviderRevision per flip. Server-Side Apply raises no conflict where the value already agrees, so the dev stack's hand-applied family object — pinned to the same tag — stays co-owned and keeps the runtimeConfigRef pinning the family ServiceAccount.

Because the object name and the recorded package must agree with the manifest byte-for-byte, the seeded demo Cloud records the manifest's digest-pinned version rather than a bare tag, and tests/workspace/seed_provider_package_pin_test.go fails the build when the two drift. The same file holds the family half of that agreement: the fleet derives the family at the member's tag, so a manifest family tag that differs is the disagreement the apiserver refuses, and the gate catches it offline instead.

A family install is visible as a record of its own. It lands as a ClusterProviderInstall under its own (cluster, source) key and emits the same Serving and Failed lifecycle events every other package does. An assignment's provider_installs array holds the declared packages only, so a family install does not appear there, and neither Serving nor Failed nor Conflict on a family reaches that array. Read it directly instead:

SELECT package_source, package_version, phase, status_message
  FROM plexsphere.management_cluster_provider_installs
 WHERE management_cluster_id = '<cluster>';

Phases

Phase	Meaning
Pending	No sweep has acted on this package yet. On the assignment read surface it also covers the window before the Project has been placed on a cluster.
Installing	The Provider is applied and the package manager has not yet reported it Healthy.
Serving	The package manager reports Healthy — the Project can provision against the Cloud.
Failed	The package manager reports unhealthy. Its own condition message travels on the record.
Conflict	The desired state cannot be applied without overwriting something the platform does not own. Nothing is applied.

Conflict arises five ways: two Clouds naming one package source at different versions; two package sources deriving one Provider object name; an existing Provider under the derived name whose spec.package serves a different source; an apply the apiserver refuses because another field manager owns spec.package at a different value; and a family package a Cloud declares with a digest pin. None is resolved automatically. Picking a winner by version would need a total order OCI tags do not have (the 0076 backfill's own v0.0.0-unrecorded placeholder sorts below every real release), and picking by approval order would make cluster state depend on something invisible from the Cloud records. Reporting both sides leaves the resolution where the disagreement is: an operator patches one of the Clouds.

The shared-name arm is the one no Cloud can see coming. A Provider is cluster-scoped and the family name carries no registry — Crossplane's own rule drops it, and the fleet reproduces the rule — so xpkg.upbound.io/upbound/provider-family-aws and mirror.example.com/upbound/provider-family-aws land on one object. Every source claiming that name records Conflict and none is applied. Converging either would hand a cluster-wide controller to whichever source happened to sort first, and that controller is the one reading every AWS Cloud's credential Secret on the cluster.

A family source reaches the competing-versions arm one step removed: two members deriving one family at different tags leave the family with two desired versions, and its record carries its own message — two or more Clouds require this family package at different versions, the versions in sorted order, then the member sources the family was derived from. The wording differs because no Cloud declares a family version directly, and the members are named because they are what an operator can actually patch.

What the installer never does

It never deletes a Provider. Revoking the last assignment that needed a package drops only the platform's install record; the controller keeps running. Deleting one would tear down a controller while managed resources created through it may still be live — a Project mid-teardown, or a resource blocked on an external API — and a revoke-then-regrant would cycle a controller shared with unrelated Projects.

This is enforced by the grant, not only by the code: the plexsphere-api ClusterRole holds get, list, watch, create, update and patch on pkg.crossplane.io providers, and deliberately no delete. Removing a package is an operator action.

A derived family object outlives more than its record. When the last member of a family leaves the desired set the family's derived entry leaves with it, the reap drops the ClusterProviderInstall, and — by the rule above — the Provider keeps running with nothing left in the platform accounting for it. For a declared package the operator removed it themselves and knows to look; for a derived one they never named it, so list the leftovers by the labels the fleet stamps:

kubectl get provider.pkg.crossplane.io \
  -l app.kubernetes.io/part-of=management-fleet

Anything in that list with no matching row in plexsphere.management_cluster_provider_installs is a package the fleet once converged and no longer tracks. Deleting one is the same operator action as removing any other package.

Failure posture

A cluster whose API server does not serve pkg.crossplane.io has no Crossplane at all. The installer skips it with a WARN rather than failing: the verify gate already owns reporting an incomplete substrate, and failing here would flip the fleet's /readyz probe on a condition the installer cannot fix. Every other apiserver or repository error fails the sweep so the ticker retries it. An unhealthy or conflicted package is recorded state, never an error — package state never flips readiness.

Ports

The context reaches every collaborator through one of six ports declared in framework-free terms — context, time, and the package's own aggregates — so the domain layer stays free of pgx, controller-runtime, and k8s.io (ports.go). The composition root wires concrete adapters; tests inject in-memory fakes.

Port	Methods	Adapter	Test seam
Repository	CreateCluster, GetCluster, ListClusters, CreateAssignment, GetAssignment, ListAssignmentsForCluster, UpdateAssignmentPhase, DeleteAssignment, CountResourcesForProject, RunInTx	Postgres in repo/managementfleet_pg.go, wrapping the sqlc-generated queries. Constraint-name dispatch maps SQLSTATE collisions and FK violations onto the sentinels. depguard confines pgx to this subpackage.	In-memory fakes in the unit tests; the repository adapter has its own pgtype-conversion and classification tests, and the integration suite drives the real adapter against a testcontainers Postgres.
ClusterClientFactory	HandleFor	Mints an opaque ClusterHandle for a registered cluster. At this milestone the binary runs inside the single management cluster, so the production wiring uses one in-cluster controller-runtime client rather than a per-cluster fan-out.	Fakes return a stub handle.
FleetHealthChecker	Verify	The Verifier in reconcile/verify.go — the read-only verify gate. depguard confines controller-runtime to the reconcile/ subpackage.	A fake controller-runtime client exercises both the healthy and unhealthy arms; the envtest and Chainsaw suites drive it against a real apiserver.
AuditSink	Record	A composition-root shim translating the local AuditEntry value object onto internal/audit.Entry — keeping the module free of an internal/audit import the no-cross-context-imports rule denies.	An in-memory recording sink.
Clock	Now	A wall-clock implementation at the composition root.	A frozen clock pins a deterministic now in unit tests.
DesiredProviderPackagesResolver	DesiredProviderPackages	A composition-root adapter joining three contexts: the fleet's assignments for the cluster, each Project's approved Cloud ids from the Cloud Assignment repository, and each Cloud's recorded package. The join lives at the composition root because it crosses bounded contexts, which a per-context repository must not do.	A stub returning a fixed desired set, including the two-versions-one-source case.

The reconcile loop additionally declares a narrow EventSink port local to the reconcile/ package (reconcile/reconcile.go): the lifecycle events it emits are an integration concern, not an aggregate write, so per the interface-segregation principle the adapter declares the smallest dependency it needs rather than widening Repository with an outbox surface.

Lifecycle events

The context defines six typed domain events (events/events.go). The EventType discriminator string is stable and becomes part of the wire contract once an event is emitted; the set is closed and pinned by a package-local drift gate.

Event type (discriminator)	Trigger	Payload
managementfleet.ManagementClusterRegistered	A cluster is first materialised in the fleet inventory.	Cluster identity, name, slug, region.
managementfleet.ProjectClusterAssigned	The region-assignment resolver places a Project onto the cluster that owns it (a fresh scheduling placement, or a re-point after a region migration).	Project identity, owning cluster identity, region, namespace name.
managementfleet.ProjectNamespaceReady	The per-Project namespace crosses into the Ready phase.	Project identity, namespace name.
managementfleet.ProjectNamespaceTerminated	The per-Project namespace crosses into the Deleted phase.	Project identity, namespace name.
managementfleet.ClusterProviderPackageServing	A provider package crosses into the Serving phase on a cluster — the moment a Project assigned a Cloud served by that package can actually provision against it.	Cluster identity, package source and version.
managementfleet.ClusterProviderPackageFailed	A provider package crosses into the Failed phase on a cluster.	Cluster identity, package source and version, and the condition message the package manager reported.

ProjectClusterAssigned was declared with the inventory model but reserved — without an emitter — until the region-assignment resolver (Region-assignment resolver) began firing it. The resolver is the event's emitter: it publishes ProjectClusterAssigned when it places a Project on the cluster matching its Domain region, and again when it re-points a migrated Project after the old assignment reaches Deleted. The low-level AssignProjectToCluster persist method deliberately does not emit the event inline — it is also called during internal region-migration bookkeeping, so emitting there would fire on writes that are not a fresh region-scheduling placement; the resolver owns the region-placement decision and is therefore the correct emitter.

The production composition root emits the reconcile events through a structured-slog sink — migration 0023 created only the two inventory tables, so there is no management-fleet outbox table to append to. Minting one is a schema change a dedicated story owns; until then the slog line is the operator-facing breadcrumb. The adapter rationale lives in the DECISION block on newManagementFleetEventSink in management_fleet_factory_prod.go.

Error sentinels

Every operation funnels through one of eight package-local sentinels. Callers branch on these via errors.Is — wrapping with fmt.Errorf("%w", …) is fine, identity must remain intact. The set is closed: adding a ninth without updating the closed-set drift gate trips the build (errors.go).

Sentinel	Layer	Trigger	Remediation
ErrManagementClusterNotFound	Repository / Service	GetCluster or a fleet lookup for a ClusterID with no inventory row.	Re-check the fleet inventory; register the cluster if it is genuinely absent.
ErrAssignmentNotFound	Repository / Service	GetAssignment for a Project that has not been placed.	Place the Project with AssignProjectToCluster, or accept the Project has no assignment.
ErrAssignmentImmutable	Service (resource-count gate inside the placement RunInTx)	A re-assignment to a different cluster while the Project already owns ≥ 1 resource.	Re-pointing would orphan live Composite Resources, ProviderConfigs, and Secrets. Tear the Project's resources down on the old cluster first, or keep the existing placement.
ErrClusterUnhealthy	FleetHealthChecker verify gate	A management cluster is missing a Crossplane / External Secrets Operator API group, or a substrate controller Deployment is not Available.	Install or repair the substrate via the deploy/ Helm chart; no Project may be assigned until the gate passes.
ErrNoClusterForRegion	Service (SelectClusterForRegion)	Placement is asked for a region with no matching registered cluster — or an empty region against a fleet that does not have exactly one cluster.	Register a cluster for the region, or pass an explicit region when the fleet has more than one cluster.
ErrAssignmentTerminating	Service (Unassign)	Unassign is attempted while the namespace phase has not yet reached Deleted.	Let the reconcile loop drain the namespace to Deleted, then retry Unassign.
ErrProviderInstallNotFound	Repository / ProviderInstaller	GetClusterProviderInstall for a (cluster, package source) pair the fleet has not recorded yet.	Not an error condition in normal operation: the installer treats it as the first observation of that package, which makes a first crossing into Serving event-worthy.
ErrInvalidInput	Aggregate constructors / port boundaries	A zero id, a zero or unordered timestamp, or an empty slug observed before any persistence call.	Programmer error at a boundary; surfaces in tests, not in steady-state production.

Two further sentinels exist outside the closed domain set because they name wiring bugs, not observable domain failures, and a misconfigured composition root must fail fast at boot rather than on the first operation:

• ErrServiceRepositoryRequired — NewService was handed a nil Repository (service.go).
• ErrReconcilerClientRequired / ErrReconcilerRepositoryRequired / ErrReconcilerEventsRequired — NewReconciler was handed a nil collaborator (reconcile/reconcile.go).

The verify gate

The verify gate (reconcile/verify.go) is a read-only readiness check. This milestone ships the gate only; installing the substrate stays with the deploy/ Helm chart. The gate performs two checks and returns on the first failure:

1. API groups present. Every provisioning API group is served by the cluster, resolved by mapping a representative GroupKind through the cluster RESTMapper. The gate probes one representative kind per group rather than the full CRD inventory — a partial CRD install is not a failure mode the upstream Helm charts produce:

   API group	Representative kind	Substrate
   apiextensions.crossplane.io	CompositeResourceDefinition	Crossplane composition
   pkg.crossplane.io	Provider	Crossplane packages
   external-secrets.io	ExternalSecret	External Secrets Operator

2. Controllers Available. Every substrate controller Deployment exists and reports the Available status condition True. The gate defaults to the canonical upstream install coordinates: the crossplane Deployment in the crossplane-system Namespace (one controller serves both Crossplane groups), and the external-secrets Deployment in the external-secrets Namespace.

   The controller Namespaces are overridable per environment. The upstream Helm charts install each controller into its own Namespace, but a flat single-namespace layout (such as the local dev stack, which hand-rolls the controllers side by side in the namespace the rest of the stack applies into) co-locates them. The composition root reads two optional env vars — PLEXSPHERE_MANAGEMENT_FLEET_CROSSPLANE_CONTROLLER_NAMESPACE and PLEXSPHERE_MANAGEMENT_FLEET_EXTERNAL_SECRETS_CONTROLLER_NAMESPACE — and threads them into the reconcile Config; an unset value falls back to the canonical Namespace, so a production binary that installs the upstream charts sets neither. The dev overlay points both at the Pod's own Namespace (via the downward API) so the gate finds the co-located controllers. The controller names (crossplane, external-secrets) and the API-group probes are not overridable — every install keeps the upstream object names.

On any failure the gate returns an error wrapping ErrClusterUnhealthy so callers branch with errors.Is. A nil return means the cluster is eligible to host Projects. Inside the reconcile loop an ErrClusterUnhealthy is not fatal — it folds into VerifyPassed = false and the namespace keeps converging — whereas a non-unhealthy infrastructure error (an apiserver outage, a RESTMapper fault) propagates and fails the tick.

Persistence

Migration 0023_management_fleet.sql introduces two tables in the plexsphere schema:

• plexsphere.management_clusters — one row per registered cluster. id is the application-minted UUID PRIMARY KEY; slug is held UNIQUE by management_clusters_slug_unique; region and kubeconfig_secret_ref are nullable (a cluster may be not-region-pinned, and the kubeconfig material lives behind a secret store, never in this table).
• plexsphere.project_cluster_assignments — one row per Project. project_id is the PRIMARY KEY (a Project is assigned to at most one cluster) and references plexsphere.projects(id); management_cluster_id references plexsphere.management_clusters(id). Both FKs use ON DELETE RESTRICT. namespace_phase is held to the six-value closed set by project_cluster_assignments_phase_check. The project_cluster_assignments_cluster_idx index backs the ListAssignmentsForCluster range scan.

Because neither table holds secret bytes or hash-chained forensic rows, the migration's Down block performs a real DROP in reverse-FK order — it cannot resurrect compliance-sensitive plaintext on a subsequent Up.

Operational model

The management fleet is opt-in at the composition root. The single load-bearing knob is PLEXSPHERE_DSN: when it is empty the binary boots without a management-fleet reconcile probe (the early-boot posture for deployments that have not yet plumbed Postgres). The in-cluster Kubernetes access is ambient — the ServiceAccount the Pod runs under — and needs no env var.

Env var	Effect	Default
PLEXSPHERE_DSN	The Postgres connection string. Empty disables management-fleet wiring entirely.	"" (inert).
PLEXSPHERE_MANAGEMENT_FLEET_RECONCILE_INTERVAL	The steady-state period between fleet reconcile sweeps. Parsed with time.ParseDuration; a non-positive value is rejected at boot.	30s.

The reconcile runs on two cadences, driven by management_fleet_factory_prod.go and registered through internal/platform/bootstrap/managementfleet_reconcile.go:

1. Boot sweep — synchronous, before the listener binds. RegisterManagementFleetReconcileProbe runs the sweep once; a failure here refuses startup, because a fleet that cannot be reconciled into its expected shape must not serve traffic.
2. /readyz probe + steady-state ticker — after the boot sweep the same closure is registered as a /readyz probe under the stable name management-fleet-reconcile, and a goroutine re-runs the sweep every reconcile interval. A failure on a later probe tick flips /readyz to HTTP 503 so Kubernetes (and operators) catch drift after the binary has already come up. The ticker exits cleanly on context cancellation.

The sweep lists every registered cluster, lists each cluster's assignments, and reconciles each one, returning the first error it hits. It is idempotent because Reconcile is total and idempotent.

Recovery runbook

This section is the operator-facing companion to the reference above. Each entry follows the same shape — Symptom, Diagnostic, Remediation — and is scoped to a single failure mode. The reconcile loop is idempotent, so unless an entry says otherwise the safe baseline action is to let the next sweep retry and watch /readyz and the management-fleet-reconcile probe recover.

1. Management cluster unreachable

Symptom. /readyz reports the management-fleet-reconcile probe failing (HTTP 503), or the boot sweep refused startup. The structured log carries management fleet reconcile tick failed with an error that wraps a controller-runtime client transport failure (connection refused, TLS handshake timeout, context deadline exceeded) rather than ErrClusterUnhealthy.

Diagnostic.

• Confirm the apiserver of the management cluster the binary runs inside is reachable: kubectl get --raw=/healthz from the Pod, or inspect the in-cluster apiserver Service and endpoints.
• Confirm the Pod's ServiceAccount token is mounted and not expired — a rotated or revoked token surfaces as a 401/403 from every Get.
• Distinguish this from a substrate failure (entry 2): an unreachable cluster fails on the first resource Get; an unhealthy substrate reaches the verify gate and surfaces ErrClusterUnhealthy.

Remediation. Restore apiserver reachability or re-issue the ServiceAccount credentials. No fleet data is lost — the inventory and assignment rows are durable in Postgres. Once the apiserver is reachable the next sweep reconciles every assignment and /readyz returns to HTTP 200 with no operator action on the fleet records.

2. Crossplane or External Secrets Operator missing or unhealthy

Symptom. Per-Project namespaces never leave Provisioning (or sit in Degraded). The verify gate returns an error wrapping ErrClusterUnhealthy; the wrapped message names the specific substrate — a missing API group (apiextensions.crossplane.io, pkg.crossplane.io, external-secrets.io) or a controller Deployment that is not Available.

Diagnostic.

• kubectl get crd | grep -E 'crossplane|external-secrets' confirms whether the substrate CRDs are installed.
• kubectl -n crossplane-system get deploy crossplane and kubectl -n external-secrets get deploy external-secrets confirm the two controller Deployments exist and report Available. On a flat single-namespace layout the controllers live in the apply Namespace instead; check whichever Namespaces the two PLEXSPHERE_MANAGEMENT_FLEET_*_CONTROLLER_NAMESPACE env vars point the gate at (unset means the canonical Namespaces above).
• Read the wrapped error: the gate stops on the first failure, so fix that substrate and re-check rather than assuming the rest are healthy.

Remediation. Install or repair the substrate via the deploy/ Helm chart — this context verifies the substrate, it does not install it. The verify failure is non-fatal to a converging namespace: the reconcile keeps converging the per-Project resources and the namespace crosses into Ready automatically once the gate passes. No assignment needs re-creating.

3. Namespace stuck in Terminating

Symptom. After Service.MarkTerminating, a per-Project namespace stays in the Terminating phase across many sweeps and never reaches Deleted. Service.Unassign fails with ErrAssignmentTerminating.

Diagnostic.

• kubectl get ns plexsphere-project-<uuid> — a namespace stuck Terminating at the Kubernetes level almost always has a resource with a finalizer that has not been removed. kubectl get all,resourcequota,rolebinding -n plexsphere-project-<uuid> and inspect .metadata.finalizers on anything that remains.
• The transition is sticky: once Terminating, the namespace can only drain to Deleted — it will not converge back. The phase advances to Deleted only when the reconcile observes all five per-Project resources absent.

Remediation. Clear the blocking finalizer on the offending object (typically a Crossplane managed resource awaiting external-API deletion). Once every per-Project resource is gone, the next sweep records Deleted and Service.Unassign succeeds. Do not delete the assignment row directly to "unstick" it — that orphans whatever is still live on the cluster and loses the teardown record.

4. Namespace deleted out of band

Symptom. An operator (or another controller) deleted a per-Project Namespace, Role, RoleBinding, ServiceAccount, or ResourceQuota directly on the cluster. The assignment row still shows Ready.

Diagnostic.

• The reconcile derives its action exclusively from live cluster facts every tick, never from cached desired state. On the next sweep the ObservedCluster snapshot reports the missing resource.
• A namespace that was Ready and lost a resource transitions to Degraded (not Provisioning), so Degraded in the log or the namespace_phase column is the fingerprint of an out-of-band deletion of a still-assigned Project.

Remediation. None required — this is the case the reconcile loop exists to handle. The next sweep emits Converge, recreates the missing resources idempotently, and the namespace returns to Ready, emitting a fresh ProjectNamespaceReady event. If the namespace does not self-heal, fall through to entry 2 (the verify gate may be failing) or entry 1 (the cluster may be unreachable).

5. Assignment-immutability conflict

Symptom. Service.AssignProjectToCluster fails with ErrAssignmentImmutable. The wrapped message names the Project, the resource count it owns, and the cluster it is currently pinned to.

Diagnostic.

• The immutability gate is the Project's resource count, read inside the same transaction as the placement write. A re-assignment to a different cluster is rejected when the count is > 0; a re-assignment to the same cluster is idempotent and returns the existing assignment unchanged; a re-assignment to a different cluster with a count of 0 is permitted (it is a misplacement correction before any resource exists).
• errors.Is(err, managementfleet.ErrAssignmentImmutable) confirms the failure class; Service.LookupAssignment shows the current pinning.

Remediation. This sentinel is a fail-closed guard, not a bug — re-pointing a Project with live resources would orphan its Composite Resources, ProviderConfigs, and materialised Secrets in the old namespace. To move a Project to a different cluster: drive its namespace through MarkTerminating → reconcile-to-Deleted → Unassign on the old cluster (which tears the resources down), then AssignProjectToCluster to the new one. If the rejection was unexpected, the resource count is the source of truth — verify what the Project actually owns before forcing anything.

6. Provider package stuck Failed or Conflict

Symptom. An approved Cloud Assignment reports a provider_installs entry whose phase is Failed or Conflict, or GET /v1/management-clusters/{id}/provider-packages shows one. A Project cannot provision against the Cloud, and nothing in the namespace reconcile explains why — the namespace is Ready.

Diagnostic.

• Read the entry's source to see which declared package is stuck, then its message: the message carries the package manager's own condition text for Failed, and names both sides for Conflict.
• A derived family package has no entry on either surface — no Cloud declares it. Query plexsphere.management_cluster_provider_installs for the cluster to see its record, as under What the installer applies.
• For Failed, confirm against the cluster: kubectl get provider.pkg.crossplane.io <name> -o yaml and read the Healthy condition. A bad reference, an unreachable registry, or an unsatisfiable dependency all surface here. The object name is the package source's final path segment for a declared package, and the dependency-derived name for a derived family entry: upbound-provider-family-aws for the dev stack's family.
• For Conflict on competing versions, the message names each version, and on a family it also names the member sources it was derived from. Find the Clouds: plexctl cloud list --all --output json | jq '.items[] | {slug, provider_packages}'.
• For Conflict on a shared object name, the message names every source claiming it. They differ only outside the part the object name keeps — typically a mirror registry beside the upstream one.
• For Conflict on a foreign object, the message names the source the existing Provider serves. Something outside the platform owns that object name.
• For Conflict on a field-ownership refusal, the message carries the apiserver's own text naming the competing field manager. On a family object that manager is usually Crossplane's package manager, which resolved the member's dependency to a different version than the member's tag.

Remediation. For Failed, fix what the message names — usually the recorded package on the Cloud: PATCH /v1/clouds/{id} with a corrected provider_packages, or plexctl cloud update. Both replace the whole declared set, so restate every package the Cloud should keep, not only the one being corrected. The next sweep re-applies and the phase follows the cluster. For competing versions, patch the Clouds into agreement; the sweep converges once they do. For a shared object name, drop one of the sources or move both Clouds onto the same registry — the object cannot serve two. For a foreign object, decide who should own the name: rename the Cloud's package source, or remove the foreign Provider by hand. For a field-ownership refusal, decide which version should run: leave the other manager's in place and align the member's tag on the Cloud, or edit the object by hand.

The platform will not resolve any of these for you, by design — every automatic resolution would silently change a controller version under a Project that asked for a different one. Note also that the fleet holds no delete on providers, so a package removed from every Cloud leaves its controller running until an operator removes it.

7. Rebuilding a cluster from the durable assignment records

Symptom. A management cluster was lost or rebuilt — every per-Project namespace and its RBAC and quota objects are gone — but the plexsphere.management_clusters and plexsphere.project_cluster_assignments rows in Postgres are intact.

Diagnostic.

• The Postgres rows are the durable record of the fleet; the cluster-side objects are derived, reconcilable state. After a cluster rebuild the inventory still knows every cluster and every Project assignment, including each namespace name and phase.
• Confirm the rebuilt cluster carries the substrate before relying on recovery: a rebuilt cluster with no Crossplane / External Secrets Operator install fails the verify gate (entry 2).

Remediation. Install the substrate on the rebuilt cluster, then let the reconcile sweep do the work. Each sweep lists every assignment for the cluster and reconciles it: every per-Project Namespace, Role, RoleBinding, ServiceAccount, and ResourceQuota is recreated idempotently from the derived namespace name, and each namespace crosses back into Ready. No assignment rows need re-creating and no Project needs re-placing — the inventory survived, so the fleet rebuilds itself from it. If a Project's old namespace had been mid-teardown, its row's namespace_phase still reflects that and the teardown arm resumes correctly.