plexsphere

Appearance

Per-node reachability — heartbeat and liveness tracking ​

This document is the authoritative bounded-context reference for the per-Node reachability sub-context that ships under internal/mesh/reachability and exposes through POST /v1/nodes/{id}/heartbeat and GET /v1/nodes/{id}/reachability. It covers the ubiquitous language, the wire envelopes, the four-state liveness machine, the per-Domain threshold policy, the periodic evaluator, the clock-skew admission rule, the audit contract, the OpenAPI cross-reference, and the threat model.

The reachability surface is the plexd → plexsphere liveness channel: a per-Node, NSK-authenticated heartbeat ingestion endpoint plus a synchronous read of the server's verdict on each Node's liveness state. Anything outside that surface — the underlying Signed SSE Event Bus that fans out tenancy.NodeReachabilityChanged events, the reconciliation pull that exposes a cold-start view, the per-Domain Tenancy aggregate that owns ReachabilityPolicy, the audit hash chain, the Dashboard's ReachabilityBadge — is a collaborator the reachability sub-context projects from or emits to, not a concern of this document. The two surfaces this document pins are the in-process liveness state machine and the wire-shape contracts that surround it.

Status — partial delivery ​

The writer-side closure for the reachability event bus is load-bearing in the production binary:

  • Outbox-backed tenancy.NodeReachabilityChanged emission — the evaluator's ReachabilityWriter routes every transition through NodeRepo.TransitionAndEmit, folding the UpdateReachabilityState write and the tenancy/events.NodeReachabilityChanged outbox append into one Postgres transaction. NewNodeReachabilityChanged in internal/identity/tenancy/events is the canonical event shape.
  • Chain-bound AuditSink — each transition lands on the per-Domain audit chain as internal/audit.Entry with Relation = "node_reachability.transition", Object = "domain:<hex16>", RelationPath = [from, to, reason].

One downstream surface remains deferred: the SSE relay drain for node_state_updated that surfaces the outbox row on GET /v1/nodes/{id}/events, tracked in ../../architecture/mesh-event-bus-roadmap.md. The outbox row is durable on every transition today; the wire envelope reaches a subscriber once the relay drain lands.

Load-bearing today (each detailed in its own section below): the pure state machine (state.go, Apply), the heartbeat handler with its six-step gate ordering and four refusal codes, the production NSK plaintext resolver (heartbeat_nsk_resolver.go), the reachability read handler, the per-Domain ReachabilityPolicy value object and its invariant gates, the periodic evaluator, the 0010_node_reachability.sql migration (six Node columns, three Domain columns, the nodes_stale_idx partial index), the MaxHeartbeatSkew golden constant, the OpenAPI surface, and the Dashboard ReachabilityBadge. The exact file references are in Cross-references.

Downstream consumers (platform audit log, node deregistration, relay-fallback heuristics, rolled-up state reports) MUST treat the deferred SSE drain as scaffolding: every wire shape is stable from day one, and the tenancy.NodeReachabilityChanged outbox row is the single dependency they consume the transition stream from rather than the read-side projection.

Cross-references ​

Ubiquitous language ​

The terms below travel together across the Go code, the OpenAPI contract, the audit log, the structured-log attributes, the Prometheus metric label values, and operator-facing tooling. Names are preserved verbatim in error messages and audit-row vocabulary so a reader chasing a string from a log line finds it in the source without translation.

TermDefinition
HeartbeatA signed liveness ping plexd POSTs to POST /v1/nodes/{id}/heartbeat at the per-Domain HeartbeatInterval cadence. The wire envelope is HeartbeatRequest{client_now, binary_checksum, binary_version, nat_summary}. The ingestion stamps the server's last_heartbeat_at on the addressed Node row.
ReachabilityThe server's verdict on a Node's liveness, derived purely from server-observed inputs. Persisted as a (state, last_heartbeat_at, changed_at) triple on plexsphere.nodes and projected onto the wire by GET /v1/nodes/{id}/reachability.
Never reportedA Reachability.state value (StateNeverReported = "never_reported"). No heartbeat has ever been admitted for the Node. It is the schema default a Node row is born with at enrolment (0073_node_never_reported.sql), and the single way out is the Node's first admitted heartbeat. The evaluator writes it on one path only: correcting a row whose persisted verdict disagrees with its own last_heartbeat_at.
HealthyA Reachability.state value (StateHealthy = "healthy"). The Node's last admitted heartbeat is fresher than the per-Domain StaleAfter threshold. A Node reaches it from NeverReported on its first admitted heartbeat, and from Stale or Unreachable when its heartbeats resume.
StaleA Reachability.state value (StateStale = "stale"). The Node's last admitted heartbeat is older than StaleAfter but younger than UnreachableAfter. The transition Healthy → Stale is driven by the periodic evaluator.
UnreachableA Reachability.state value (StateUnreachable = "unreachable"). The Node's last admitted heartbeat is older than UnreachableAfter. The transition Stale → Unreachable is driven by the periodic evaluator. The verdict says a Node's heartbeats stopped, so it never applies to a Node that has yet to send its first one: that Node reads NeverReported.
ReachabilityPolicyThe per-Domain value object ({HeartbeatInterval, StaleAfter, UnreachableAfter}) on the Tenancy aggregate that drives the evaluator's thresholds. Persisted as the three reach_* columns on plexsphere.domains.
HeartbeatIntervalThe cadence agents must honour. Floor ≥ 10s, ceiling ≤ 1h. Default 30s. Pinned in internal/identity/tenancy/domain.go's DefaultReachabilityPolicy.
StaleAfterThe grace window before the evaluator declares a Node Stale. Floor ≥ 3 × HeartbeatInterval (so a single dropped beat does not trip the verdict), ceiling ≤ 1h. Default 90s.
UnreachableAfterThe grace window before the evaluator declares a Node Unreachable. Floor ≥ 2 × StaleAfter (avoids flapping near the Stale boundary), ceiling ≤ 1h. Default 300s.
BinaryChecksumThe raw SHA-256 digest (exactly 32 bytes) of the agent binary, base64-encoded on the wire. The aggregate's RecordHeartbeat method enforces the 32-byte length invariant. Empty / wrong-length checksums are refused with 400 binary_checksum_empty.
BinaryVersionThe agent's semver string. Empty values (after trimming) are refused with 400 binary_version_empty.
NATSummaryThe optional advisory NAT/firewall summary the agent observed at envelope creation. The handler does not parse it; the aggregate persists it verbatim as a JSON byte slice for downstream observability.
NSK (Node Session Key)The Node's per-session bearer credential, presented as Authorization: Bearer nsk_<env>_<...>. The NSKAuthenticator middleware in internal/identity/authn/middleware/nsk.go resolves it to a tenancy.Node and refuses path-id ≠ resolved-Node-id with 403 node_id_mismatch.
MaxHeartbeatSkewThe inclusive upper bound on |client_now − server_now| accepted by the heartbeat handler. Pinned at 60 * time.Second in internal/platform/clock/skew.go against a golden literal; widening is a security regression, not a tuning knob.
Reachability evaluatorThe per-Domain background ticker in internal/mesh/reachability/evaluator.go that paginates two partial-index-backed candidate sets per tick — the stale sweep over nodes_stale_idx and the recovery sweep over nodes_recovery_idx — runs Apply per row, and writes any transition through UpdateReachabilityState's single-statement UPDATE … RETURNING with idempotency guarded by the reachability_state <> $2 predicate.
AuditRelationNodeHeartbeatThe canonical Relation string the heartbeat handler stamps on every audit row: node_heartbeat.record. Stable so audit consumers can filter by relation without parsing free-form text. Pinned in internal/transport/http/v1/handlers/heartbeat_deps.go.
AuditRelationNodeReachabilityReadThe canonical Relation string the read-side handler stamps on every audit row: node_reachability.read. Pinned in heartbeat_deps.go alongside the heartbeat relation.
node-agent relationThe ReBAC relation enforced on the operator arm of GET /v1/nodes/{id}/reachability. Shared with GET /v1/nodes/{id}/state and GET /v1/nodes/{id}/events so a single operator principal cone authorises the cold-start view, the push channel, and the reachability projection. A plexd Node presenting its own NSK takes the alternative NSK arm instead — an equality gate (authenticated Node = addressed {id}) with no ReBAC round-trip; a cross-Node NSK is refused with 403 node_id_mismatch.

Wire shape — POST /v1/nodes/{id}/heartbeat ​

The request envelope is the HeartbeatRequest value type defined in internal/transport/http/v1/handlers/heartbeat.go, mirrored on the OpenAPI spec at api/openapi/plexsphere-v1.yaml under components.schemas.HeartbeatRequest.

FieldTypeMeaning
client_nowRFC 3339 timestampThe agent's local wall-clock at envelope creation. Compared against the server's now via clock.WithinSkew — |delta| > MaxHeartbeatSkew rejects with 400 clock_skew. NEVER drives state transitions; the server uses its own now for verdict logic.
binary_checksumbase64-encoded []byte (32 bytes raw SHA-256)The aggregate's RecordHeartbeat invariant enforces the 32-byte length. Empty / wrong-length values reject with 400 binary_checksum_empty.
binary_versionstringThe agent's semver string. Empty after trimming rejects with 400 binary_version_empty.
nat_summaryjson.RawMessage (optional)The agent's NAT/firewall advisory blob. Forwarded verbatim to the aggregate for persistence; the handler does not parse it.

The success response envelope is the HeartbeatResponse value type in the same file.

FieldTypeMeaning
accepted_atRFC 3339 timestampThe server-side admission instant the handler stamped after the clock-skew gate passed and before the aggregate write.
reconcileboolHint to the agent that it should issue a reconciliation pull at GET /v1/nodes/{id}/state. ALWAYS false today — becomes load-bearing once the snapshot-divergence detection lands as a follow-up to the reconciliation-pull surface.
rotate_keysboolHint to the agent that it should rotate its NSK. ALWAYS false today — becomes load-bearing once the NSK rotation workflow lands.

The four refusal codes the handler emits (Problem-Details code field):

CodeHTTPTriggerSource file
clock_skew400|client_now − server_now| > MaxHeartbeatSkew (60s)heartbeat.go step 4
node_id_mismatch403URL path id ≠ resolved NSK Node id (defense-in-depth; the NSK middleware also runs this gate)heartbeat.go step 2
nsk_revoked401NSK middleware resolves to a revoked NSK row (resolver responsibility)internal/identity/authn/middleware/nsk.go
binary_checksum_empty400len(binary_checksum) ≠ 32 (also triggered by an empty / nil slice)heartbeat.go step 5 (ErrHeartbeatChecksumInvalid translation)

A binary_version_empty (400) variant fires for empty / whitespace- only binary_version values; it shares the same well-formed-envelope requirement row as binary_checksum_empty but is a distinct Problem code.

Wire shape — GET /v1/nodes/{id}/reachability ​

The success response envelope is the ReachabilityResponse value type defined in internal/transport/http/v1/handlers/reachability.go, mirrored on the OpenAPI spec under components.schemas.Reachability.

FieldTypeMeaning
statestring ("never_reported" / "healthy" / "stale" / "unreachable")The server's current verdict. A Node enrols into "never_reported" and leaves it for a heartbeat-derived verdict once its first heartbeat is admitted.
last_heartbeat_atRFC 3339 timestamp (UTC), optionalThe most recent successful ingestion. The key is ABSENT until the first heartbeat is admitted, so an agent-less Node answers {"state":"never_reported"} with no last_heartbeat_at at all.
changed_atRFC 3339 timestamp (UTC)The most recent state transition instant. Always present; on a Node that has never sent a heartbeat it carries the enrolment instant, the moment the row was materialised.

An empty state is not one of the four verdicts. The column defaults to never_reported and 0073_node_never_reported.sql corrected every row that predates it, so an empty label can only come from a row written outside the schema. The read path passes such a value through verbatim rather than substituting a plausible verdict that would hide the breakage from the operator.

The handler serves two credential arms. On the operator arm it emits 401 unauthorized (no Principal), 403 insufficient_relation (no node-agent relation on node:<id>), 404 node_not_found (the addressed Node id does not resolve to a row), and 501 reachability_not_provisioned (one or more of ReachabilityReader, RelationChecker, NodeRepo is nil at composition time — the reachability_dispatch.go gate fires). On the NSK arm a plexd Node authenticates with its own NSK: the authenticated Node must equal the addressed {id}, so a cross-Node NSK is refused with 403 node_id_mismatch, a revoked NSK with 401 nsk_revoked, and a missing or malformed NSK with 401 nsk_invalid; the NSK arm runs no ReBAC check.

State machine ​

The Apply function in internal/mesh/reachability/state.go is the SINGLE exported transition rule. Every consumer (the heartbeat handler's downstream observability, the evaluator's per-row loop, the reconciliation pull's reachability block) funnels through it rather than branching on elapsed time itself, so the matrix has exactly one home.

mermaid
stateDiagram-v2
    [*] --> NeverReported : schema default (`reachability_state = 'never_reported'`)
    NeverReported --> Healthy : evaluator: first heartbeat admitted, elapsed < StaleAfter
    NeverReported --> Stale : evaluator: first heartbeat admitted, elapsed >= StaleAfter && elapsed < UnreachableAfter
    NeverReported --> Unreachable : evaluator: first heartbeat admitted, elapsed >= UnreachableAfter
    Healthy --> Stale : evaluator: elapsed >= StaleAfter && elapsed < UnreachableAfter
    Stale --> Unreachable : evaluator: elapsed >= UnreachableAfter
    Healthy --> Unreachable : evaluator: elapsed >= UnreachableAfter (skipped Stale)
    Stale --> Healthy : admitted heartbeat: elapsed < StaleAfter
    Unreachable --> Healthy : admitted heartbeat: elapsed < StaleAfter
    Unreachable --> Stale : admitted heartbeat: elapsed >= StaleAfter && elapsed < UnreachableAfter
    Healthy --> NeverReported : evaluator: drift correction, no heartbeat on record
    Stale --> NeverReported : evaluator: drift correction, no heartbeat on record
    Unreachable --> NeverReported : evaluator: drift correction, no heartbeat on record

The transition rules Apply encodes (in transitionFor):

Elapsed (now − lastHeartbeatAt)Result
lastHeartbeatAt is zero (no admitted heartbeat)StateNeverReported
elapsed >= UnreachableAfterStateUnreachable
StaleAfter <= elapsed < UnreachableAfterStateStale
elapsed < StaleAfterStateHealthy

Apply returns (next, changed) where changed = (next != prev) on every path, so a no-op transition (e.g. Healthy × elapsed < StaleAfter, Stale × elapsed between thresholds) returns (prev, false) and the caller short-circuits the eventing pipeline. The zero-heartbeat row follows the same rule: a Node already carrying NeverReported is inert — no state write, no event, no audit row, tick after tick — while a row carrying any OTHER verdict with no heartbeat on record is a drifted row and is corrected onto NeverReported once. The guard also sits ahead of the elapsed comparison, where now.Sub(zero) would otherwise read as an enormous elapsed and stamp the Node Unreachable. The fuzz harness at internal/mesh/reachability/state_fuzz_test.go pins the contract that an invalid prev MUST NOT panic — the function may return any State for invalid input, but never crashes the caller.

The four transition origins:

  • First-heartbeat exits (NeverReported → Healthy, NeverReported → Stale, NeverReported → Unreachable) are the only way a Node leaves NeverReported. The first admitted heartbeat stamps last_heartbeat_at past the enrolment instant, which makes the row a recovery-sweep candidate, and the next sweep runs the elapsed matrix against that one heartbeat. A fresh heartbeat lands the Node on Healthy; a heartbeat already older than a threshold lands it on Stale or Unreachable, because such a Node reported once and then stopped.

  • Drift corrections (Healthy → NeverReported, Stale → NeverReported, Unreachable → NeverReported) are the only way a Node moves back INTO NeverReported at run time. They fire when a row's persisted verdict disagrees with its own last_heartbeat_at — healthy with nothing on record is what an operator's manual UPDATE during incident response, or a restore from a pre-0073 backup, leaves behind. Such a row would otherwise advertise reachability forever and be picked as a relay bridge by chooseBridge, so the sweep writes the verdict the row's own columns support, emits one event carrying evaluator: no heartbeat on record (corrected to never reported), and is inert from the next tick on. A row born NeverReported at enrolment, or corrected into it by the 0073_node_never_reported.sql backfill, never takes this edge.

    This is the one transition whose event carries no heartbeat, because the row it records has none. The last_heartbeat_at field of the tenancy.NodeReachabilityChanged payload is the zero timestamp (0001-01-01T00:00:00Z) on this edge and only on this edge, so a consumer that reads the field must treat the zero value as "nothing on record" rather than as a date. The write side enforces the pairing in both directions: TransitionAndEmit refuses to move a row with a NULL last_heartbeat_at to any state other than NeverReported, and refuses to move a row that HAS one INTO NeverReported. So a zero-timestamp payload can never claim a state that implies an observation, and — because a Node's first heartbeat can commit between the sweep's read of the row and the write transaction's re-read of it — a NeverReported payload can never carry a real timestamp that contradicts its own reason.

  • Recovery edges (Stale → Healthy, Unreachable → Healthy, Unreachable → Stale) are driven by an admitted heartbeat resetting the elapsed clock back below the relevant threshold. The heartbeat handler stamps last_heartbeat_at and the next evaluator sweep observes elapsed < StaleAfter (or between thresholds) and applies the transition. The handler itself does NOT write reachability_state — see the read-path-rejector DECISION below .

  • Advance edges (Healthy → Stale, Stale → Unreachable, Healthy → Unreachable) are driven solely by the periodic evaluator. The handler is a thin admit-and-write that updates last_heartbeat_at and the NAT/binary advisories; the state- machine vocabulary lives entirely in the evaluator.

TTL / threshold defaults & per-Domain knobs ​

The per-Domain ReachabilityPolicy value object on the Tenancy aggregate (internal/identity/tenancy/domain.go) carries three thresholds. Defaults and invariant gates:

ThresholdDefaultFloorCeilingInvariant
HeartbeatInterval30s≥ 10s≤ 1h(none beyond floor / ceiling)
StaleAfter90s≥ 3 × HeartbeatInterval≤ 1hA single dropped beat must not trip the verdict
UnreachableAfter300s≥ 2 × StaleAfter≤ 1hAvoids flapping near the Stale boundary

The defaults are pinned by DefaultReachabilityPolicy in domain.go:

go
var DefaultReachabilityPolicy = ReachabilityPolicy{
    HeartbeatInterval: 30 * time.Second,
    StaleAfter:        90 * time.Second,
    UnreachableAfter:  300 * time.Second,
}

A fully-zero ReachabilityPolicy{} on NewDomain / Hydrate is treated as the DEFAULT signal and replaced with DefaultReachabilityPolicy at build time. A partial fill (e.g. only HeartbeatInterval set) is REJECTED with errInvariant rather than silently extended — a caller who meant to set a custom StaleAfter but spelled the field wrong sees the error in their face. Each gate returns errInvariant("REQ-006", "Domain", <field>, <bound>) so a grep on the requirement id surfaces every refusal site.

The persistence layer mirrors the value object on plexsphere.domains as three interval columns (reach_heartbeat_interval, reach_stale_after, reach_unreachable_after), defaulted to the same (30s, 90s, 300s) triple at the schema level (0010_node_reachability.sql). The Tenancy aggregate's Hydrate boundary re-validates the multipliers on load so a hand-edited Domain row that violates the invariants fails fast.

Evaluator scheduling ​

The reachability evaluator (internal/mesh/reachability/evaluator.go) is the per-Domain background ticker that writes every state transition, advance and recovery alike. The composition root in cmd/plexsphere/evaluator_factory_prod.go wires it on the same PLEXSPHERE_DSN env-var as the other production-wired surfaces; an empty DSN leaves the evaluator unwired (no goroutine, no sweep, no transitions).

Tick loop ​

Run(ctx, tick) drives Tick(ctx) on a fixed-cadence ticker. The production tick is read from PLEXSPHERE_REACH_EVAL_TICK with a 5 * time.Second default (DefaultEvaluatorTick). Five seconds is the scaffold balance between responsiveness (a Stale → Unreachable transition is observable within one full cycle) and Postgres pressure (a 1024-row LoadStaleNodes page is a single index scan, so 5s ticks land < 1k SELECT/min on a quiet Domain). Cancelling the context terminates the sweep.

Per-tick procedure ​

Each Tick(ctx) call runs two paging loops: the stale sweep over the Nodes whose heartbeat has gone quiet, then the recovery sweep over the Nodes parked at Unreachable or NeverReported whose heartbeat has arrived. Both loops feed their rows through the same per-row evaluation, so a transition looks the same whichever loop produced it.

  1. Reads now = clock.Now().UTC() via the injected Clock port (a virtual clock in tests so a Tick() call advances "now" without sleeping the test goroutine). Both sweeps judge their rows against this one now.
  2. Calls Lister.ListStale(ctx, now, limit, cursor) — paginates the nodes_stale_idx partial index by (reachability_changed_at, id). The cursor's zero value bootstraps the first page; a short page (len < limit) drains the candidate set for this tick.
  3. For each returned tenancy.Node row:
  • Resolves the Node's Domain ReachabilityPolicy via Policies.LookupPolicy(ctx, node.DomainID()). A non-nil error short-circuits the loop on this Node and surfaces as a WARN log; the evaluator CONTINUES to the next Node — one misconfigured Domain must not stall every other Domain's sweep.
  • Runs Apply(prev, now, node.Reachability.LastHeartbeatAt, policy.StaleAfter, policy.UnreachableAfter). If changed == false, skips the row — the eventing pipeline short-circuits.
  • Calls Writer.Update(ctx, nodeID, string(next), now) — a single UPDATE … RETURNING with a reachability_state <> $2 guard for idempotency. Returns (transitioned bool, err). A (false, nil) arm means another writer (or an earlier sweep) already landed this transition — the evaluator suppresses event + audit so only the row that actually transitioned is the single emitter.
  • On a successful transition, calls Events.Emit(ctx, ev) to emit the tenancy.NodeReachabilityChanged event (slog-only today — see status section), then Audit.Record(ctx, entry) to emit the audit row.
  1. Advances the cursor to the last row of the page; loops if the page was full, leaves the stale sweep otherwise.
  2. Calls Recovery.ListRecoveryCandidates(ctx, limit, cursor) — paginates the nodes_recovery_idx partial index by (reachability_changed_at, id), oldest stamp first. There is no asOf argument: both SQL filters read columns of the same row (last_heartbeat_at > reachability_changed_at for Unreachable, last_heartbeat_at IS NOT NULL for NeverReported), so there is no wall-clock cutoff to supply.
  3. For each returned row: the step-3 pipeline verbatim, from LookupPolicy through Apply, Writer.Update, Events.Emit, Audit.Record. What the heartbeat is worth follows from its own age against the same thresholds: younger than StaleAfter moves the Node to Healthy; between the two thresholds moves it to Stale, which also moves the row out of nodes_recovery_idx and into nodes_stale_idx; older than UnreachableAfter moves a NeverReported Node to Unreachable and leaves an already Unreachable one where it is, so the sweep writes nothing for that second row. The NeverReported rows the sweep returns are the ones whose first heartbeat has landed, so every one of them transitions.
  4. Advances the recovery cursor to the last row of the page; loops if the page was full, returns otherwise.

Failure handling across the two sweeps: both always run, and Tick returns their errors joined. They are different statements against different partial indexes with different predicates, so a ListStale failure says nothing about whether ListRecoveryCandidates is healthy — aborting the tick on the first error would silently disable Node recovery fleet-wide for as long as the stale read is unhappy. Whichever sweep succeeds persists its transitions; whichever fails contributes its error to the joined result the Run loop logs as a WARN. Per-Node failures on either sweep are logged and the loop continues to the next row.

Both sweeps share one per-Tick ReachabilityPolicy memo, so a Domain costs one LookupPolicy round-trip per tick however many of its Nodes either sweep returns. The memo is discarded when Tick returns — an operator tightening StaleAfter sees the change on the next tick, with no cache window to wait out.

Failed lookups are memoised too, so a Domain whose policy read is down costs one attempt per tick rather than one per row: a database failover cannot turn a single dead read into a per-row burst of doomed transactions across both sweeps. The WARN line follows the round-trip, not the row — the row that performed the failing lookup logs it, every row the memo answers is silent — so the same failover cannot turn one dead read into a per-row burst of log lines either. The line names the Domain rather than a Node: it reports one dead Domain read, and that Domain's rows are skipped for the remainder of the tick and retried on the next one.

Partial-index paging ​

Each sweep rides its own partial b-tree, and the two predicates are disjoint: reachability_state is either in ('healthy','stale') (the states the elapsed matrix can still advance) or in ('unreachable','never_reported') (the two states a Node sits parked in until a heartbeat arrives). No row is a candidate for both sweeps on the same tick.

The stale sweep rides the nodes_stale_idx partial b-tree on (reachability_changed_at) WHERE reachability_state IN ('healthy','stale'), added by 0010_node_reachability.sql, and 0073_node_never_reported.sql deliberately leaves it alone. Keeping the two parked states out holds the working set small: an Unreachable Node has no further advance transition, and a NeverReported Node has no elapsed-heartbeat gap to measure at all. The recovery direction is covered by a second index rather than by widening this one.

The recovery sweep rides nodes_recovery_idx, a partial b-tree on (reachability_changed_at, id) WHERE reachability_state IN ('unreachable', 'never_reported'), added by 0072_node_recovery_index.sql and rebuilt on the widened predicate by 0073_node_never_reported.sql. The condition that actually selects a candidate is not in the index. It is a per-row filter in the query, and it is stated once per admitted state: an Unreachable row qualifies when last_heartbeat_at > reachability_changed_at ("a heartbeat was accepted after the flip"), a NeverReported row when last_heartbeat_at IS NOT NULL ("the Node has reported at all"). The asymmetry is deliberate. On the NeverReported arm the two columns come from two different host clocks — reachability_changed_at is the DEFAULT now() the Postgres host stamped at enrolment, while last_heartbeat_at is stamped by the API host — so comparing them would hide a correctly-beating Node from the sweep for the whole clock-skew window. The query's DECISION block in 90_heartbeat.sql carries the reasoning; on the Unreachable arm both sides are application-clock values written by the evaluator, so the comparison there is clock-consistent.

A Node with no admitted heartbeat at all carries a NULL last_heartbeat_at and fails both arms, so an agent-less Node is never a candidate. It is not free, though: it stays in the index permanently. The row becomes a candidate the moment its FIRST heartbeat is admitted; the next tick then runs the elapsed matrix on it and writes the exit to Healthy, Stale, or Unreachable.

The permanent residency is the read-side cost an operator sizes against, and it is the one place where the two admitted states differ. An Unreachable row leaves the index when its heartbeats resume; a NeverReported row whose agent is never started leaves it never — nothing rewrites reachability_state for it. Those rows also carry the oldest reachability_changed_at values (their enrolment stamps) and the recovery sweep restarts from the head of the index on every tick, so Postgres walks them before reaching any row that qualifies. Budget one index-scan entry plus one heap fetch per enrolled-but-never-started Node per tick, growing monotonically with bulk enrolment: 10 000 such Nodes on the default 5 s cadence is 10 000 entries and 10 000 heap fetches per sweep, returning nothing. The migration's DECISION block records the two rejected ways out — naming last_heartbeat_at in the index predicate, and moving the first-heartbeat transition onto the heartbeat write path.

Keeping last_heartbeat_at out of the index is the load-bearing decision here, and it is a write-path one. Before this migration no index on plexsphere.nodes referenced any column RecordHeartbeat writes, so every heartbeat UPDATE was a HOT update: a new heap tuple in the same page, zero index-tuple writes. Postgres derives its HOT-blocking attribute set from the key and predicate columns of every non-summarizing index on the table, so naming last_heartbeat_at anywhere in this index would make it hot-blocking table-wide — the partial predicate does not scope that to the parked rows. Every heartbeat in the fleet would then write a tuple into all five indexes, bloating the four that are otherwise write-static. reachability_state and reachability_changed_at are already hot-blocking through nodes_stale_idx, so the shipped shape adds no write cost at all. The read-side price is that the index spans every parked Node rather than only the ones whose heartbeat has landed; the migration's DECISION block carries the comparison. The id column breaks ties so the keyset cursor is total, and both cursor columns are immutable while a row is in the index — writing reachability_changed_at means writing reachability_state, which removes the row — so a concurrent transition can never move a row past the sweep's anchor.

An Unreachable Node whose post-flip heartbeat is itself already older than its Domain's UnreachableAfter passes the filter and still does not recover: Apply returns Unreachable for it, unchanged, so the sweep writes nothing and the row is a candidate again on the next tick, until its next heartbeat arrives. A NeverReported Node in the same position does transition, to Unreachable, because that is a different verdict from the one it holds. The migration's DECISION block records the rejected alternative: re-stamping reachability_changed_at past last_heartbeat_at so the row stops matching. changed_at carries the instant of the most recent state transition on the API contract, and buying a scan-width reduction with a fabricated transition time is the wrong trade. Re-listing such a row costs an index entry and a heap fetch — the per-Tick policy memo keeps it from costing a LookupPolicy round-trip as well.

One race is accepted rather than closed. A heartbeat that lands between the evaluator's read of a Node and its flip to Unreachable leaves the row with last_heartbeat_at < reachability_changed_at, so the filter rejects it and the row is not a recovery candidate. The agent's next heartbeat, one HeartbeatInterval later (30s on the default policy), moves last_heartbeat_at past changed_at and re-arms it. An agent that heartbeats exactly once inside that window and never again stays Unreachable, which is the correct verdict for it.

The defaultStalePageLimit constant pins the per-tick fan-out at 1024 rows per page, on both sweeps. The number is large enough to drain a small Domain in one tick and small enough to keep the worst-case tx duration bounded under heavy load. A future per-Domain knob slots into the EvaluatorDeps.Limit field without touching the evaluator.

Read-path-rejector posture ​

The DECISION block at the top of evaluator.go records why the evaluator is the SINGLE WRITER of reachability_state — the heartbeat handler does NOT flip the row to Healthy on every ingestion. The trade-off:

  • Alternative considered: drive transitions from the heartbeat handler itself. Every successful ingestion would flip the row's state to Healthy in the same transaction.
  • Rejected because (a) the handler would need to know the Domain's StaleAfter and UnreachableAfter to decide whether the ingestion implies a transition; (b) the state-machine vocabulary would leak into a hot HTTP path that is supposed to remain a thin admit-and-write; (c) two writers on the same row would force pessimistic locking on every heartbeat.
  • Resolution: the evaluator owns the read path. It consults the Node row, runs Apply, and writes the transition (with idempotency guarded by the SQL-side reachability_state <> $2 predicate in UpdateReachabilityState). The handler stamps last_heartbeat_at and the evaluator picks up the next sweep — clean separation of concerns, one writer per row.

Clock-skew rules ​

The heartbeat handler admits a heartbeat only when |server_now − client_now| ≤ MaxHeartbeatSkew (60s). The constant is pinned in internal/platform/clock/skew.go against a golden literal (the test file pins the value at 60 * time.Second so an accidental adjustment surfaces in code review). Widening the bound is a security regression, not a tuning knob — see the file-level DECISION block.

The WithinSkew(t1, t2) helper is symmetric (callers may pass server and client time in either order) and inclusive at the boundary (delta == MaxHeartbeatSkew admits). The implementation branches on time.Time.Before to keep the subtraction in the non-negative half of the duration range — a naive t1.Sub(t2) followed by a sign-flip would saturate at the int64 duration range and let a zero time.Time vs a real timestamp silently pass the gate.

The DECISION block in internal/mesh/reachability/state.go records why client-side timestamps NEVER drive a reachability transition. The only inputs Apply consumes are the server's now and the lastHeartbeatAt value the server itself stamped at admit time. A forged or drifted client clock could otherwise:

  • Mask a genuinely absent Node by claiming to have heartbeat in the future.
  • Trigger a spurious Unreachable verdict by claiming to have heartbeat in the deep past.
  • Trigger a spurious recovery by claiming to have heartbeat after a long quiet period.

All three failure modes would erode the state machine's trustworthiness as the source of operator alerts. The MaxHeartbeatSkew admission gate runs at the handler boundary ; once that gate has passed, the server has no remaining reason to re-introduce client-clock dependence into the verdict logic.

Audit contract ​

Every decision branch in the heartbeat surface emits exactly one audit.Entry per layer. The heartbeat surface has TWO layers — the NSK middleware (admission) and the handler (ingestion) — so a granted heartbeat produces TWO rows from the operator's perspective: one with relation = node_heartbeat.authenticate, outcome = granted from the middleware (NSKAuditSink) and one with relation = node_heartbeat.record, outcome = granted from the handler (handlers.AuditSink). Audit consumers therefore SHOULD NOT expect a 1:1 mapping between heartbeat requests and audit rows; they SHOULD filter by relation to scope to the layer they care about (admission vs ingestion). A nil AuditSink is tolerated — the slog lines stand in for observability — but the security gates still execute.

The relation strings live in internal/transport/http/v1/handlers/heartbeat_deps.go:

ConstantValueLayer / origin
AuditRelationNodeHeartbeatAuthenticatenode_heartbeat.authenticateNSK middleware admission gate (internal/identity/authn/middleware/nsk.go).
AuditRelationNodeHeartbeatnode_heartbeat.recordHandler ingestion phase: clock-skew, malformed body, invariant violation, recorder failure, granted.
AuditRelationNodeHeartbeatPathGatenode_heartbeat.path_gateHandler defense-in-depth path-id gate. Distinct relation so dashboards can alert on "middleware was bypassed but handler caught it" without conflating it with ingestion-phase entries.
AuditRelationNodeReachabilityReadnode_reachability.readRead-side handler.
node_reachability.transition(literal in evaluator)Per-Domain reachability evaluator (slogReachabilityAuditSink.Record at cmd/plexsphere/evaluator_factory_prod.go).

The closed AuditOutcome set is enumerated in internal/transport/http/v1/handlers/events_deps.go. The heartbeat surface uses six distinct values so audit consumers triage clock-spoofing from malformed-body from invariant-violation from middleware-bypass from internal-error WITHOUT parsing free-form reason strings:

ValueEmitter / cause
grantedEvery layer on success.
node_id_mismatchHandler path-id gate (defense-in-depth).
clock_skewHandler clock-skew gate (client_now outside MaxHeartbeatSkew).
malformed_requestHandler JSON-decode arm.
invariant_violationRecorder rejection on the 32-byte checksum or non-empty version invariant.
internal_errorRecorder returned a non-sentinel error (Postgres outage, adapter bug).
insufficient_relationNSK middleware admission denial; reachability read-side ReBAC denial.
signature_failure(SSE bus only.)
BranchSourceRelationOutcomeReason
NSK middleware admission (granted)nsk.go middlewarenode_heartbeat.authenticategrantednsk_authenticated
NSK middleware refusal (revoked / invalid)nsk.go middlewarenode_heartbeat.authenticateinsufficient_relationnsk_invalid / nsk_revoked
Heartbeat admittedheartbeat.go step 6node_heartbeat.recordgrantedheartbeat ingested
Clock-skew refusalheartbeat.go step 4node_heartbeat.recordclock_skewclient_now outside MaxHeartbeatSkew window
Path-id mismatch (defense-in-depth)heartbeat.go step 2node_heartbeat.path_gatenode_id_mismatchresolved Node id does not match URL path id (defense-in-depth)
Malformed bodyheartbeat.go step 3node_heartbeat.recordmalformed_requestmalformed heartbeat envelope: <reason>
Checksum invariant violationheartbeat.go handleHeartbeatRecordErrornode_heartbeat.recordinvariant_violationbinary checksum invariant violated
Version invariant violationheartbeat.go handleHeartbeatRecordErrornode_heartbeat.recordinvariant_violationbinary version invariant violated
Recorder internal errorheartbeat.go handleHeartbeatRecordErrornode_heartbeat.recordinternal_errorheartbeat persistence failed: <reason>
Reachability projection read (granted)reachability.go step 3node_reachability.readgrantedreachability projection read
Reachability projection read (refused)reachability.go step 2node_reachability.readinsufficient_relationprincipal lacks node-agent relation
Transition Healthy → Staleevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat overdue (stale threshold exceeded)
Transition Stale → Unreachableevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat absent (unreachable threshold exceeded)
Transition Healthy → Unreachableevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat absent (skipped stale, hit unreachable)
Transition Stale → Healthyevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat resumed (back to healthy)
Transition Unreachable → Healthyevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat resumed (recovered from unreachable)
Transition Unreachable → Staleevaluator evaluateNodenode_reachability.transitiongrantedevaluator: heartbeat resumed (partial recovery to stale)
Transition NeverReported → Healthyevaluator evaluateNodenode_reachability.transitiongrantedevaluator: first heartbeat admitted (healthy)
Transition NeverReported → Staleevaluator evaluateNodenode_reachability.transitiongrantedevaluator: first heartbeat admitted (already past stale threshold)
Transition NeverReported → Unreachableevaluator evaluateNodenode_reachability.transitiongrantedevaluator: first heartbeat admitted (already past unreachable threshold)
Transition * → NeverReported (drift correction)evaluator evaluateNodenode_reachability.transitiongrantedevaluator: no heartbeat on record (corrected to never reported)

The full reason-string table lives in TransitionReason in evaluator.go so audit consumers can switch on the exact strings without parsing free-form text. Idempotent no-op writes ((false, nil) from Writer.Update) suppress the audit + event arms — only the row that actually transitioned is the single emitter.

OpenAPI surface reference ​

The wire surface is pinned in api/openapi/plexsphere-v1.yaml under the mesh tag. The relevant operationIds and schemas:

OpenAPI artefactWhere in the specGo counterpart
PostNodeHeartbeat operationpaths./v1/nodes/{id}/heartbeat.postHandlers.PostNodeHeartbeat in heartbeat_dispatch.go (gated entry) and postNodeHeartbeat in heartbeat.go (body).
GetNodeReachability operationpaths./v1/nodes/{id}/reachability.getHandlers.GetNodeReachability in reachability_dispatch.go (gated entry) and getNodeReachability in reachability.go (body).
HeartbeatRequest schemacomponents.schemas.HeartbeatRequesthandlers.HeartbeatRequest in heartbeat.go.
HeartbeatResponse schemacomponents.schemas.HeartbeatResponsehandlers.HeartbeatResponse in heartbeat.go.
Reachability schemacomponents.schemas.Reachabilityhandlers.ReachabilityResponse in reachability.go; mirrored on the bounded-context value object tenancy.Reachability.
NodeStateSnapshot.reachability fieldcomponents.schemas.NodeStateSnapshot.reachability ($ref: Reachability)The reconciliation-pull surface (./reconciliation-pull.md) carries the same Reachability projection on the cold-start envelope so plexd recovers the verdict without a second round-trip.
Problem schemacomponents.schemas.Problem (shared)The four heartbeat refusal codes are clock_skew, node_id_mismatch, binary_checksum_empty, binary_version_empty; the read-side handler emits unauthorized, insufficient_relation, node_not_found (plus node_id_mismatch / nsk_revoked / nsk_invalid on the NSK arm); both surfaces emit heartbeat_not_provisioned / reachability_not_provisioned on the 501 fail-closed stub.

The Reachability schema pins the four-value state enum (never_reported, healthy, stale, unreachable) and lists state and changed_at as its only required properties. last_heartbeat_at is optional on the wire and absent until the first heartbeat is accepted, which is exactly what a never_reported Node answers with. The same schema is $ref-ed by the NodeStateSnapshot.reachability block, so the reconciliation pull and the reachability read agree value for value.

The OpenAPI byte-equality drift gate at tests/workspace/openapi_drift_test.go asserts that the source spec and the embedded mirror under internal/transport/http/v1/handlers/plexsphere-v1.yaml are byte- identical, so a spec drift surfaces at go test time, not at runtime.

Threat model ​

The heartbeat surface mitigates four classes of attack along the plexd → plexsphere ingestion path. Each mitigation is implemented in a single, named place so a reader chasing a security claim does not have to assemble it from multiple files.

  1. Clock spoofing. A compromised plexd with a forged client clock could mask absence by stamping client_now in the future, or trigger a spurious Unreachable verdict by stamping it in the distant past, or trigger a spurious recovery by stamping it after a long quiet period. Mitigation: the handler admits the heartbeat only when |client_now − server_now| ≤ MaxHeartbeatSkew (60s) — pinned at internal/platform/clock/skew.go — and the state machine then drives transitions purely from the server's now and the server-stamped last_heartbeat_at. Client- claimed timestamps NEVER reach Apply's input.
  2. Cross-Node impersonation. A Node with a valid NSK could attempt to heartbeat on behalf of another Node by POSTing to a sibling's /v1/nodes/{other_id}/heartbeat. Mitigation: the NSKAuthenticator middleware in internal/identity/authn/middleware/nsk.go resolves the bearer to a tenancy.Node and enforces a path-id- equals-resolved-Node-id gate; a mismatch surfaces as 403 node_id_mismatch. The handler runs the same check as defense-in- depth (heartbeat.go step 2) so a misconfigured router that mounts the handler without the middleware still rejects. The per-handler-path mount in cmd/plexsphere/heartbeat_factory_prod.go pins the middleware on /v1/nodes/{id}/heartbeat only — see the file-level DECISION block on why a global NSK gate would either reject every cookie-authenticated dashboard call or admit an NSK envelope where cookie/JWT was expected.
  3. Checksum strip / replay. An attacker with a captured heartbeat could attempt to replay it to mask a paused or compromised agent, or strip the binary_checksum field to bypass downstream integrity checks. Mitigation: the aggregate's RecordHeartbeat invariant rejects empty / wrong-length checksums with 400 binary_checksum_empty, and the persistence-layer CHECK constraint on plexsphere.nodes.binary_checksum is the schema-level backstop. The drifted-checksum case (a heartbeat whose checksum no longer matches the canonical artefact registry) is NOT refused here — that refusal lives in the artifact registry context once the canonical checksum store is in place; this context's invariant is "well-formed shape", not "matches the canonical artefact".
  4. Stale or revoked NSK admission. A stolen NSK that the operator has rotated could attempt to refresh a Node's liveness verdict after revocation. Mitigation: the NSK middleware's Resolver port refuses to admit a heartbeat whose NSK row is in the revoked set; the production binding will surface 401 nsk_invalid (mapped to the nsk_revoked audit reason) once the real plaintext-resolver lands. Today the failClosedNSKResolver returns ErrNSKResolverNotProvisioned for every Resolve call so EVERY NSK envelope is refused — fail-closed by construction.

What this context is NOT ​

This context ships the heartbeat ingestion + per-Node liveness-tracking surface. The following adjacent concerns are explicit non-goals — each is owned by a downstream context:

  • Hash-chained durable audit log. The audit entries this context emits flow through a slog-backed sink. The durable hash-chained sink, the audit relation registry, and the audit-log query API land in the Platform Audit Log context.
  • Node deregistration / lifecycle. A Node that goes Unreachable is NOT removed from the Domain by this surface. The deregistration workflow, the per-Node lifecycle state machine, and the operator-driven force_deregister action are owned by the Key & Peer Manager context — the lifecycle and deregistration sibling for the per-Node liveness surface. The reachability state remains Unreachable indefinitely until either a recovery heartbeat arrives or an operator action removes the Node.
  • Relay-fallback reachability heuristics. The bridge / relay orchestrator's choice of fallback bridge per peer pair MAY consult the reachability projection, but the heuristics themselves (latency, load, churn) are owned by the relay-fallback assignment context. The per-Node reachability projection is one of several inputs that consumer takes; this surface simply exposes the read.
  • Node state service reports. The state and reports blocks on NodeStateSnapshot (the reconciliation-pull envelope) ship as JSON null placeholders today and become populated value objects in the node state service context. The reachability projection on the same envelope is the only block this context populates; the rolled-up state reports are the node state service's responsibility.
  • Operator-tunable MaxHeartbeatSkew. The 60-second bound is a security invariant, not a tuning knob — the file-level DECISION block in internal/platform/clock/skew.go rejects the configuration alternative because widening would silently erode the replay-resistance the threat model relies on. A future per-Domain knob is NOT planned.
  • Rate limiting heartbeats below HeartbeatInterval. A plexd that POSTs faster than the per-Domain HeartbeatInterval is NOT rate-limited here — the surface admits every well-formed heartbeat. Per-Node rate limiting is a future surface; the brief explicitly lists it as a deferred edge case.
  • Driving the Reconcile / RotateKeys hint flags. The HeartbeatResponse envelope carries the two boolean hints today but the handler always emits them as false. Reconcile becomes load-bearing once snapshot-divergence detection lands as a follow-up to the reconciliation-pull surface; RotateKeys becomes load-bearing once the key rotation workflow lands. The wire shape is stable today so plexd's reconcile loop can pin against it.