plexsphere

Appearance

Background leader ​

internal/platform/db/leader elects one background leader per API deployment. Every API replica runs one Elector. The replica whose Elector holds a PostgreSQL session advisory lock is the leader. Every other replica is a standby and tries the lock again on a fixed interval.

This page is the reference for:

See also:

The lock key ​

The leader lock is pg_try_advisory_lock(1886156915, 1), the two-int4 form of the advisory lock functions. 1886156915 is 0x706C7873, ASCII plxs. The held lock shows up in pg_locks as:

ColumnValue
locktypeadvisory
classid1886156915
objid1
objsubid2

PostgreSQL keeps two-int4 keys and single-bigint keys apart: pg_locks shows a two-int4 key with objsubid = 2 and a bigint key with objsubid = 1. The transaction-scoped locks the repositories take through pg_advisory_xact_lock(bigint) therefore never collide with the leader lock, not even the bigint whose halves equal the pair, (1886156915::bigint << 32) | 1.

Advisory locks are scoped to one database. Run this query in the plexsphere database to find the leader's backend:

sql
SELECT pid, granted
FROM pg_locks
WHERE locktype = 'advisory'
  AND classid = 1886156915
  AND objid = 1
  AND objsubid = 2
  AND database = (SELECT oid FROM pg_database WHERE datname = current_database());

One granted row is the leader's backend; pg_stat_activity maps its pid to a client address. No row means no replica leads at the moment.

The PostgreSQL manual covers advisory locks and the pg_locks view.

Config ​

leader.New(cfg) builds an Elector from a leader.Config:

FieldTypeDefaultPurpose
Pool*pgxpool.Poolnone, requiredSupplies the connections the lock is tried on. New returns ErrNilPool when it is nil.
CheckIntervaltime.Duration5s (DefaultCheckInterval)How often a standby tries the lock and the leader verifies it, and how long the leader's SELECT 1 may take before the check fails. Zero or a negative value uses the default.
Logger*slog.Loggerslog.Default()Receives acquisition, loss, and failure lines.
Registererprometheus.RegisterernilReceives the plexsphere_background_leader gauge. Nil registers no gauge.

Elector API ​

Run ​

Run(ctx) takes part in the election until ctx is cancelled. It checks once when it starts and then every CheckInterval:

  • On a standby, a check acquires a connection from Pool and runs pg_try_advisory_lock on it. When another session holds the lock, the connection goes back to the pool unchanged. When the lock is granted, the connection is hijacked out of the pool, the replica leads, and a new term begins.
  • A failed acquire or query is logged at WARN as leader: acquire lock connection: <cause>. The replica stays a standby and tries again at the next check. Run does not return.
  • On the leader, a check runs SELECT 1 on the held connection. The check fails when the query returns an error and when it does not answer within one CheckInterval. Either one demotes the replica, see Demotion and the overlap window.
  • When ctx is cancelled, Run runs pg_advisory_unlock under a 2-second context that outlives ctx, closes the connection, ends the term, and returns nil. A standby replica takes the lock at its next check, within one CheckInterval.

When a panic unwinds Run, it ends the term with ErrLeadershipLost and closes the lock connection before the panic propagates. It does not unlock, because the panic may have left the connection mid-query; the server releases the lock when the session ends. When the replica led, it logs background leader lock abandoned after a panic at WARN. A replica whose election has stopped therefore never keeps a lock that no check verifies any more.

Never run two Run calls at once. Run may be called again after an earlier call returned or panicked, which is what the Supervisor of internal/platform/worker does when it restarts a worker.

Term ​

Term(parent) returns a context, its cancel function, and a bool. On the leader the bool is true and the context ends when parent ends or when the replica loses leadership, whichever comes first. After a loss, context.Cause on the context returns ErrLeadershipLost. On a standby the bool is false and the context is already cancelled. Always call the cancel function.

Do ​

Do(ctx, fn) runs one unit of background work on the leader. fn receives the term context, so it can stop when leadership ends.

Situationfn calledResult
Standbyno(false, nil)
Leader, leadership held until fn returnsyes(true, err), where err is fn's error
Leader, leadership lost while fn runs and ctx still liveyes(false, nil), logged at INFO

A demotion never counts as a failed tick.

IsLeader ​

IsLeader() reports whether the replica holds the lock. It reads an atomic and is safe to call from any goroutine.

Which loops run on the leader ​

The background-leader surface builds the Elector on the worker tier pool and publishes it on Runtime.Leader. It holds index 0 of the surface registry, so every later surface reads the elector while its own bundle is still being built and gates its loops on it. A boot without PLEXSPHERE_DSN builds no Elector, and such a boot wires no background loop at all, so nothing is left ungated by accident.

Every loop, and the Elector itself, runs as a supervised worker of internal/platform/worker, registered by name. The election is the worker background-leader, and each loop below is named in its table's first column. The supervisor starts all of them just before the API serves its routes.

Two launchers in ../../../cmd/plexsphere/background_loops.go place a loop. runLeaderTicker ticks a pass on a fixed cadence and asks Do before each tick. runWhileLeader hands a self-paced loop the context of one term and starts it again in the next one.

These tick on the leader through runLeaderTicker:

LoopPass
invitations-expireExpires the elapsed pending invitations.
approvals-expireExpires the proposals past their deadline.
actions-timeoutTimes out the expired live executions.
access-session-sweepRevokes the expired and idle sessions.
auth-state-sweepDrains the elapsed sign-in states, browser sessions and device authorizations.
rate-budget-sweepReclaims the rate-budget token buckets idle for longer than an hour.
sse-nonce-sweepDeletes the signed-event-bus nonces last seen before the freshness window, once per PLEXSPHERE_SSE_NONCE_TTL.
credentials-sweeperExpires the credentials past their TTL.
cloud-credentials-sweeperExpires the cloud credentials past their TTL.
management-fleet-reconcilePlaces Projects and converges cluster packages.
provisioning-broker-reconcileReconciles the declared provisioned resources.
artifacts-refreshRe-indexes the artifact catalogue.
blueprint-catalogs-trackingRe-imports the track-tag catalog sources.
peers-endpoint-sweeperTombstones the endpoint observations past TTL.
reachability-evaluatorRuns the stale and the recovery pass over Nodes.

These run while leader through runWhileLeader, because they pace themselves instead of ticking:

LoopPass
audit-archiverDrains the audit chains to the object store.
authz-outbox-consumerMirrors the outbox rows into SpiceDB.

These stay on every replica:

LoopWhy
capacity-collector, the capacity collector's sampling loopEvery replica samples and tracks the same hysteresis. Only the audit row for a crossing is gated, through capacity.WithCrossingGate.
observability-routing-consume, the observability routing consume loopsA durable spreads its deliveries across every puller bound to it, so each replica drains its share. Only the durable removals are gated, through services.WithDurableDeletion.
sse-relay and sse-signing-key-rotation-watcher, the signed event bus relay and the signing-key rotation watcherThe FOR UPDATE SKIP LOCKED lock on the relay's cursor row serialises the ticks across replicas, so a leader would only add a handover delay. The watcher runs on every replica because each replica has its own in-memory key cache to invalidate.
peers-relay-assigner, the peers relay-assigner heartbeatIt re-flips a local readiness flag and takes no write.

A gate that is given no predicate stays open, so a single-replica deployment records every crossing and removes every orphaned durable.

The boot-time passes are not gated. A sweep that drains the rows which fell due while the process was offline runs on every replica before it reports ready, and the compare-and-set writes those passes take keep the outcome correct whichever replica gets there first.

Readiness ​

Seven of the ticked loops carry a /readyz probe: invitations-expire, approvals-expire, access-session-sweeper, actions-timeout, management-fleet-reconcile, provisioning-broker-reconcile, and artifacts-refresh. A probe reports the outcome of the last tick its loop ran on this replica, and it runs no pass of its own.

Replica stateWhat the probe reports
Booted, no tick yetThe boot pass, which gates readiness before the API serves its routes
Leader, last tick succeededPass
Leader, last tick failedThat tick's error, until a later tick succeeds
StandbyPass, because a standby runs no pass and has no outcome of its own

A demoted leader's probe clears on its first standby tick, so a replica that stops leading does not keep reporting the error of the last pass it ran. None of the seven is boot-latched: a failing tick flips the leader to HTTP 503 and takes it out of its Service until a later tick succeeds.

A pass that panics counts as a failed tick, and the loop keeps ticking. The replica logs background loop tick panicked at ERROR with the loop, panic, and stack attributes, and the loop's probe reports plexsphere: background loop "<loop>": tick panicked: <value> until a later tick succeeds. A loop without a probe leaves only that line and the WARN line every failed tick writes.

A sweep probe reports the outcome of a tick, so it cannot see a loop that has stopped ticking altogether. A loop that stops after exhausting its restart budget is reported by the workers probe instead, see Background workers.

The probe name access-session-sweeper differs from its loop name access-session-sweep; the other six match.

Metric ​

MetricTypeLabelsValue
plexsphere_background_leadergaugenone1 while the replica leads, 0 otherwise

Summed over all replicas of a deployment, the gauge is 1 in steady state. It is 0 while no replica holds the lock and can be 2 for the length of the overlap window below.

New reuses a gauge already registered on the same Registerer, so two electors that share a registry write to one series. Tests that run two electors in one process give each its own prometheus.NewRegistry().

Demotion and the overlap window ​

The server drops a session advisory lock when the session ends: a terminated backend, a server restart, or a broken connection. The leader notices at its next check, when SELECT 1 fails or stays unanswered for one CheckInterval. It then:

  1. ends the term with cause ErrLeadershipLost, which cancels every Term context and every Do in flight, turns IsLeader false, and sets the gauge to 0;
  2. closes the connection;
  3. logs a WARN line with the ping error.

The replica is then a standby and tries the lock again at its next check.

The timeout covers the connection that breaks without the client learning it. After a network partition, or a failover to a new primary, SELECT 1 gets no answer and the kernel retransmits it for minutes before it reports the socket broken. An unbounded check would hold the term open for that whole time, while the server has already let another replica take the lock.

A new leader can start before the old one notices, so leadership can overlap for at most one CheckInterval until the old leader's next check, one more for a SELECT 1 that never answers, and one tick that was already running. The Elector does not fence writes made in that window. Work run on the leader has to stay correct when two replicas run it at once, for example by writing with compare-and-set updates.

Session connection requirement ​

The lock lives on one database session, held on one connection that the Elector hijacks out of the pool. The pool no longer tracks that connection, so MaxConnLifetime, idle recycling, and pool health checks never close it while it holds the lock. A connection left in the pool would be retired at MaxConnLifetime, and the lock would go with it.

The hijack happens only once the lock is granted. A standby's check borrows a connection from the pool and releases it again when the lock is refused, so an election that is refused every few seconds for the life of the process costs the pool nothing. Hijacking before the outcome is known would destroy one pooled connection per check, and the pool would open a fresh backend to replace each one.

While a replica leads it keeps that one connection open.

Bound on an orphaned lock session ​

The session that carries the lock takes an idle_session_timeout of four CheckIntervals, set on the connection as soon as the lock is granted. The leader's SELECT 1 runs once per interval, so a leader that is alive never comes close to it.

The bound matters when a leader's process stops without closing the connection: a node that loses power, a hard kill, a partition. The server-side backend then sits waiting for bytes that never come, and it holds the lock while it waits. TCP keepalives eventually tear it down — with Linux defaults, hours later. Until then no standby can take the lock, and the whole background plane of the deployment stands still. idle_session_timeout ends that session within four check intervals instead, and the lock goes with it.

The timeout is not a lease. It bounds a session that has stopped talking; it never expires under a leader that is merely slow, because a leader that reaches its own check runs a query, which ends the idle period.

A transaction-mode connection pooler, such as PgBouncer with pool_mode = transaction, cannot carry a session lock, because it hands the server session to another client after every transaction. Connect the API to PostgreSQL directly or through a session-mode pooler.

Cross-references ​