Rate budgets

internal/platform/db/ratebudget holds the per-Node and per-Domain rate budgets the API admits requests through. Each bucket is one row in the database every replica shares, so a deployment admits the configured rate however many replicas serve it.

This page is the reference for:

• The five scopes
• The table
• What a take does
• Store API
• When the store cannot answer
• The idle sweep
• Reading a budget in production

See also:

• db.md: the pools the composition root builds, two of which this package's callers bind to.
• leader.md: the elector the idle sweep ticks on.

The five scopes

A scope names which budget a bucket belongs to. The set is closed: it is the CHECK on the table and the constants in the package, and a sixth budget adds both in one change.

Scope	Keyed by	Cost of one request	Spent by
secret_read_node	Node	1 read	internal/secrets/repo
secret_read_domain	Domain	1 read	internal/secrets/repo
access_issuance_domain	Domain	1 issuance	internal/access/repo
ingest_node	Node	wire bytes	internal/observability/ingest/repo
ingest_domain	Domain	wire bytes	internal/observability/ingest/repo

Each bounded context declares its own limiter port and binds it to this store through an adapter in its own persistence package. No context depends on another, and no domain package sees the table.

The table

plexsphere.rate_budgets, created by migration 0088_rate_budgets.sql:

Column	Type	Meaning
scope	text	One of the five above.
bucket_key	uuid	The Node or the Domain whose budget it is.
tokens	double precision	What is left. Never negative.
refilled_at	timestamptz	When the tokens were last brought up to date.

The primary key is (scope, bucket_key), and there is an index on refilled_at for the sweep.

The budget itself — the rate and the ceiling — is not a column. It travels with every take. The numbers come from environment knobs each replica reads once at boot, so a stored copy would hold whichever replica last wrote it, and an operator lowering a budget would wait for a rewrite that nothing owns. Passing it also lets one caller size a bucket from a Domain's own policy while another uses a process-wide default, which is exactly what the issuance limiter and the secret-read limiter do.

bucket_key carries no foreign key. The column holds Node ids under two scopes and Domain ids under three, so no single parent table exists, and a bucket is an admission tally rather than a record of the caller. A row whose Node or Domain is gone simply stops being taken from, and the idle sweep deletes it.

What a take does

One statement, TakeRateBudget in Z4_rate_budgets.sql:

1. A key with no row gets a bucket minted at the ceiling and debited in the same statement. Starting full is what makes the configured burst of back-to-back requests admissible before the sustained rate applies.
2. An existing bucket is refilled by elapsed × rate, capped at the ceiling, and debited only when that refilled level covers the cost.
3. A take that does not cover the cost matches no row, writes nothing, and comes back as a refusal. Leaving the row alone matters: stamping a refusal would discard the tokens accrued since, so a caller that retried under a full bucket would be throttled harder the more often it asked.

Two concurrent takes on one bucket serialise on the row. PostgreSQL evaluates the statement's conflict arm against the latest committed row version, so the last token is granted once rather than to both callers. That is the property the whole design rests on, and tests/integration/rate_budget_test.go pins it with a hundred takes issued from two pools at one instant.

now is the caller's clock, not the server's, because the limiters are driven by an injected clock so their rate-limit branches are testable without sleeping. The statement clamps a negative elapsed interval to zero and never moves refilled_at backwards, so a replica whose clock runs behind under-refills for the skew rather than granting tokens twice.

Store API

type Scope string
type Key [16]byte
type Budget struct {
    PerSecond float64
    Burst     float64
}

func NewStore(db sqlcgen.DBTX) *Store

func (s *Store) Take(
    ctx context.Context,
    scope Scope,
    key Key,
    cost float64,
    budget Budget,
    now time.Time,
) (bool, error)

func (s *Store) DeleteIdle(ctx context.Context, before time.Time) (int64, error)

Take reports true when the bucket covered the cost and has been debited, and false when it did not. A refusal is not an error: every caller owns a refusal sentinel of its own that names its budget for its transport layer, so a sentinel here would only be translated away.

Two requests never reach the database:

• A zero cost is admitted outright. It cannot exhaust a bucket, and a round trip for it would be waste on a hot path.
• A cost above the ceiling is refused outright. No bucket ever holds that much, and the insert arm would otherwise mint a negative balance and trip the table's non-negative CHECK — handing the caller a constraint violation where it asked a question.

NewStore panics on a nil handle, and Take returns ErrUnknownScope, ErrZeroKey, ErrNegativeCost or ErrInvalidBudget before any query.

When the store cannot answer

A non-nil error from Take means the budget could not be evaluated — a third outcome, distinct from a refusal. Every caller fails the request closed and propagates it, and none of them reports it as a breach.

That distinction is load-bearing. The two secret-read sentinels, the issuance limit error and the two ingest quota sentinels all render as 429, so an unreachable budget that matched one would tell a whole fleet of agents to back off from a cap none of them reached. Instead the wrapped error reaches each handler's default arm and renders as the generic 500 a write funnel already answers during a datastore outage, with the cause in the log. See failure-modes.md.

The idle sweep

The composition root ticks DeleteIdle once an hour on the background leader, as the loop rate-budget-sweep. It deletes every bucket last taken from more than an hour ago.

The retention is not a tuning choice. Every shipped budget refills to full within seconds of being left alone — five for the issuance bucket, two for the secret-read ones, four for the ingest quotas — so a bucket idle for an hour is a full bucket, and deleting it is exactly what the next take would mint. What it reclaims is the rows of Nodes and Domains that are gone.

The loop carries no readiness probe. A bucket that has not been swept makes no surface wrong, because the take statement refills from the stored instant whatever the row's age, so a sweep that never ran costs rows rather than correctness.

Reading a budget in production

The buckets are readable with SQL. One Domain's, across every scope:

SELECT scope, tokens, refilled_at
FROM plexsphere.rate_budgets
WHERE bucket_key = '<domain-uuid>'
ORDER BY scope;

A low tokens with a recent refilled_at is a caller spending its budget. A row that is absent is not a problem: the next take mints it full.

To see whether a deployment is sharing its budgets, count the buckets behind one Domain's ingest quota. It is one row however many replicas serve the fleet:

SELECT count(*)
FROM plexsphere.rate_budgets
WHERE scope = 'ingest_domain'
  AND bucket_key = '<domain-uuid>';