Skip to content

A cell's heartbeat reports serves, marked and the router's answer beside its fingerprint - #693

Merged
zlareb1 merged 31 commits into
mainfrom
feat/cell-heartbeat-keys
Nov 16, 2026
Merged

zlareb1 merged 31 commits into
mainfrom
feat/cell-heartbeat-keys

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Improvement

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

A cell's heartbeat reports whether each process serves traffic, whether its Postgres sessions still carry the tenancy marker, and, on a first cell, what the router's directory says of it. A readiness check fails on an enforced cell whose sessions lost the marker. A cell refuses to boot when a set LOOM_CELL_ROUTER_URL is not an http or https URL with a host, a port of at most 65535 and no ? or #.

Why

The first cell's attach can only trust processes that will be refused correctly once its mode rises. That means processes that serve, run a build that checks tenancy, keep the tenancy marker on their pooled sessions, and agree with the router on the cell's state. Attach reads each live process's heartbeat to judge this, so the heartbeat has to report those facts. The attach command that reads them is a later change.

Design:

  • Heartbeat keys. The keys go beside the fingerprint in the heartbeat's JSON (instances.cell_row()), outside fields and never in cell.as_value(). They are never written to loom.cell_self, the directory or a placement body. No comparison of fingerprints reads them, so a build N-1 beside N ignores them. No column is added.
    • serves: settings.serves_traffic.
    • marked: whether this process's pooled connection keeps loom.tenancy_checked.
      • It is read in the process's own cell refresh as one more column of cell._READ, so every process reports a fresh value with no prober.
      • The refresh runs RESET ALL and then the read in one transaction (pg.transaction()). A startup value survives RESET ALL and a SET does not, so a pooler, which a cell must not have, reads unmarked whether it strips startup parameters or sends them again with SET.
      • It is null until a read lands, and stays null under the ClickHouse authority. An unmarked session reads false.
    • router and router_why, with LOOM_CELL set: null until the first row is read, then on a first cell only: whether this process reads the directory's state of its cell, and why not.
      • A server process runs one background reader. It asks every min(instance_heartbeat_seconds, tenancy_mode_cache_s) and stops once it reads serving.
      • A CLI or the workers child asks on demand, at most once per tenancy_mode_cache_s, with waiting requests sharing one call.
      • A 401 turns router false only once 401s persist past router_directory_cache_s, since a credential rotation passes through one. A transient failure keeps the last value.
    • tenancy_mode reports the state the rows make (open, recording, enforced or partial).
  • Readiness. A cell_marker check (the name loom doctor gives the same fact) reads the same cached marked. It fails in enforced and partial, and passes with a reason in recording. It is registered only with LOOM_CELL set under the Postgres authority.
  • Unscoped callers on a first cell. An unscoped caller (no principal to place for) is answered 503 while the cell is attaching or the directory cannot be read, and WRONG_CELL once it is serving or on a later cell. The router-state read moves from server/auth/tenancy.py into server/core/cell_router.py, which the reader and ensure_held share.
  • Boot check. cell.settings_problem refuses a non-empty LOOM_CELL_ROUTER_URL that fails is_http_url: it must parse as a URL, have scheme http or https, a non-empty host, a numeric port of at most 65535 if any, and no ? or # anywhere. It runs only with LOOM_CELL set; an empty value stays allowed.
  • The router's answer is cached per replica. Router.cell_state (the cell-credential GET /cells/v1/cells/<cell>) reads the directory at most once per router_directory_cache_s per cell. A failed read is held as a 503 for its window. The entry is dropped after attach/serving, and the cache is cleared on any credential or registration change. So the directory sees one read per replica per window, however many processes ask.

Scope and risk

  • server/router/app.py (the per-replica state cache), server/router/directory.py (a docstring), server/core/cell_router.py (new), server/core/cell.py, server/core/cell_triggers.py (the readiness check only), server/core/instances.py, server/core/pg.py (fetchrow_reset), server/auth/tenancy.py, server/boot.py, server/cli/_ch.py (a refused org no longer names loom cell place, which does not exist yet; the place PR adds the command), server/settings/catalog.py (is_http_url made public; three settings comments, with the generated docs/reference/configuration.md, schemas/loom-settings.schema.json and loom.example.yaml), docs/operations/upgrades.md (step 17), docs/architecture/cells.md, docs/operations/troubleshooting.md.
  • On a cell, the refresh costs three extra round trips each tenancy_mode_cache_s, on one pooled connection; elsewhere it costs none.
  • A deployment that is not a cell (LOOM_CELL unset) runs no router reader and gets no new boot refusal. Under the Postgres authority two things change for it (its refresh stays one plain read; the reset is a cell's alone): its heartbeat row's tenancy_mode reports the derived state (open) instead of ''; and its fingerprint JSON gains serves and marked.

Compatibility, rollout and rollback

  • Wire: none. The heartbeat keys are JSON beside the fingerprint, which an N-1 reader ignores.
  • Breaks:
    • A cell pod whose LOOM_CELL_ROUTER_URL fails the rule above now refuses to boot, where the previous build served.
    • On a first cell, a platform, *-org or principal-less call for an org the cell does not hold now answers 503 while the directory says attaching or cannot be read, where the previous build answered 409 WRONG_CELL. It answers WRONG_CELL once the cell is serving, and on a later cell.
    • On the router, a cell credential replaced mid-window keeps reading the cached state until the window passes, then gets 401.
    • A process with LOOM_CELL set whose Postgres sessions lose the tenancy marker (a pooler between it and Postgres) is now unready in enforced and partial (the cell_marker readiness check), where the previous build served and its triggers refused its writes for unheld orgs. Before upgrading such a cell, point LOOM_PG_DSN straight at Postgres (troubleshooting).
  • Operator steps: every router replica runs this build before any cell's processes do (upgrades.md, step 17). A cell process of this build asks the router for its state every period, and only this build's router answers that from a per-replica cache; an older router would read the directory on every call. The troubleshooting page names the new readiness warning.
  • Rollback: roll the cells' processes back first and the routers after, the reverse of upgrades.md step 17, so no router of the earlier build answers this build's state reads; nothing else to undo.

Tests

  • tests/unit/test_cell_heartbeat_keys.py: each key's value and its null before the first read, the keys kept out of as_value(), marked false for an unmarked session, router absent with LOOM_CELL unset or on a later cell, the cell_marker readiness check in each state, and the router URL boot check.
  • tests/unit/test_cell_router.py: the reader's period and stop at serving, the on-demand path's single call and its lateness at non-default settings, the 401 window with transient failures between 401s, the state read moved from tenancy, and the unscoped-caller answers for each of the four directory answers.
  • tests/unit/test_cell.py, test_cell_unset_gate.py, test_cell_tenancy.py, test_cell_cli_orgs.py: their fakes moved to fetchrow_reset and the derived tenancy_mode.
  • tests/unit/test_router.py and tests/unit/test_router_attach.py: the router's state cache: one directory read per replica per window, timed from the read's end; a failed read held as a 503 for its window; the cache cleared when a credential is replaced; the entry dropped after attach/serving, including while a read is in flight; a read begun before a change not cached.
  • tests/integration/test_cell_heartbeat.py: the heartbeat's tenancy_mode expected as open under the Postgres authority.
  • tests/unit/conftest.py: an autouse fixture that resets cell_router's process state around every test.
  • tests/integration/test_cell_org_triggers_pg.py: the real refresh reading marked on a real Postgres: true on a session with the startup marker, false after a session SET (which RESET ALL drops), false with no marker.
  • make format lint typecheck pass. The cell, readiness, ratchet and doc unit files pass (980), and the integration files test_cell_org_triggers_pg, test_cell_unset_gate_pg, test_cell_tenancy_pg, test_cell_cli_unset_pg and test_cell_prepare pass against a local Postgres with a synchronous standby, and test_cell_heartbeat passes against a server built from this branch under the Postgres authority. The full unit suite runs in CI; it is not run locally because the build machine is low on memory.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Claude Code; the design was reviewed in rounds to zero findings, and the tests above were run.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint and make typecheck pass, and the unit test files named above pass; the full make test-unit runs in CI.
  • Generated files are regenerated in this PR, not by hand.
  • A new migration is additive and numbered after the latest on main; no merged migration is edited.
  • A behaviour change to any file a plugin ships bumps both plugin manifests' version.
  • The frontend builds, if frontend/ changed.
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers.
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks.

🤖 Generated with Claude Code

zlareb1 and others added 30 commits November 15, 2026 07:45
…ide its fingerprint

The cell heartbeat's fingerprint JSON carries keys no fingerprint compares:
`serves` (LOOM_SERVES_TRAFFIC), `marked` (whether the process's pooled sessions
keep the tenancy marker, read in the cell refresh after RESET ALL in the same
transaction) and, with LOOM_CELL set on a first cell, `router` and `router_why`
from one background reader of the router's state route. The reader's answer is
three-valued with a sticky `serving` for the tenancy check, which answers an
unscoped caller 503 on a first cell still attaching. The `cell_triggers`
readiness check reads the same marker. The heartbeat's tenancy mode is the
derived state. `cell_triggers.ensure_table` runs one table's DDL alone, with
`recording_first` for attach. A non-empty LOOM_CELL_ROUTER_URL that cannot be
requested refuses a cell's boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…uter read counts lateness by its own period

The cell-credential state route answers from each router replica's read of the
cell's row, made at most once per LOOM_ROUTER_DIRECTORY_CACHE_S, a failed read
answering 503 for its window, so every first-cell process asking each period
costs the directory one read per replica per window. A process reading the
route on demand judges an answer late by its own window
(LOOM_TENANCY_MODE_CACHE_S plus the timeout), not the background reader's
shorter period. Only an answer that proves the configuration ends a run of 401s.
A failed reset read rolls its transaction back before the release. The
router reader's state is reset around every unit test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…he state cache drops a row under its lock

The upgrade procedure gains a step: every router replica runs the build before any
cell's processes do, since a first cell's processes ask the state route from their
first beat and only this build's router answers it from its cache. The drop after
attach/serving takes the cache's lock, so a read in flight cannot store the
pre-commit row for a window; tests cover the drop after attach/serving, a
rotation and a remove, and the guard on a read begun before a change. The
settings comments of the router timeout, the mode cache and the router's cache
name what they now bound, and the troubleshooting fix for a pooler that strips
the marker says a restart is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…and names an absent first row

The background reader's reads share one httpx client, closed when the reader
ends or is cancelled at shutdown; an on-demand read keeps a client of its own.
A warning is logged when the read's cause changes, not on every read of the
same failure. While a first cell's `first` row is absent after a read,
`router_why` says to run `loom cell prepare` rather than that the row is not read
yet. An integration test reads `marked` through the refresh on a real session:
a startup marker reads true, a leaked session SET and no marker read false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…trized one covers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…entry; a first cell reads the state route until serving

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…he heartbeat test expects open under Postgres

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ther state on the unscoped path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…with its caller

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…refusal for an unplaced org is pinned and documented as naming the org

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ll a 401 by its constant cause

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… seam; the state row is read unguarded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…gs and new tenancy rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s with its caller; unset router settings open no client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ave; a server names its unset router settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…why before the first row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…kResults

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…hy the reset read sends its own BEGIN

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…marker reason names any pooler

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…r check builds its failure directly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ding-insert sentence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…lient only when it can ask

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ader refusal branch covers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…as false

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ment the docstring holds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s three round trips more

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… its read rate rule

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… before refusing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…router settings logs nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 merged commit c95cea0 into main Nov 16, 2026
48 checks passed
@zlareb1
zlareb1 deleted the feat/cell-heartbeat-keys branch November 16, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant