Repository navigation
A cell's heartbeat reports serves, marked and the router's answer beside its fingerprint - #693
Merged
Merged
Conversation
…ide its fingerprint The cell heartbeat's fingerprint JSON carries keys no fingerprint compares: `serves` (LOOM_SERVES_TRAFFIC), `marked` (whether the process's pooled sessions keep the tenancy marker, read in the cell refresh after RESET ALL in the same transaction) and, with LOOM_CELL set on a first cell, `router` and `router_why` from one background reader of the router's state route. The reader's answer is three-valued with a sticky `serving` for the tenancy check, which answers an unscoped caller 503 on a first cell still attaching. The `cell_triggers` readiness check reads the same marker. The heartbeat's tenancy mode is the derived state. `cell_triggers.ensure_table` runs one table's DDL alone, with `recording_first` for attach. A non-empty LOOM_CELL_ROUTER_URL that cannot be requested refuses a cell's boot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…uter read counts lateness by its own period The cell-credential state route answers from each router replica's read of the cell's row, made at most once per LOOM_ROUTER_DIRECTORY_CACHE_S, a failed read answering 503 for its window, so every first-cell process asking each period costs the directory one read per replica per window. A process reading the route on demand judges an answer late by its own window (LOOM_TENANCY_MODE_CACHE_S plus the timeout), not the background reader's shorter period. Only an answer that proves the configuration ends a run of 401s. A failed reset read rolls its transaction back before the release. The router reader's state is reset around every unit test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…he state cache drops a row under its lock The upgrade procedure gains a step: every router replica runs the build before any cell's processes do, since a first cell's processes ask the state route from their first beat and only this build's router answers it from its cache. The drop after attach/serving takes the cache's lock, so a read in flight cannot store the pre-commit row for a window; tests cover the drop after attach/serving, a rotation and a remove, and the guard on a read begun before a change. The settings comments of the router timeout, the mode cache and the router's cache name what they now bound, and the troubleshooting fix for a pooler that strips the marker says a restart is needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…and names an absent first row The background reader's reads share one httpx client, closed when the reader ends or is cancelled at shutdown; an on-demand read keeps a client of its own. A warning is logged when the read's cause changes, not on every read of the same failure. While a first cell's `first` row is absent after a read, `router_why` says to run `loom cell prepare` rather than that the row is not read yet. An integration test reads `marked` through the refresh on a real session: a startup marker reads true, a leaked session SET and no marker read false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…trized one covers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…entry; a first cell reads the state route until serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…he heartbeat test expects open under Postgres Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ther state on the unscoped path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…with its caller Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…refusal for an unplaced org is pinned and documented as naming the org Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ll a 401 by its constant cause Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… seam; the state row is read unguarded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…gs and new tenancy rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s with its caller; unset router settings open no client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ave; a server names its unset router settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…why before the first row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…kResults Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…hy the reset read sends its own BEGIN Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…marker reason names any pooler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…r check builds its failure directly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ding-insert sentence Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…lient only when it can ask Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ader refusal branch covers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…as false Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ment the docstring holds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s three round trips more Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… its read rate rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… before refusing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…router settings logs nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
A cell's heartbeat reports whether each process serves traffic, whether its Postgres sessions still carry the tenancy marker, and, on a first cell, what the router's directory says of it. A readiness check fails on an
enforcedcell whose sessions lost the marker. A cell refuses to boot when a setLOOM_CELL_ROUTER_URLis not anhttporhttpsURL with a host, a port of at most 65535 and no?or#.Why
The first cell's attach can only trust processes that will be refused correctly once its mode rises. That means processes that serve, run a build that checks tenancy, keep the tenancy marker on their pooled sessions, and agree with the router on the cell's state. Attach reads each live process's heartbeat to judge this, so the heartbeat has to report those facts. The attach command that reads them is a later change.
Design:
instances.cell_row()), outsidefieldsand never incell.as_value(). They are never written toloom.cell_self, the directory or a placement body. No comparison of fingerprints reads them, so a build N-1 beside N ignores them. No column is added.serves:settings.serves_traffic.marked: whether this process's pooled connection keepsloom.tenancy_checked.cell._READ, so every process reports a fresh value with no prober.RESET ALLand then the read in one transaction (pg.transaction()). A startup value survivesRESET ALLand aSETdoes not, so a pooler, which a cell must not have, reads unmarked whether it strips startup parameters or sends them again withSET.nulluntil a read lands, and staysnullunder the ClickHouse authority. An unmarked session readsfalse.routerandrouter_why, withLOOM_CELLset:nulluntil thefirstrow is read, then on a first cell only: whether this process reads the directory's state of its cell, and why not.min(instance_heartbeat_seconds, tenancy_mode_cache_s)and stops once it readsserving.tenancy_mode_cache_s, with waiting requests sharing one call.routerfalse only once 401s persist pastrouter_directory_cache_s, since a credential rotation passes through one. A transient failure keeps the last value.tenancy_modereports the state the rows make (open,recording,enforcedorpartial).cell_markercheck (the nameloom doctorgives the same fact) reads the same cachedmarked. It fails inenforcedandpartial, and passes with a reason inrecording. It is registered only withLOOM_CELLset under the Postgres authority.WRONG_CELLonce it isservingor on a later cell. The router-state read moves fromserver/auth/tenancy.pyintoserver/core/cell_router.py, which the reader andensure_heldshare.cell.settings_problemrefuses a non-emptyLOOM_CELL_ROUTER_URLthat failsis_http_url: it must parse as a URL, have schemehttporhttps, a non-empty host, a numeric port of at most 65535 if any, and no?or#anywhere. It runs only withLOOM_CELLset; an empty value stays allowed.Router.cell_state(the cell-credentialGET /cells/v1/cells/<cell>) reads the directory at most once perrouter_directory_cache_sper cell. A failed read is held as a 503 for its window. The entry is dropped afterattach/serving, and the cache is cleared on any credential or registration change. So the directory sees one read per replica per window, however many processes ask.Scope and risk
server/router/app.py(the per-replica state cache),server/router/directory.py(a docstring),server/core/cell_router.py(new),server/core/cell.py,server/core/cell_triggers.py(the readiness check only),server/core/instances.py,server/core/pg.py(fetchrow_reset),server/auth/tenancy.py,server/boot.py,server/cli/_ch.py(a refused org no longer namesloom cell place, which does not exist yet; the place PR adds the command),server/settings/catalog.py(is_http_urlmade public; three settings comments, with the generateddocs/reference/configuration.md,schemas/loom-settings.schema.jsonandloom.example.yaml),docs/operations/upgrades.md(step 17),docs/architecture/cells.md,docs/operations/troubleshooting.md.tenancy_mode_cache_s, on one pooled connection; elsewhere it costs none.LOOM_CELLunset) runs no router reader and gets no new boot refusal. Under the Postgres authority two things change for it (its refresh stays one plain read; the reset is a cell's alone): its heartbeat row'stenancy_modereports the derived state (open) instead of''; and its fingerprint JSON gainsservesandmarked.Compatibility, rollout and rollback
LOOM_CELL_ROUTER_URLfails the rule above now refuses to boot, where the previous build served.*-org or principal-less call for an org the cell does not hold now answers 503 while the directory saysattachingor cannot be read, where the previous build answered 409WRONG_CELL. It answersWRONG_CELLonce the cell isserving, and on a later cell.LOOM_CELLset whose Postgres sessions lose the tenancy marker (a pooler between it and Postgres) is now unready inenforcedandpartial(thecell_markerreadiness check), where the previous build served and its triggers refused its writes for unheld orgs. Before upgrading such a cell, pointLOOM_PG_DSNstraight at Postgres (troubleshooting).Tests
tests/unit/test_cell_heartbeat_keys.py: each key's value and itsnullbefore the first read, the keys kept out ofas_value(),markedfalse for an unmarked session,routerabsent withLOOM_CELLunset or on a later cell, thecell_markerreadiness check in each state, and the router URL boot check.tests/unit/test_cell_router.py: the reader's period and stop atserving, the on-demand path's single call and its lateness at non-default settings, the 401 window with transient failures between 401s, the state read moved from tenancy, and the unscoped-caller answers for each of the four directory answers.tests/unit/test_cell.py,test_cell_unset_gate.py,test_cell_tenancy.py,test_cell_cli_orgs.py: their fakes moved tofetchrow_resetand the derivedtenancy_mode.tests/unit/test_router.pyandtests/unit/test_router_attach.py: the router's state cache: one directory read per replica per window, timed from the read's end; a failed read held as a 503 for its window; the cache cleared when a credential is replaced; the entry dropped afterattach/serving, including while a read is in flight; a read begun before a change not cached.tests/integration/test_cell_heartbeat.py: the heartbeat'stenancy_modeexpected asopenunder the Postgres authority.tests/unit/conftest.py: an autouse fixture that resetscell_router's process state around every test.tests/integration/test_cell_org_triggers_pg.py: the real refresh readingmarkedon a real Postgres: true on a session with the startup marker, false after a sessionSET(whichRESET ALLdrops), false with no marker.make format lint typecheckpass. The cell, readiness, ratchet and doc unit files pass (980), and the integration filestest_cell_org_triggers_pg,test_cell_unset_gate_pg,test_cell_tenancy_pg,test_cell_cli_unset_pgandtest_cell_preparepass against a local Postgres with a synchronous standby, andtest_cell_heartbeatpasses against a server built from this branch under the Postgres authority. The full unit suite runs in CI; it is not run locally because the build machine is low on memory.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code; the design was reviewed in rounds to zero findings, and the tests above were run.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lintandmake typecheckpass, and the unit test files named above pass; the fullmake test-unitruns in CI.main; no merged migration is edited.frontend/changed.docs/describe the system as it now is, and this description names any operator steps and wire changes.git commit -s), as CONTRIBUTING.md asks.🤖 Generated with Claude Code