Repository navigation
Conversation
…are takes the same lock The attach command holds the attach lock on a direct, marked session of the cell's Postgres for its run and resumes by the directory row and the cell's mode. It refuses before any write on the stores' identity, the session, the roles, settings and subscriptions that would let an unchecked writer through, the database encoding, durability, its own grants, the ClickHouse writers, the table list and the processes' heartbeats; posts attach/begin; creates the triggers under the migration lock; in recording scans and writes the tenancy rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable, waits out every transaction that could have read recording, rescans, places every org in batches and marks the cell serving. A refusal after the mode commit leaves the cell enforced and attaching, and a rerun resumes there. Prepare refuses a session behind a pooler or a database that is not UTF8, and takes the attach lock. Doctor runs attach's Postgres reads again and reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows inside the widened window, and PRUNE_AFTER_S names the sweep's hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end docs/operations/cell-attach.md is the operator's procedure: the preconditions attach checks and those it relies on, the flags, the point of no return at the mode commit, the orgs attach does not place and the manual removal of an org past the length limit, and doctor's reads after serving. cells.md describes the first attach as it runs (the lock, the refusals, the trigger pass, the mode commit and its durable read, step 4's wait, rescan and final checks), with its stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py measures memory.set and a delivery-sized insert through an attached cell with the org triggers on and off; results.md records a local run. Attach compares the final root-secret fingerprint with the one its begin wrote, and the nonce cleanup's message names no single command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module instances.read_recent takes the window it reads back; attach's heartbeat module computes it from the live window and a stopping process's drain and retirement budgets. PRUNE_AFTER_S is marked as the contract it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown The refusals name every table the trigger pass would alter whose owner role attach's role is not a member of, before any write; ensure_table and _run_ddl answer applied, busy or denied, so a denial during the pass refuses at once rather than after --wait as a busy lock. --wait is wall time: each bounded wait charges its elapsed time, lock tries included, and every lock try waits at most what is left; scan time stays outside it. From the mode commit on, a stop that comes before the durable read says the commit's outcome is unknown and to rerun attach, for a refusal and for any other error alike. The prune-window check runs with the refusals, before any write; an enforced resume runs one trigger pass; the lock session opens inside the block that closes it; the durable read retries a timeout or an outage only; refusals name no command that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure --stray-orgs labels only a spent budget as tables left unchecked; any other refusal names its table and the read continues. A failed catalog read reports both of attach's Postgres checks as not read. The runbook and doctor describe removing a stray or over-long org by hand, and name no command that does not exist yet. cells.md labels the first attach's list with the design's step names, so step 1 to step 4 mean the same everywhere, rewrites the stuck-process failure row and the planned tests for a mode that only rises, and says --wait is wall time. The trigger-cost harness uses cell_triggers.create_statement and deletes the rows its inserts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #693
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
loom cell attachturns a prepared first cell into a serving one: it places every org thecell holds in the router's directory, raises the cell to
enforcedand creates its orgtriggers, refusing on anything that would let an unchecked writer through.
loom cell preparenow refuses a Postgres session behind a pooler or a database that is not UTF8, andloom doctorwarns on the Postgres roles, settings and subscriptions attach refuses andnames stray orgs with
--stray-orgs.Why
A first cell is prepared (mode
recording, directory rowattaching) and its processesrun the gate, the marker and the heartbeat keys, but nothing turns it into a
servingcell: the directory holds no placement for its orgs, the mode never reaches
enforced, thetriggers are never created on the first cell, and every later cell waits on the first
being
serving. This is the command that does it, with the doctor reads and the runbookits design assigns to it.
Scope and risk
server/cli/cell_attach.py(loom cell attach),cell_attach_pg.py(the Postgresside: the session checks, the attach lock, the tenancy-privilege and unmarked-session
checks doctor shares, the trigger pass, the mode commit, the durable read, step 4's
wait),
cell_attach_beats.py(the heartbeat judgment and the rescan's watch). Attachuses the org scan and the ClickHouse writer check library (
cell_scan,cell_writers)as it is, adding only the step-4 move-source order (
cell_scan.MOVE_SOURCESand thetables a view reads, scanned first).
cell_triggers.ensure_table(new) runs one table's trigger DDL alone, inenforcedor,for attach, in a first cell's
recording, with the lock timeout and the try count asarguments, so attach can run one try at a time under the migration lock and give an
autovacuum one try just past
deadlock_timeout.ensureshares its per-tablestatements (
_table_statements, with no change in what a boot runs);cell_triggers.holdersandplanned_tablesare public for attach.instances.read_recent(stopped rows inside the widened window),widened_window_s,boot_allowance_sandPRUNE_AFTER_S = 3600, which the leader's sweep now uses inplace of its literal hour. The value is a cross-build contract (attach reads stopped
rows back that far while the leader may run another build); a unit test pins it.
refuses a database that is not UTF8. A prepare that ran behind a pooler now refuses:
the break is that
LOOM_PG_DSNfor prepare must reach Postgres directly, as a cell'salready must.
cell_tenancy_privilegeandcell_unchecked_writerson every cell run (with--pg-allow-role), and--stray-orgs/--stray-orgs-budget, WARN only.enforcedandattaching, and a rerun resumes from the durable read. The point of no return is themode commit; before it the cell can be backed out by the troubleshooting page's branch.
update,place,restore, per-cell SCIM (attachrefuses with SCIM on), the doctor checks that need the router's read-only scope (tenancy
rows the directory does not place, a non-zero exit when the directory checks did not
run), and the stray-org erase command.
Compatibility, rollout and rollback
loom cell attach; newloom doctorflags--pg-allow-role,--stray-orgs,--stray-orgs-budget.docs/operations/cell-attach.md; upgrades.md step 18(rotate the ClickHouse writer credential after the
LOOM_CELLrollout, before attach).Prepare must reach Postgres directly.
placements and
enforcedmode stay; an older build serves it as an enforced cell).Tests
tests/unit/test_cell_attach.py, 40 tests on fakes; full unit suite 36,172 passed, the one failuretest_deeply_nested_hidden_html_parses_in_bounded_timefails on the base commit too on this host): every heartbeat refusal and wait, thewait budget and its one extension, the rescan's watch (swept rows, a gap, the up-front
refusal), the placement batches' count and byte bounds, the move-source order, the
router answers, the session and lock refusals, the encoding, the privilege, settings and
subscription refusals, attach's own grants and
max_prepared_transactions, step 4'srefusals, the mode commit's retry, termination and superuser refusal, the durable
read's checks, the
servingdone path.tests/integration/test_cell_attach.py, 27 tests; on this branch rebased on main, it andtest_cell_org_triggers_pg,test_cell_prepare,test_cell_unset_gate_pgandtest_cell_tenancy_pggave 92 passed; own Postgres with a synchronousstandby, own ClickHouse, the router in process): a recording first cell with orgs in
loom.tenancyalone, a Postgres table and a ClickHouse table attaches toservingwithevery org placed and the triggers clean, a malformed org skipped and named, and a rerun
exits 0; a process with
LOOM_CELLunset appearing after the mode commit refuses atonce, leaves
enforcedandattaching, and a rerun resumes toserving; a second runand a prepare are refused at the lock;
--dry-runwrites nothing; an org pastORG_MAX_BYTESrefuses before any tenancy row; each right the tenancy-privilege checkcounts (column grants,
TRIGGER,CREATE ON SCHEMA publicandloom, the parametergrants,
pg_write_server_files, an ADMIN-only membership of the owner role, a functionowner owning no table); a mixed-case
ALTER ROLE … IN DATABASE … SETrefusing andanother database's not; a disabled subscription; step 4 held by a read-committed writer
and by a repeatable-read snapshot; the session checks and a LATIN1 database; the mode
commit waiting out a
FOR SHAREholder; doctor's reads and--stray-orgs.loom router serve), two server processesof one cell (Postgres authority with a synchronous standby, auth on), an org written
before prepare and one recorded in
recordingafter theLOOM_CELLrollout.loom cell prepare(now under the attach lock) registered the cell;loom cell attach --dry-runlisted both orgs per source (the whole scan of the cell's tables took 0.7 s on that small
data);
loom cell attachreachedservingin 2.6 s with both orgs placed and 25 triggerscreated, both servers judged from their real heartbeats (
serves,marked,router).After
servinga new org was placed live by the servers, a rerun printeddone, andloom doctor --stray-orgspassed every cell check. One seam: the shared host's healthroute was asked at the first server's port, since no TLS front ran locally.
bench/cell_trigger_gate.py, recorded indocs/benchmarks/results.md): on the attached two-replica cell, three rounds with theorg triggers on and dropped in turn. The delivery-sized 1,000-row insert from a marked
session shows no cost (on minus off +1.3, -1.9, -2.1 ms on about 10 ms), within the 10%
gate.
memory.setp99 within-round differences were +214.8, +47.0, +24.3 ms in this runand -71.7, -10.0, -42.4 ms in an earlier one on a host at load 14 to 19: no consistent
sign, so the local run cannot resolve the 10 ms p99 gate; the statement-level bench
bounds the marked cost under a microsecond a row.
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code; the design note was followed
step by step and every refusal has a test.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed.docs/reference/cli.md,llms.txt, the CLI snapshots.main; no merged migration is edited. (none)docs/describe the system as it now is, and this description names any operator steps and wire changes.git commit -s).🤖 Generated with Claude Code