Skip to content

loom cell attach turns a prepared first cell into a serving one, placing its orgs and creating its triggers - #735

Open
zlareb1 wants to merge 7 commits into
mainfrom
feat/cell-attach-cli
Open

zlareb1 wants to merge 7 commits into
mainfrom
feat/cell-attach-cli

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 16, 2026

Copy link
Copy Markdown
Member

Related: #693

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

loom cell attach turns a prepared first cell into a serving one: it places every org the
cell holds in the router's directory, raises the cell to enforced and creates its org
triggers, refusing on anything that would let an unchecked writer through. loom cell prepare now refuses a Postgres session behind a pooler or a database that is not UTF8, and
loom doctor warns on the Postgres roles, settings and subscriptions attach refuses and
names stray orgs with --stray-orgs.

Why

A first cell is prepared (mode recording, directory row attaching) and its processes
run the gate, the marker and the heartbeat keys, but nothing turns it into a serving
cell: the directory holds no placement for its orgs, the mode never reaches enforced, the
triggers are never created on the first cell, and every later cell waits on the first
being serving. This is the command that does it, with the doctor reads and the runbook
its design assigns to it.

Scope and risk

  • server/cli/cell_attach.py (loom cell attach), cell_attach_pg.py (the Postgres
    side: the session checks, the attach lock, the tenancy-privilege and unmarked-session
    checks doctor shares, the trigger pass, the mode commit, the durable read, step 4's
    wait), cell_attach_beats.py (the heartbeat judgment and the rescan's watch). Attach
    uses the org scan and the ClickHouse writer check library (cell_scan, cell_writers)
    as it is, adding only the step-4 move-source order (cell_scan.MOVE_SOURCES and the
    tables a view reads, scanned first).
  • cell_triggers.ensure_table (new) runs one table's trigger DDL alone, in enforced or,
    for attach, in a first cell's recording, with the lock timeout and the try count as
    arguments, so attach can run one try at a time under the migration lock and give an
    autovacuum one try just past deadlock_timeout. ensure shares its per-table
    statements (_table_statements, with no change in what a boot runs);
    cell_triggers.holders and planned_tables are public for attach.
  • instances.read_recent (stopped rows inside the widened window), widened_window_s,
    boot_allowance_s and PRUNE_AFTER_S = 3600, which the leader's sweep now uses in
    place of its literal hour. The value is a cross-build contract (attach reads stopped
    rows back that far while the leader may run another build); a unit test pins it.
  • Prepare holds the attach lock for its run after the same direct-session check, and
    refuses a database that is not UTF8. A prepare that ran behind a pooler now refuses:
    the break is that LOOM_PG_DSN for prepare must reach Postgres directly, as a cell's
    already must.
  • Doctor: cell_tenancy_privilege and cell_unchecked_writers on every cell run (with
    --pg-allow-role), and --stray-orgs / --stray-orgs-budget, WARN only.
  • Mode only rises: every refusal after the mode commit leaves the cell enforced and
    attaching, and a rerun resumes from the durable read. The point of no return is the
    mode commit; before it the cell can be backed out by the troubleshooting page's branch.
  • Not in this PR: later-cell attach, update, place, restore, per-cell SCIM (attach
    refuses with SCIM on), the doctor checks that need the router's read-only scope (tenancy
    rows the directory does not place, a non-zero exit when the directory checks did not
    run), and the stray-org erase command.

Compatibility, rollout and rollback

  • Wire: none. New CLI command loom cell attach; new loom doctor flags
    --pg-allow-role, --stray-orgs, --stray-orgs-budget.
  • Operator steps: the runbook docs/operations/cell-attach.md; upgrades.md step 18
    (rotate the ClickHouse writer credential after the LOOM_CELL rollout, before attach).
    Prepare must reach Postgres directly.
  • Rollback: reverting leaves an attached cell attached (its triggers, tenancy rows,
    placements and enforced mode stay; an older build serves it as an enforced cell).

Tests

  • Unit (tests/unit/test_cell_attach.py, 40 tests on fakes; full unit suite 36,172 passed, the one failure test_deeply_nested_hidden_html_parses_in_bounded_time fails on the base commit too on this host): every heartbeat refusal and wait, the
    wait budget and its one extension, the rescan's watch (swept rows, a gap, the up-front
    refusal), the placement batches' count and byte bounds, the move-source order, the
    router answers, the session and lock refusals, the encoding, the privilege, settings and
    subscription refusals, attach's own grants and max_prepared_transactions, step 4's
    refusals, the mode commit's retry, termination and superuser refusal, the durable
    read's checks, the serving done path.
  • Integration (tests/integration/test_cell_attach.py, 27 tests; on this branch rebased on main, it and test_cell_org_triggers_pg, test_cell_prepare, test_cell_unset_gate_pg and test_cell_tenancy_pg gave 92 passed; own Postgres with a synchronous
    standby, own ClickHouse, the router in process): a recording first cell with orgs in
    loom.tenancy alone, a Postgres table and a ClickHouse table attaches to serving with
    every org placed and the triggers clean, a malformed org skipped and named, and a rerun
    exits 0; a process with LOOM_CELL unset appearing after the mode commit refuses at
    once, leaves enforced and attaching, and a rerun resumes to serving; a second run
    and a prepare are refused at the lock; --dry-run writes nothing; an org past
    ORG_MAX_BYTES refuses before any tenancy row; each right the tenancy-privilege check
    counts (column grants, TRIGGER, CREATE ON SCHEMA public and loom, the parameter
    grants, pg_write_server_files, an ADMIN-only membership of the owner role, a function
    owner owning no table); a mixed-case ALTER ROLE … IN DATABASE … SET refusing and
    another database's not; a disabled subscription; step 4 held by a read-committed writer
    and by a repeatable-read snapshot; the session checks and a LATIN1 database; the mode
    commit waiting out a FOR SHARE holder; doctor's reads and --stray-orgs.
  • Two-replica run, on own containers: a router (loom router serve), two server processes
    of one cell (Postgres authority with a synchronous standby, auth on), an org written
    before prepare and one recorded in recording after the LOOM_CELL rollout. loom cell prepare (now under the attach lock) registered the cell; loom cell attach --dry-run
    listed both orgs per source (the whole scan of the cell's tables took 0.7 s on that small
    data); loom cell attach reached serving in 2.6 s with both orgs placed and 25 triggers
    created, both servers judged from their real heartbeats (serves, marked, router).
    After serving a new org was placed live by the servers, a rerun printed done, and
    loom doctor --stray-orgs passed every cell check. One seam: the shared host's health
    route was asked at the first server's port, since no TLS front ran locally.
  • Trigger cost, local (bench/cell_trigger_gate.py, recorded in
    docs/benchmarks/results.md): on the attached two-replica cell, three rounds with the
    org triggers on and dropped in turn. The delivery-sized 1,000-row insert from a marked
    session shows no cost (on minus off +1.3, -1.9, -2.1 ms on about 10 ms), within the 10%
    gate. memory.set p99 within-round differences were +214.8, +47.0, +24.3 ms in this run
    and -71.7, -10.0, -42.4 ms in an earlier one on a host at load 14 to 19: no consistent
    sign, so the local run cannot resolve the 10 ms p99 gate; the statement-level bench
    bounds the marked cost under a microsecond a row.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Claude Code; the design note was followed
    step by step and every refusal has a test.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed.
  • Generated files are regenerated in this PR, not by hand: docs/reference/cli.md, llms.txt, the CLI snapshots.
  • A new migration is additive and numbered after the latest on main; no merged migration is edited. (none)
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers.
  • Commits are signed off (git commit -s).

🤖 Generated with Claude Code

zlareb1 and others added 7 commits November 16, 2026 07:48
…are takes the same lock

The attach command holds the attach lock on a direct, marked session of the
cell's Postgres for its run and resumes by the directory row and the cell's
mode. It refuses before any write on the stores' identity, the session, the
roles, settings and subscriptions that would let an unchecked writer through,
the database encoding, durability, its own grants, the ClickHouse writers,
the table list and the processes' heartbeats; posts attach/begin; creates the
triggers under the migration lock; in recording scans and writes the tenancy
rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable,
waits out every transaction that could have read recording, rescans, places
every org in batches and marks the cell serving. A refusal after the mode
commit leaves the cell enforced and attaching, and a rerun resumes there.

Prepare refuses a session behind a pooler or a database that is not UTF8,
and takes the attach lock. Doctor runs attach's Postgres reads again and
reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows
inside the widened window, and PRUNE_AFTER_S names the sweep's hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end

docs/operations/cell-attach.md is the operator's procedure: the preconditions
attach checks and those it relies on, the flags, the point of no return at the
mode commit, the orgs attach does not place and the manual removal of an org
past the length limit, and doctor's reads after serving. cells.md describes
the first attach as it runs (the lock, the refusals, the trigger pass, the mode
commit and its durable read, step 4's wait, rescan and final checks), with its
stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse
writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py
measures memory.set and a delivery-sized insert through an attached cell with
the org triggers on and off; results.md records a local run.

Attach compares the final root-secret fingerprint with the one its begin
wrote, and the nonce cleanup's message names no single command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module

instances.read_recent takes the window it reads back; attach's heartbeat
module computes it from the live window and a stopping process's drain and
retirement budgets. PRUNE_AFTER_S is marked as the contract it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown

The refusals name every table the trigger pass would alter whose owner role
attach's role is not a member of, before any write; ensure_table and
_run_ddl answer applied, busy or denied, so a denial during the pass refuses
at once rather than after --wait as a busy lock.

--wait is wall time: each bounded wait charges its elapsed time, lock tries
included, and every lock try waits at most what is left; scan time stays
outside it. From the mode commit on, a stop that comes before the durable read
says the commit's outcome is unknown and to rerun attach, for a refusal and
for any other error alike.

The prune-window check runs with the refusals, before any write; an enforced
resume runs one trigger pass; the lock session opens inside the block that
closes it; the durable read retries a timeout or an outage only; refusals
name no command that does not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure

--stray-orgs labels only a spent budget as tables left unchecked; any other
refusal names its table and the read continues. A failed catalog read reports
both of attach's Postgres checks as not read. The runbook and doctor describe
removing a stray or over-long org by hand, and name no command that does not
exist yet.

cells.md labels the first attach's list with the design's step names, so step 1
to step 4 mean the same everywhere, rewrites the stuck-process failure row and
the planned tests for a mode that only rises, and says --wait is wall time. The
trigger-cost harness uses cell_triggers.create_statement and deletes the rows
its inserts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant