Skip to content

Mirrored sources sync first in their own share of a pass, with their state under the mirror and a cursor for their lists - #732

Open
zlareb1 wants to merge 4 commits into
mainfrom
feat/mirror-pass-schedule
Open

zlareb1 wants to merge 4 commits into
mainfrom
feat/mirror-pass-schedule

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Not for changelog

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

None: no request can add a mirrored source yet, so nothing a caller sees changes.

Why

The document sources design (docs/architecture/document-sources.md, Every complete listing reads the lists again; The mirror) gives the mirrored sources their own share of a sync pass. Their listings read every file's permission list, so they run first, in at most a third of the pass and under their own document bound, and the namespace sources' progress says nothing about a mirror's size. A source with more files than one turn can read goes on from a cursor, and its listing row keeps the start of the turn that began the cycle. A mirrored source's sync state is filed under its mirror and source id, and what its turn spends is the mirror's, so nothing derived from a mirror's documents sits under the namespace. The previous step (the mirrored turn) listed each of these as required before the op that adds a mirrored source.

Known limit, decided: one listing walk (metadata only, a thousand files a page) must fit in the mirrors' third of the pass; a walk that runs out writes nothing and fails closed, and the listing has no cursor of its own. The implementing step of the ops measures the walk for a large source.

Staging, decided: the removed mirrored source's copies retired by tag, the mirror's own audit rows and a person's erasure scrubbing their email from every list go in the next step; group members are read with the access check.

Scope and risk

  • run_sync (server/knowledge/app/source_sync.py):
    • After an admin's delete requests, the mirrored sources run first (_sync_mirrored) under one Pass of their own: until a third of the pass's seconds after the pass's start (_leg, never past the pass's end), so time an admin's delete requests took comes out of the mirrors' third, never the rest and LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS documents (new, default 40, live).
    • Order: the one synced longest ago first, so every mirrored source gets turns however many there are; no cursor slot is needed. A sync request takes its place in that rotation rather than going first: a source too large for one turn ends every turn incomplete and keeps its request, and going first on it would starve the rest.
    • The namespace sources then keep the pass's own budget, its start and its end, so they have two thirds of it or more and the extraction leg's read deadline still counts from the pass's start. Their document bound is unchanged and no longer shared with mirrors, so the documents they write say nothing about a mirror's size; when they start does, which is timing, out of scope in the design's threat model (the design page now says so in place of a fixed start point).
  • State (_write_state, _write_mirrored_state, mirror_state):
    • A mirrored source's namespace row keeps its sync request (requested_at, full), status and last sync time only, and a turn drops any other field from it. That row is what knowledge.source.list and inspect read, so they show no count, reason, error or file id even without sync_shown.
    • Everything else (waits, counts, reasons, error, full_listed_at, the lists cursor) is written under (mirror, source id) in the same state table. A source added again under the name has a new id and inherits nothing; the namespace's erasure already purges the table's rows of every mirror (code_index.purge over the mirror-expanded scope).
    • A turn that failed before it knew its mirror, or that raised (its result is then a fresh one), writes the namespace codes only; the mirror row keeps what the last turn that returned recorded.
  • Spend: once the turn knows its mirror, it runs under customer.for_org(org, namespace=mirror) and the worker identity names the mirror, so the fetches' and embeddings' spend rows are the mirror's. No token reads an acl- name, and org-wide cost reads scope through resolve_org_scope, which leaves the mirrors out even for an org admin, so no cost read shows it.
  • The lists cursor (source_sync_mirror._Cycle):
    • Stored copies are read from the cursor's address on, then those before it. The first copy whose list the deadline left unread is the next cursor.
    • A turn that read every list lands its listing row with its own start. One that stopped past the wrap finishes the cycle: its row carries lists_since, the start of the turn that began the cycle, and a new cycle begins at the stopping point with this turn's start. One that stopped before the wrap lands no row and moves the cursor on.
    • A turn that vouches for nothing (a listing that did not search everything, a list it could not write, a refused account, a publication owed at the read or at the end) leaves the cycle as it was.
    • A publication owed when the copies are read moves no cursor: that head was not among the copies, so its list was not read in the cycle.
    • The safety argument: each list in a cycle was read at or after lists_since; a copy written during the cycle carries the list read with its write; a copy the whole listing does not have reads unread on every turn, needing no read.
  • _record_turn writes a turn's record for both the namespace and the mirror row.
  • Docs: the design page's pass-schedule sentence and its threat model's out-of-scope list (when a pass reaches the namespace sources); rolling-upgrade rule 55; rule 54 loses its sentence about the shared document bound; both module docstrings; the configuration reference, the settings schema and the settings template are regenerated.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none. The new setting defaults to the namespace sources' value.
  • Rolling-upgrade rule 55: N-1 runs a mirrored source under the whole pass's bounds and keeps its whole state in the namespace row; N reads neither N-1's fields there nor N-1 the mirror row, so a mixed fleet reads some lists again and retries some waits, and the next N turn drops N-1's fields from the namespace row. No request can add a mirrored source yet.
  • Rollback: the mirror rows of the state table stay unread; N-1 writes its state in the namespace row again.

Tests

  • tests/unit/test_knowledge_source_mirror_pass.py:
    • the mirrored sources run first under their own document bound, and the namespace sources still write their full bound;
    • a leg ends at its share of the pass counted from the pass's start, and never past its end, and the namespace sources keep the pass's own budget and read deadline after the mirrors;
    • the mirrored source synced longest ago goes first, even past a request;
    • a publication owed at the read moves no cursor;
    • a mirrored source's namespace row holds only its codes, and its counts and error are under the mirror;
    • a source added again under the name inherits no given-up wait;
    • a mirrored turn's fetch is attributed to the mirror;
    • the lists go on from the cursor across turns, and the cycle's row lands past the wrap with the start of the turn that began it;
    • a cycle begun at a wrap keeps its start until its own wrap;
    • a publication parked by the end keeps the cycle, and a turn that reads every list mid-cycle lands its own start and ends the cycle;
    • a turn that vouches for nothing keeps the cycle, and one that fails before it knows its mirror leaves the mirror's state as it was.
  • tests/unit/test_knowledge_source_sync_mirror.py reads the turn's state from under the mirror, and its document-bound test uses the mirrors' setting.
  • make format-check lint typecheck pass, and the full unit suite locally.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Claude Code; checked against the design page and the mirrored turn.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed (the full unit and integration suites run in CI).
  • Generated files are regenerated in this PR, not by hand: the SDK op tables, docs/reference/configuration.md and the docs/reference/ tables (python scripts/codegen/docs_reference.py), the reach manifest, schemas/schema.sql (make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).
  • A new migration is additive and numbered after the latest on main; no merged migration is edited.
  • A behaviour change to any file a plugin ships (hooks/, lib/, bin/, skills/, .mcp.json, the Codex plugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.
  • The frontend builds (npm --prefix frontend run build) and its tests pass, if frontend/ changed.
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers; the history goes in this description (tests/unit/test_public_copy.py).
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks.

🤖 Generated with Claude Code

zlareb1 and others added 4 commits November 16, 2026 07:26
…ir state under the mirror and read their lists on from a cursor

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nd a turn's record is written by one helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…uest waits its turn among the mirrored sources

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…reat model say what the code does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant