Repository navigation
Conversation
…ir state under the mirror and read their lists on from a cursor Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nd a turn's record is written by one helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…uest waits its turn among the mirrored sources Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…reat model say what the code does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
None: no request can add a mirrored source yet, so nothing a caller sees changes.
Why
The document sources design (docs/architecture/document-sources.md, Every complete listing reads the lists again; The mirror) gives the mirrored sources their own share of a sync pass. Their listings read every file's permission list, so they run first, in at most a third of the pass and under their own document bound, and the namespace sources' progress says nothing about a mirror's size. A source with more files than one turn can read goes on from a cursor, and its listing row keeps the start of the turn that began the cycle. A mirrored source's sync state is filed under its mirror and source id, and what its turn spends is the mirror's, so nothing derived from a mirror's documents sits under the namespace. The previous step (the mirrored turn) listed each of these as required before the op that adds a mirrored source.
Known limit, decided: one listing walk (metadata only, a thousand files a page) must fit in the mirrors' third of the pass; a walk that runs out writes nothing and fails closed, and the listing has no cursor of its own. The implementing step of the ops measures the walk for a large source.
Staging, decided: the removed mirrored source's copies retired by tag, the mirror's own audit rows and a person's erasure scrubbing their email from every list go in the next step; group members are read with the access check.
Scope and risk
run_sync(server/knowledge/app/source_sync.py):_sync_mirrored) under onePassof their own: until a third of the pass's seconds after the pass's start (_leg, never past the pass's end), so time an admin's delete requests took comes out of the mirrors' third, never the rest andLOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASSdocuments (new, default 40, live)._write_state,_write_mirrored_state,mirror_state):requested_at,full), status and last sync time only, and a turn drops any other field from it. That row is whatknowledge.source.listandinspectread, so they show no count, reason, error or file id even withoutsync_shown.full_listed_at, the lists cursor) is written under(mirror, source id)in the same state table. A source added again under the name has a new id and inherits nothing; the namespace's erasure already purges the table's rows of every mirror (code_index.purgeover the mirror-expanded scope).customer.for_org(org, namespace=mirror)and the worker identity names the mirror, so the fetches' and embeddings' spend rows are the mirror's. No token reads anacl-name, and org-wide cost reads scope throughresolve_org_scope, which leaves the mirrors out even for an org admin, so no cost read shows it.source_sync_mirror._Cycle):lists_since, the start of the turn that began the cycle, and a new cycle begins at the stopping point with this turn's start. One that stopped before the wrap lands no row and moves the cursor on.lists_since; a copy written during the cycle carries the list read with its write; a copy the whole listing does not have readsunreadon every turn, needing no read._record_turnwrites a turn's record for both the namespace and the mirror row.Compatibility, rollout and rollback
Tests
tests/unit/test_knowledge_source_mirror_pass.py:tests/unit/test_knowledge_source_sync_mirror.pyreads the turn's state from under the mirror, and its document-bound test uses the mirrors' setting.make format-check lint typecheckpass, and the full unit suite locally.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code; checked against the design page and the mirrored turn.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed (the full unit and integration suites run in CI).docs/reference/configuration.mdand thedocs/reference/tables (python scripts/codegen/docs_reference.py), the reach manifest,schemas/schema.sql(make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).main; no merged migration is edited.hooks/,lib/,bin/,skills/,.mcp.json, the Codexplugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.npm --prefix frontend run build) and its tests pass, iffrontend/changed.docs/describe the system as it now is, and this description names any operator steps and wire changes.tests/unit/test_public_copy.py).git commit -s), as CONTRIBUTING.md asks.🤖 Generated with Claude Code