Repository navigation
Conversation
… its term A term leg reads the addressed namespace's shared definitions beside the other legs, under memory_term_leg_timeout_ms, matches each term and alias against the query by the script of its edges and the case of each word, and checks the matched definitions' sources. Governed hits join after the verdict and every floor, lead the served list (after a computed row), replace other hits under their key and carry governed: true; term_leg says what the leg did. loom_search, context.pack and memory.ask keep them first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… latest promotion Each name is prepared once and skipped unless its folded text is in the folded query, matching yields every 50 definitions, and definitions that share a name with a matched one are checked so the latest promotion holds it. The leg launches after admission, under the search's wall time, with its vector read inside the same deadline. The reader block's date order keeps governed hits first. Default bound 200 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…mory.ask keeps governed hits first The search path reads each definition's term and aliases, then full values and sources only for the named definitions and those sharing a name, all without sequential consistency. memory.ask puts governed hits first again after its reranks and injections and keeps them out of the similarity knee. A user-bounded reader is served the governed definition and no other shared row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ly reorders The excerpt is built from the stored value, so text in any script reads as written. The ask pool reorders the governed hits it still holds and the similarity knee sets them aside; one shared-definitions reader serves both promote and the leg; the served counter counts what was served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… every governed hit memory.promote stores a value with its non-ASCII characters escaped, so the excerpt is built from the decoded value, and the names matched come from that same value. The ask pool partitions by the governed flag, so a hit a sub-search served leads too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The query vector is ready at admission, so the leg reads its hits' shared embeddings before it returns and the whole leg sits under memory_term_leg_timeout_ms; LegTask and its second deadline go. A copy is judged before its names are prepared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #552, #679, #685
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
memory.searchserves a governed definition first when the query names its term or an alias, markedgoverned: true, and reports what its term leg did interm_leg. It runs insidememory.ask,context.packandloom_searchtoo.Why
An admin who promotes a definition into the shared layer wants every reader who asks about the term to get that definition, not whatever vector search ranks highest. Before this change a governed definition competed like any other row: a query such as "what does NPS mean for the team" abstained although the namespace held the admin's definition of NPS. The curation loop design (#552) specifies a term leg for this; the read-time governance check (#679) and the withheld-definition stage (#685) are its prerequisites.
Scope and risk
server/memory/domain/definitions.pygains the matcher (pure). Each name is prepared once (Name.of), and a name whose case-folded text is not a substring of the case-folded query is skipped before the per-position edge check: the query and each name lose format characters and go through NFKC with whitespace collapsed; combining marks join the character before them; a name matches at whole characters with no neighbour that continues its edge, by the edge's script (Latin letters, digits,_,+,#; other spaced scripts; Hangul, whose particles follow; unspaced scripts and punctuation); case is decided per word (all capitals, or a capital after the first letter, match only as written). A fold that changes a character's length (ßtoss) matches only as written.server/memory/app/search/term_leg.py: launched once the search is admitted (_admit, so its reads are sequential when the Postgres barrier waited), as a deferred task bounded bymemory_term_leg_timeout_ms(new setting, default 200 ms,memory.search's p99 budget) and the search's wall time left. It scores its hits against the search's query vector (already embedded at admission) inside that bound. Matching yields to the loop every 50 definitions. It reads the names of the addressed namespace's live shared definitions (shared.shared_definitions, the readermemory.promote's name check already used, now with the promotion fields, newest first and astrongkeyword: key, term, aliases, extracted in ClickHouse, at mostmemory_shared_definitions_maxplus one), matches every name, then reads the full values (shared.definition_values) and reads and grades the sources (source_reasons, extracted frommemory.definitions.listso both share one check) of the matched ones and those that share a name with one. The search path reads without sequential consistency, unlike the admin list: a promotion on another replica a moment ago may miss a search, and the write mark already covers stored answers. Names a definition loses to another governed one are not matched; the definitions that share a name with a matched one are checked too, so the latest promotion holds a name even when the query's case matches only an older one (or the winner is excluded), asmemory.definitions.listreports it. Matched definitions are ordered by where their names first appear, the longer name first at one start, then the latest promotion.mergeputs them after a computed row, removes any other hit under the same key in the addressed namespace, counts them withintop_k(cut last), and moves the ordinary hits they displace to the head of the parked tail. A response that serves one hasabstainfalse;retrieval_confidenceandretrieval_riskdescribe the other hits. Similarity is the shared embedding's (live_vectors, cosine fromspans.cosine), 0 without one;rank_scoreis above every hit's but a computed row's. The excerpt is built from the decoded value (memory.promotestores the value with every non-ASCII character escaped), so text in any script reads as written; the names matched are taken from that same value.memory_type, hardtags/category,as_of,min_body_chars,recent_first/anchor_first,legswithoutmemory, and a query that is not valid Unicode.exclude_keysdrops keys.loom_search's answer-first stages take governed hits out and put them back first (they count within the tracker verdict's memory cap, cut last); the repository-decision merge never places a decision above one;context.packgives each governed hit its own evidence group, so the per-group cap never moves one down;memory.askputs the governed hits it holds first again, from whichever of its searches served them, once its reranks and injections have filled the pool (governed_firstpartitions by the flag and only reorders: a stage that dropped one decided so), sets them aside from the similarity knee, and its restated-value check neither retires nor is retired by one (its stale-sentence masks leave fact rows alone already); the reader block's date order (evidence_order, forrender="reader",loom_searchandmemory.ask) keeps governed hits first._verdictin the handler joins the leg after the calibration and_admitlaunches it once admitted; both keephandle_memory_searchwithin the function-length gate. Recall credit is unchanged: it is taken before the noise floor, as onmain, so governed hits, which join after it, are not credited.memory.searchorloom_searchthat serves a governed hit, or whose leg did not reportcompleteorempty(offincluded), records no knowledge-gap miss.scope="private". These are the two exceptions the design names;read_scope.py's docstring and theSCOPED_OPScomment, and thescopefield's description say so.loom.sharedper search (the ceiling bounds it), and for a query that names a term, one source read, the grading of each matched source and one vector read. Matching 500 definitions of three names each against a 2000-character query takes 25 ms of CPU, in batches. On one local server (where sequential consistency would cost nothing; the search path no longer asks for it) with 500 promoted definitions, each with an 800-character body andrests_on(40 searches each): with a 2000 ms bound, a query naming a term p50 90 ms / p95 125 ms and one naming none p50 88 / p95 91 ms; with a 150 ms bound, 2 of 40 term searches timed out (p95 167 ms), hence 200 ms rather than the design page's 50, at which every term search timed out. With the names-only read, at the 200 ms bound: a term search p50 91 / p95 126 ms with no timeout, a search naming no term p50 91 / p95 98 ms.Ranking-neutral evidence
Free, fake providers only. The same corpus and the same three promoted definitions were written to two local servers on one ClickHouse, one built from
mainplus #697, which does not touch search (no term leg) and one from this branch, and every query was searched on both:bench/datasets/synthetic_memory_qa.jsonplus 5 near-misses such as "arrival time of the order" and "it was late"): served keys, order, similarities,abstainandretrieval_riskidentical on both.bench/quality.pyon both servers: identical per-question CSV (MRR 0.933, precision@10 0.653, gold ranked first 0.867).Compatibility, rollout and rollback
governed: trueonly when set; responses carryterm_leg. The SDKs and generated references are regenerated.memory_term_leg_timeout_msis live.Reproducing the evidence
The scripts are not committed: two servers from this branch and from
mainon one ClickHouse with fake providers, each fed the same corpus, three promoted definitions and the same queries, compare(memory_key, similarity)per query, plusbench/quality.py --base-url <each server>.Tests
tests/unit/test_definition_term_match.py(the matching table per script, marks, case per word, start position);tests/unit/test_term_leg.py(only named definitions' sources are read, a name held by a newer definition the query's case does not match,join's placement, scores and verdict, scores from the shared embedding, names matched from the served value, a launch with no wall time left, the ask pool's order, the reader block's date order, the restated-value check, query order, lapsed and excluded definitions, a name two hold, a copy not from its namespace-wide row, capped, the fields that turn the leg off and those that keep it on, timeout and failure, merge placement, same-key removal, top_k cut, wire shape);tests/unit/test_knowledge_gaps.py(no miss when a governed hit is served or the leg did not finish);tests/unit/test_context_pack_decisions.py(four governed definitions lead the block past the per-group cap). Thecontext.pack, reader-order and restated-value tests fail with their fix removed.tests/integration/test_term_leg.pythrough the server: a definition in Japanese is served first with its text as written; a reader pinned to one user is served the governed definition and no other shared or namespace-wide row (which a whole-namespace reader is served);emptybefore any definition; the governed definition first and once after its promotion inmemory.search,search.query,context.packandmemory.ask; no governed hit for a query naming no term orarrin lower case; none after the source is edited.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed.docs/reference/configuration.mdand thedocs/reference/tables (python scripts/codegen/docs_reference.py).main; no merged migration is edited.hooks/,lib/,bin/,skills/,.mcp.json, the Codexplugin-defaults.json) bumps both plugin manifests.npm --prefix frontend run build) and its tests pass, iffrontend/changed.docs/describe the system as it now is, and this description names any operator steps and wire changes.tests/unit/test_public_copy.py).🤖 Generated with Claude Code