Skip to content

memory.redact scrubs a shared copy of the text it redacts - #697

Merged
zlareb1 merged 1 commit into
mainfrom
feat/redact-reaches-shared-copy
Nov 16, 2026
Merged

zlareb1 merged 1 commit into
mainfrom
feat/redact-reaches-shared-copy

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Related: #552, #685

Changelog category (leave one):

  • Bug Fix

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

memory.redact also scrubs a shared copy of the redacted text: a version promoted to the shared layer is no longer served by a scope='shared' search after it is redacted.

Why

memory.redact scrubbed a past version everywhere in the memory plane but left loom.shared out of scope. A version promoted with memory.promote and then edited kept its old text in its shared copy and that copy's search row, so a redacted secret stayed in every scope='shared' search of the namespace. The curation loop design (#552) wanted the redact to reach the shared copy before the term leg serves shared definitions to every reader.

Scope and risk

  • redact_version (server/memory/app/curation.py) gains _scrub_shared_copies, after the key's own derived rows are scrubbed. It reads the (memory_key, version, value) of every loom.shared version whose promoted_from names the redacted key, with the key's next shared version from any source (shared_copies, a sequential read; live, demoted or superseded versions alike), and keeps those whose value has the redacted version's content_sha256 (the hash the redact keeps) or is already "[REDACTED]". It runs two scrubs (scrub_shared_copies in server/memory/store/redaction.py), with their count checks under sequential_reads:
    • a lightweight DELETE of the shared search rows (agent='_shared', scope='shared') of each copy's key: those below the key's next shared version after its last copy, or all of them when that copy is the newest. A promotion writes a copy's vector at the copy's version, a demote and the embedding worker above it, and the rows below are older shared versions the copy replaced, which FINAL would serve once the copy's rows are gone. A later copy of the key from another source keeps its rows;
    • an ALTER TABLE loom.shared UPDATE value = '"[REDACTED]"' of those versions, matched by source and (memory_key, version), skipping a value already scrubbed. The source clause cannot narrow the pairs (they come from that source's copies, and a version is a nanosecond stamp); it stays so the statement says by itself whose rows it rewrites.
  • The search rows go first. A copy an earlier run scrubbed is reached again (its search rows, the cached answers under its key, the write mark), so a redact cut off part way is finished by the next run; a run with nothing left to rewrite issues no mutation, though every later redact of the key reaches such a copy again (its caches and the write mark).
  • A redact that reaches a copy moves the namespace's memory write mark (namespace_write_marks.bump), since a mutation moves no mark. The redact deletes cached memory.ask / code.ask answers and precedents that held a copy's key, as it already does for the redacted key (a copy promoted under target_key sits under another key).
  • A live copy stays promoted, holding "[REDACTED]", until an admin demotes it (a definition's copy counts toward the ceiling until then, and is not governed).
  • A copy is left as it is while the live head holds the redacted text too (an edit reverted byte for byte), as memory.redact leaves the live head itself.
  • Not covered: a shared copy promoted from a different identity or key with the same text (its writer did not ask); a search row the embedding worker writes for a copy whose inline embedding failed, if its pass read the stub before the purge and writes after it (the next redact run purges it); a promotion of the key racing the redaction, which loses its search row until it is promoted again (as a profile fold racing a redaction does); a demote that read the copy before the scrub, whose tombstone keeps the text until that version is redacted again; a copy a later promotion replaced and loom.shared has merged away, which is no longer found; and the knowledge_gap / query_history candidates the design names, which do not exist yet; the design page keeps that part for the PR that adds those candidate kinds.
  • Docs: the redact table in history and provenance gains the loom.shared row and the race list, the security page counts the third mutation and the shared copy's key in the cache reach, the memory-architecture op table names the read and the write, the redaction module's docstring and the curation loop design state the rule as built, and the cache_rows_deleted / precedent_rows_deleted descriptions name the shared copy's key.
  • Cost: one sequential read of loom.shared by (org, namespace) and has(promoted_from, …) per redact whose live head does not hold the text. When a copy matches: two count() reads and up to two mutations, one write-mark bump, and the cache, code-answer and precedent deletions for each copy key other than the redacted one. memory.redact is an admin call.
  • Risk: a new ALTER TABLE … UPDATE on loom.shared, classified in tests/unit/test_clickhouse_mutation_inventory.py.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none.
  • Rollout: no migration. A redact a build N-1 replica serves does not reach the shared copy, as before; running it again on N does (the design page's rolling-upgrade paragraph says so).
  • Rollback: deploy the previous build; copies already scrubbed stay scrubbed.

Tests

  • Unit (tests/unit/test_redact_shared_copies.py), with fakes: the copies whose value is the redacted text are scrubbed under the source's key and a renamed one, a copy of another text is not, a copy already scrubbed is reached again (also by a redact of a text never promoted), and the write mark moves; a live head holding the text reads and scrubs nothing; no copy rewrites nothing.
  • Unit, on clickhouse-local (same file):
    • test_another_sources_copy_of_the_key_keeps_its_row: the copy's stub, the embedding worker's vector above it and a demote's retired vector go, another source's later copy of the key keeps its row, and a row a cut-off earlier run left is purged;
    • test_a_copy_merged_away_under_a_later_promotion_is_not_found: after loom.shared merges, a replaced copy is not found (named on the history page);
    • test_an_older_shared_row_of_the_key_does_not_resurface: an older shared vector below the newest copy goes, so FINAL serves no older text, and so does the worker's fill of the copy's stub;
    • test_a_key_whose_copies_are_all_scrubbed_gets_no_mutation: a rerun with nothing left to rewrite issues no statement;
    • test_a_copy_followed_by_its_own_demote_loses_every_row: the stub, the worker's fill and the demote's retired vector all go.
  • Unit (tests/unit/test_profile_redaction.py): the handler deletes the cached answers whose window held a shared copy's key and moves the write mark; the file's ClickHouse-local schema gains loom.shared. tests/unit/test_erasure_cache_key_axis.py stubs the new read.
  • Unit (tests/unit/test_redact_scrubs_every_value_column.py): the shared scrub writes the one placeholder the others write.
  • Integration (tests/integration/test_memory_history_redact.py): through the server, a version promoted and then edited is still found by a shared search after the edit, until it is redacted; after the redact the copy's value is "[REDACTED]" and the search does not find it. With the head reverted to the redacted text, the copy keeps its text and is still found.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Claude Code.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed.
  • Generated files are regenerated in this PR, not by hand: the SDK op tables, docs/reference/configuration.md and the docs/reference/ tables (python scripts/codegen/docs_reference.py).
  • A new migration is additive and numbered after the latest on main; no merged migration is edited.
  • A behaviour change to any file a plugin ships (hooks/, lib/, bin/, skills/, .mcp.json, the Codex plugin-defaults.json) bumps both plugin manifests.
  • The frontend builds (npm --prefix frontend run build) and its tests pass, if frontend/ changed.
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers; the history goes in this description (tests/unit/test_public_copy.py).

🤖 Generated with Claude Code

A version promoted to the shared layer, edited and then redacted kept its text in
loom.shared and in that copy's search row, so a shared search still found the redacted
secret. memory.redact now reads the shared versions promoted from the key, scrubs those
holding the redacted text (or scrubbed by an earlier run) and purges the key's shared
search rows below the next shared version, all of them when the copy is newest. It
moves the namespace's write mark and deletes the cached answers and precedents under
each copy's key. A copy stays as it is while the live head holds the text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the feat/redact-reaches-shared-copy branch from 2708650 to e09b394 Compare November 16, 2026 08:06
@zlareb1
zlareb1 merged commit d246174 into main Nov 16, 2026
48 checks passed
@zlareb1
zlareb1 deleted the feat/redact-reaches-shared-copy branch November 16, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant