Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/architecture/memory-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1250,13 +1250,13 @@ Table-level retention and tiering are operator TTLs, off by default

## History, snapshot, restore and rollback

| Operation | Reads | Writes |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `memory.history` | `memory_versions` for one key | Nothing. |
| `memory.redact` | The row of org-wide `(memory_id, version)` (`loom.memory_revisions` under the Postgres authority), then the later versions of its key | Scrubs one historical version in place, the later versions of its key that copy its text (the live head left alone), and every profile version whose trace names the memory, live heads included. Refuses the memory's own live head (delete it first). |
| `namespace.snapshot` | Live `memory` rows, type configs, curated rows | A portable payload with `version`, `content_sha256` and the attestation columns, plus a `loom.namespace_snapshots` audit row. |
| `namespace.restore` | A snapshot payload | Rows into a target namespace (created if missing; the caller becomes admin). Idempotent per `snapshot_id`. Never strips `verified_by`. |
| `namespace.rollback` | `memory` heads and `memory_versions` at `as_of` | Each changed key's older version as a new head; keys absent at `as_of` tombstoned; embedding stubs for the worker. Dry run by default. |
| Operation | Reads | Writes |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `memory.history` | `memory_versions` for one key | Nothing. |
| `memory.redact` | The row of org-wide `(memory_id, version)` (`loom.memory_revisions` under the Postgres authority), then the later versions of its key and the `loom.shared` versions promoted from it | Scrubs one historical version in place, the later versions of its key that copy its text (the live head left alone), every profile version whose trace names the memory, live heads included, and the shared copies promoted from the key that hold the text, with their search rows. Refuses the memory's own live head (delete it first). |
| `namespace.snapshot` | Live `memory` rows, type configs, curated rows | A portable payload with `version`, `content_sha256` and the attestation columns, plus a `loom.namespace_snapshots` audit row. |
| `namespace.restore` | A snapshot payload | Rows into a target namespace (created if missing; the caller becomes admin). Idempotent per `snapshot_id`. Never strips `verified_by`. |
| `namespace.rollback` | `memory` heads and `memory_versions` at `as_of` | Each changed key's older version as a new head; keys absent at `as_of` tombstoned; embedding stubs for the worker. Dry run by default. |

Rollback restores heads, not exact state. `version` is a wall-clock stamp, so the head
at T is the largest version at or before T. Triples, records, summaries and aggregates
Expand Down
36 changes: 18 additions & 18 deletions docs/architecture/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -1353,24 +1353,24 @@ A redaction (`memory.redact`) scrubs the version it names, that key's derived ro
every version of the user's profiles whose trace names the memory: the structured profile
and the free-text fold's `consolidated_profile` name the memories they read in their trace
(`profile.fold:<label>:<memory ids>`), and each version whose trace names the redacted
memory is scrubbed with its key's derived rows, the live one included. The trace names
the memory and not its version, so a profile built from another version of it is scrubbed
too. The named version and every profile version reached are scrubbed in one statement
per version table, so the redaction issues two mutations however many profile versions
name the memory. The cached
answers and ask precedents whose window held a scrubbed profile key are deleted with those
of the memory's own key. A window records bare keys, and every user's profile has the same
key, so this reach is org-wide: one redaction clears every cached answer and precedent in
the org that rendered any user's profile, in every namespace, and each profile key reached
adds one deletion per table. A profile version whose trace names no memory ids
(`profile.fold:<label>` alone, as a build that does not record its sources writes it,
including an older build during a rolling upgrade) is not reached and keeps the text until
the user is erased. A lock on a profile key reached refuses the redaction
before anything is scrubbed, as a lock on the memory's own key does. A scrubbed live
profile is left out of `profile.render`, `profile.structured` and profile search, and so
of every pack and primer, until the worker rebuilds it from the user's current facts; the
fold rebuilds `consolidated_profile` only on its own schedule, and never for a user below
its minimum memory count or in a namespace that has turned the fold off.
memory is scrubbed with its key's derived rows, the live one included. The trace names the
memory and not its version, so a profile built from another version of it is scrubbed too.
The named version and every profile version reached are scrubbed in one statement per
version table, so the redaction issues two mutations however many profile versions name the
memory, and a third when a shared copy promoted from the key holds the text. The cached
answers and ask precedents whose window held a scrubbed profile key or a shared copy's key
are deleted with those of the memory's own key. A window records bare keys, and every user's
profile has the same key, so this reach is org-wide: one redaction clears every cached
answer and precedent in the org that rendered any user's profile, in every namespace, and
each profile key reached adds one deletion per table. A profile version whose trace names no
memory ids (`profile.fold:<label>` alone, as a build that does not record its sources writes
it, including an older build during a rolling upgrade) is not reached and keeps the text
until the user is erased. A lock on a profile key reached refuses the redaction before
anything is scrubbed, as a lock on the memory's own key does. A scrubbed live profile is
left out of `profile.render`, `profile.structured` and profile search, and so of every pack
and primer, until the worker rebuilds it from the user's current facts; the fold rebuilds
`consolidated_profile` only on its own schedule, and never for a user below its minimum
memory count or in a namespace that has turned the fold off.

A structured profile is served only from a row written as `worker:structured-profile`, an
actor kind no token maps to, so a caller cannot forge one
Expand Down
90 changes: 49 additions & 41 deletions docs/development/curation-loop-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,26 +107,35 @@ Namespace erasure reaches `loom.shared`: its `shared.tombstone` step takes every
row of the namespace and its census counts them, beside the user erasure's match by
provenance.

A redact of a past version of a source (`memory.redact` refuses the live head) reaches its
shared copies: every `loom.shared` version promoted from the key whose `value` is the
version's text (by the `content_sha256` the redact keeps) is scrubbed, and the key's search
rows below the next shared version after it (all of them when the copy is the newest) are
deleted, older shared rows it replaced included. A redact that reaches a copy marks the
namespace's memory plane (`namespace_write_marks.bump`), since a mutation moves no mark. A
copy is left as it is while the live head holds the redacted text too (an edit reverted byte
for byte), as `memory.redact` leaves a live copy of a text it scrubs from history. A live
copy keeps its row, holding `"[REDACTED]"`, until an admin demotes it; a definition's copy
counts toward the ceiling until then, and the read-time check does not govern it.

A copy an earlier redact scrubbed is reached again: a run cut off part way is finished by
the next one, and each later redact of the key purges its search rows and cached answers
again and moves the mark.

Writes that race the redaction are listed on [the history
page](../features/memory/history-and-provenance.md#scrubbing-history-memoryredact).

The shared layer needs these changes still, each in the pull request that adds the term
leg except where a change names another, because the term leg serves shared rows to every
reader:

- **A redact** of a past version of a definition's source (`memory.redact` refuses the
live head) reaches two things, and no more:
- the shared row whose `value` has the version's served hash (read from the version
before the scrub, or, when an older build already scrubbed it, the `content_sha256`
the redact keeps, so a redact run again still finds it): its text is scrubbed and its
embedding retired, which a redact does not reach today. The redact then marks the
namespace's memory plane (`namespace_write_marks.bump`), since a mutation moves no
mark;
- every `knowledge_gap` and `query_history` candidate whose drafted value has that hash,
or that the version's `candidate-id:` tag names, promoted or not: its text is blanked,
since `learnings.list` shows it to admins, and a blanked candidate is not promoted
(`learnings.review` answers it not found, as it does a blanked `assistant_turn`
candidate today); the pull request that adds those candidate kinds adds this part;
- nothing else: no other draft of the term, and neither the shared copy nor the drafts
when the live head still holds the redacted text (an edit reverted byte for byte), as
`memory.redact` leaves a live copy of a text it scrubs from history.
- **A redact** of a past version of a definition's source, in the pull request that adds the
`knowledge_gap` and `query_history` candidate kinds, also reaches every such candidate
whose drafted value is the version's text, or that the version's `candidate-id:` tag
names, promoted or not: its text is blanked, since `learnings.list` shows it to admins,
and a blanked candidate is not promoted (`learnings.review` answers it not found, as it
does a blanked `assistant_turn` candidate). No other draft of the term is reached, and no
draft while the live head holds the redacted text.
- **The review path's promotion** of a `knowledge_gap` or `query_history` draft, in the
pull request that adds those kinds: the approved hash is the canonical-JSON hash of the
drafted value; `decided_by` and the audit subject come from the token, never the body's
Expand Down Expand Up @@ -272,24 +281,23 @@ namespaces](../concepts/identities-and-namespaces.md#agent-written-rows-and-the-
and the generated operations reference.

**Stored answers.** `memory.ask`'s ordinary precedent path reruns retrieval and reuses an
answer only when its synthesis window is the same, so a governed hit entering or leaving
the window already stops the reuse. Its fast path, which the ClickHouse authority runs,
reuses an answer while the namespace's write mark has not moved. Every insert into
`loom.shared` (a promotion, a demotion) moves the namespace's memory write mark
(`PLANE_BY_TABLE` in `server/core/namespace_write_marks.py`), so once the mark lands, no
answer from before a promotion or demotion on N is reused after it. It lands at once in
the process that wrote it, and in every other process once the writing process flushes its
marks (`namespace_write_marks_flush_interval_s`). An answer retrieved between a
promotion's shared row and its vector (including a vector the embedding worker fills in
later) is still reused. A relabel to `medium` or `high` moves the mark through its new
version of the source row in `loom.memory`, and the redact's scrub of a shared copy will
move it (`namespace_write_marks.bump`, the redact change above). Rolling upgrades below
covers N-1's. A lapse with no write, a source passing its `expires_at` or `valid_to`, does
not move the mark, and the precedent path's evidence check compares each cited row's live
hash (catching a tombstone, a redaction or an edit), not its status or validity. An answer
that quoted such a definition is reused for up to
`memory_ask_precedents_fast_max_age_hours` (with no bound when it is 0), as for any row
that expires today.
answer only when its synthesis window is the same, so a governed hit entering or leaving the
window already stops the reuse. Its fast path, which the ClickHouse authority runs, reuses
an answer while the namespace's write mark has not moved. Every insert into `loom.shared` (a
promotion, a demotion) moves the namespace's memory write mark (`PLANE_BY_TABLE` in
`server/core/namespace_write_marks.py`), so once the mark lands, no answer from before a
promotion or demotion on N is reused after it. It lands at once in the process that wrote
it, and in every other process once the writing process flushes its marks
(`namespace_write_marks_flush_interval_s`). An answer retrieved between a promotion's shared
row and its vector (including a vector the embedding worker fills in later) is still reused.
A relabel to `medium` or `high` moves the mark through its new version of the source row in
`loom.memory`, and the redact's scrub of a shared copy moves it
(`namespace_write_marks.bump`). Rolling upgrades below covers N-1's. A lapse with no write,
a source passing its `expires_at` or `valid_to`, does not move the mark, and the precedent
path's evidence check compares each cited row's live hash (catching a tombstone, a redaction
or an edit), not its status or validity. An answer that quoted such a definition is reused
for up to `memory_ask_precedents_fast_max_age_hours` (with no bound when it is 0), as for
any row that expires today.

**Ranking-neutral rule:** when no governed term matches, the leg adds nothing and the
served list is exactly what it was before. The memory-plane rule asks for a paired
Expand Down Expand Up @@ -610,13 +618,13 @@ promoted again:
them, the definition is written again from its shared copy's text.

An N-1 relabel of a definition's namespace-wide source to `medium` or `high` leaves its
shared copy served by N-1's search; N's search leaves it out by that row's label. A copy
N-1 promoted from another row, which it does not refuse (a `medium` one included), is
judged by the namespace-wide row under its key, not by the row it came from: the
re-promotion step above replaces it with one promoted from that row. N-1's redact does not
reach the shared row's text, as today: one run during the rollout or after a rollback is
run again once N serves. A namespace erasure an older build without the shared step ran is
cleared as the rolling-upgrades rule "A namespace's promoted rows" says ([rolling
shared copy served by N-1's search; N's search leaves it out by that row's label. A copy N-1
promoted from another row, which it does not refuse (a `medium` one included), is judged by
the namespace-wide row under its key, not by the row it came from: the re-promotion step
above replaces it with one promoted from that row. N-1's redact does not reach the shared
row's text: one run during the rollout or after a rollback is run again once N serves. A
namespace erasure an older build without the shared step ran is cleared as the
rolling-upgrades rule "A namespace's promoted rows" says ([rolling
upgrades](../architecture/rolling-upgrades.md)).

A rollback to N-1 meets what N wrote. Before it, the operator turns
Expand Down
Loading
Loading