Skip to content

Agent Memory Namespace for PostgreSQL - #45

Open
andreimerlescu wants to merge 3 commits into
Graphicle-AI:mainfrom
andreimerlescu:pgsql-memory
Open

andreimerlescu wants to merge 3 commits into
Graphicle-AI:mainfrom
andreimerlescu:pgsql-memory

Conversation

@andreimerlescu

Copy link
Copy Markdown
Contributor

Private memory per credential (memory namespaces)

A credential made with a memory_namespace reads the shared project (graph, sources, wiki) as any credential does, and reads and writes only its own namespace's memory: never shared (NULL) memory, never another namespace's. A credential without one behaves exactly as before. This lets one Graphicle instance serve many developers on one shared workspace (graphicle.dev)
without their agents' memories mixing.

Apart from whose memory it reaches, a namespaced key gets exactly what a key without a namespace gets: dense retrieval, cached query vectors, Qdrant, observer notes, the same MCP tools and descriptions, and replies with the same fields. A test proves this field by field (below).

How isolation is enforced

Three layers, because no one of them is enough on its own:

  1. Types. MemoryScope (now store::scope, re-exported as memory_scope) has private fields and pub(in crate::store) constructors. Principal's scope and self_review fields are private. The only way to get a scope is Principal::memory_scope(), and the only way to get a namespaced Principal is to authenticate as that credential. Every public memory helper takes &MemoryScope, never a namespace string: memory::{list_of_scoped, save_scoped, reviewed_state_scoped_of}, structured_memory::{attach, close_replaced}, claims::apply_memory_review, packs::import_scoped, work_state::start_section, store::event_scoped. The raw Option<&str> versions are private to their modules. Two compile_fail doctests check that a scope or principal can't be forged from outside store (both fail with E0624/E0616 privacy errors, not incidental ones). Principal::shared(..) is the public constructor for tests and extensions, and it can only make a shared principal.
  2. SQL lint. Rust can't see SQL text, so memory_namespace::every_sql_statement_on_namespaced_tables_is_scoped reads every string literal in crates/graphicle/src. It fails on any statement that names a namespaced table without memory_namespace, unless the statement is in an allowlist keyed by file and table with an exact count and a reason. Adding an unscoped statement fails the test until someone reviews it and updates the count. The scanner is a small Rust lexer, not a regex. The regex this branch started with paired quotes across '"' char literals, and so silently skipped the statements after line 539 of deletion.rs (20 statements on 7 namespaced tables). Those are now reviewed and listed, and a test pins the lexer on char literals, lifetimes, comments and raw strings.
  3. Behaviour. Credential B is pointed at everything A wrote (A's memory
    ids, claim ids, idempotency key, the words of its memory) through every
    read path, over the store, the command API and MCP. Every reply is
    searched for any trace of A, and for the string memory_namespace.
    Errors for A's ids must equal errors for an unknown id, status and
    message both.
  4. Parity. a_namespaced_key_sees_what_a_shared_key_sees runs one
    history twice: in one store a shared agent with a reviewer, in another
    namespace A next to namespace B (B writes, embeds, recalls and fills its
    query cache between A's writes and reads). The history writes a reviewed
    memory, work state and a claim, embeds, runs the observer, recalls with
    dense retrieval, again from the cached query vector, with observer note
    keys, and over Qdrant after the owner publishes, then reads 27 command
    and MCP paths (context, memories, search, export, status, workflows,
    pack, observer context, work state, history, claims, template,
    recall_project, tools/list, initialize, ...). With ids, digests,
    clocks and timings masked, every reply of A equals the shared agent's,
    and none carries B's text or id. A mock serves the embeddings, the
    observer model and a Qdrant collection.

I checked the suite catches real leaks by planting them and re-running:

Mutation Caught by
memory listing ignores the namespace 3 behaviour tests
work-state read ignores the namespace 2 behaviour tests
claims::current ignores the namespace nothing, because the claim is still filtered by the scoped memory list (defence in depth; no observable leak)
claim.history ignores the namespace nothing, for the same reason
claim.history ignores the namespace and the memory filter 2 behaviour tests (claim.history, shared claims)
a work-state count without the predicate (no behaviour change) the SQL lint
memory vectors stamped shared instead of with their namespace parity (A's own vector is no longer read: recall differs)
the semantic arm reads am_vectors without the namespace predicate the SQL lint
query-vector eviction across the whole project parity (B's 601 cached vectors trimmed)
memory rows keep the memory_namespace column parity and the shared-key test (memory_namespace in a reply)
observer notes withheld from namespaced keys (as before this change) parity (recall with note keys fails)
Qdrant reader without the id, fingerprint and partition checks parity (5 semantic candidates instead of 4: B's point)
the embedding cache ignores the namespace embedding_cache_namespaces (B's identical memory is a cache hit)

Tables

Namespaced: a nullable memory_namespace column, indexed with the column
that ties the row to its project. The list is memory_scope::TABLES, which
the migration, owner export/delete and the lint all share.

Table Why
credentials the key's namespace; set at creation, never updated
repo_memories, repo_memory_revisions agent memory and its immutable revisions
am_work_state agent work state
am_claims, am_claim_events, am_entities, am_entity_links interpreted claims of memory; namespaced entity ids are derived with the namespace, so they never collide with shared ones
repo_events events naming memory or job ids (event_scoped, job_event)
am_jobs, am_call_reservations, am_model_calls queued work and its model usage
am_model_cache, am_document_reads caches of model results over namespaced input; lookups and pruning are keyed by namespace
repo_runs, repo_tool_events session keys
am_vectors a memory passage's vector carries its memory's namespace; a source passage's stays NULL. A scope reads (memory_namespace IS NOT DISTINCT FROM ? OR identity NOT LIKE 'memory:%'): the sources and its own memory, then only its own candidates' identities and fingerprints. Memory ids are unique per project, and the upsert refuses to take over a row of another namespace
am_query_cache cached query vectors: written with, looked up by and evicted per namespace (newest 500 each). The request hash already names the credential
gf_observations, gf_mental_models, gf_alias_proposals, am_alias_proposals Core writes none of these; the column is there for editions that do
am_memory_namespaces (new) (workspace_id, name): self_review, created_at, last_read_at, last_write_at

Looked at and left alone:

Table Why
gf_episodes, gf_episode_labels, repo_pr_scratch the code graph's episodes, their labels and PR scratch projects: derived from sources, shared like the graph. GPT-6's draft added the column to these; this patch takes it out of the migration. A store already migrated by that draft keeps an always-NULL column, which is harmless.
am_passage_segments/files/terms index the project's files only, never memory text
am_observer_notes, am_observer_reflections a note's model input is its source unit alone, never memory, so notes are shared like the sources they observe and served under the same view rule (source in the reader's view, same hash). Namespaced keys run the observer and read notes as any key does. GPT-6's draft (and this branch until now) refused them; the column is out of TABLES, so the migration no longer adds it
am_embedding_cache the opt-in cross-store cache: a namespace's memory passages are keyed in their own space (passage:<model>:memory_namespace:<n>), sources and shared memory in the shared one as before
gf_symbol_pins keyed by memory_id with ON DELETE CASCADE; it follows its memory
am_reports, am_communities, am_preparation_steps no Core reader or writer; project deletion clears them
gf_served, gf_outcomes, gf_hints no Core writer (extension tables); memory deletion still strips ids from them
repo_evidence_links no longer read (see memory.rs)
reviews, decisions sync reviews of sources, not agent memory
am_served_usage metering per credential, owner-only

Read paths

All filter to the caller's scope in SQL, before any row is decoded,
counted or checked for staleness. Each is exercised by the behaviour test,
except where noted.

  • recall_project / context: snapshot memories, memory-source loading,
    the "has reviewed memory" graph-walk switch, and the session-start
    section (work state and memories, capture.rs)

  • search_project_memory / memory.search (identity and lexical)

  • project_memory_history / memory.history (including known_at)

  • project_memory / memories, memory.export, memory.template

  • work_state_read and the work-state count bound

  • repository_status / status: memory counts, and the graph with its
    claim and entity nodes (graph.rs)

  • claim.history, interpretation.prepare (claim targets),
    prepare_memory_refresh

  • workflow_status / workflow.status, workflow.cancel,
    workflow.retry: another namespace's job id is now "Job unavailable",
    like an unknown id, instead of "denied". The owner still reaches every
    job. Job dedup by input hash is scoped too.

  • wiki (project_wiki.rs) and muse_code.rs read memory through
    memory::list (scoped). prompt_context.rs reads no memory. These are
    scoped by construction and not separately probed: they need a model or a
    local session.

  • workspace_map.rs and symbol_links.rs: symbol links of memories

  • pack, packs::import_scoped (memory.import_pack)

  • MCP initialize

  • dense retrieval: the embed job, the recall/answer semantic arm, Qdrant
    publication and ranking, and the local context dense arm (same
    predicate; it needs an ONNX model, so only the ignored local-inference
    tests run it)

  • observer notes: observer_notes jobs, note_keys, observer_notes on
    answers, observer.context

Writes stamp the caller's scope: memory.propose, note, memory.import,
memory.import_pack, learning.review, learning.refresh.propose,
interpretation.propose, work_state.write, memory.delete, job enqueue,
model calls and caches (by job, in SQL), memory vectors and cached query
vectors.

Replies. Every reader that returns a namespaced row drops its
memory_namespace column (memory_scope::conceal): memory listings (and
so recall, search, export, pack, MCP), claims, and a job's model calls. A
memory revision no longer copies the namespace into its JSON (the column
holds it). A key's namespace is the key's, not part of what it reads.

Qdrant. Only the owner publishes, and owner keys are never namespaced,
so a publication holds what the owner's view holds current plus each
namespace's memory vectors. Each point carries a partition (source,
memory, memory:<n>). A reader asks for its own candidates' ids and
accepts a point only if its fingerprint matches and its partition is the
sources' or its own. Points of earlier releases have no partition and are
read as before until the next publication replaces them.

Review inside a namespace. self_review: true, set by the owner when
the namespace's first key is made, lets that namespace's keys that have
propose approve its own drafts (role_grants). The policy is immutable:
a later key that restates it differently gets a 409. Grounding checks
(quotes of current sources, size limits, <private> spans, credential
redaction) are unchanged. A namespace without self_review can't approve
(tested). Shared reviewers can't see a namespace's drafts at all.

The switch for a later "shared plus mine" mode is
memory_scope::SHARED_FALLBACK_ENABLED (false). Nothing reads it yet.

Owner operations

GET .../memory-namespaces/{n}/stats, GET .../export,
POST .../import, POST .../delete (owner only):

  • stats: rows, bytes per table, last_read_at, last_write_at. A
    request that needs only read stamps the read time, and anything else
    stamps the write time, so a recall is a read even though it runs on the
    write path.
  • export: every row of the namespace across the workspace's projects,
    with a SHA-256 of the canonical JSON.
  • import: restores into the same namespace only. The target must be
    empty, and parents must exist in the workspace. It refuses a tampered
    archive, another namespace's archive, unknown tables, invalid column
    names and rowid overrides.
  • delete: one transaction, reverse dependency order, a residue scan of
    every table that must find zero rows, and the hash-chained receipt of
    deletion.rs.

Project deletion cascades through repository_id and clears every
namespace's caches (tested over every table in TABLES). memory.delete
by a namespaced key purges that namespace's cached results. It used to
purge none.

Downgrades

A program without this change ignores memory_namespace: it would
authenticate a namespaced key as an ordinary key of its role and serve
namespaced memory as shared. So the first namespaced key a store gets
records store format 2 in store_readers (READER_FORMAT is now 2), and
0.1.3 to 0.1.18 refuse such a store at open with "needs Graphicle 0.1.19 or
newer". Every open also records it for a store that has a namespace but
lacks the record (a copy, or a build of this branch from before). A store
that never had a namespace stays at format 1 and can still be downgraded.
Test: a_store_with_a_namespace_refuses_programs_that_cannot_keep_it.

Where this differs from the brief

The export format is a lossless table archive
(graphicle.namespace.v1), not the format memory.import_pack reads.
import_pack re-drafts and redacts what it imports, so an export → delete
→ import round trip in that format could not give "the same recall answers
as before the export", which the brief's own test requires. The archive
keeps revisions, review state, claims and work state.

Earlier drafts of this branch kept namespaced keys out of dense vectors,
query embeddings and observer notes, and added a sentence to their memory
tools' descriptions. Both are gone: those features are partitioned (or, for
observer notes, shared like their sources), and every key lists the same
tools with the same descriptions.

Not changed, worth knowing

Captured agent sessions (session.capture) are connector documents of the
project's agent-sessions connector, which only owner keys read. A namespaced
key's captured sessions are therefore readable by the owner, as every key's
are; they are sources, not memory, and this patch leaves them as they were.

Also in this patch

  • Test fixtures that inserted rows positionally (INSERT INTO t VALUES)
    into tables that gained a column now name their columns (deletion.rs,
    backends.rs, product_workspace.rs, revocation.rs). No production
    code inserts positionally into these tables. An edition or extension that
    does must name its columns.
  • self_review is bound as an integer (PostgreSQL won't take a boolean in
    an INTEGER column).
  • rustfmt on the files this branch touches, including two graph-crate
    lines from the earlier clippy fixes.
  • embedding_cache::decode and query_file::sum use as_chunks for
    Clippy 1.99 (no behaviour change).

Test runs

Ubuntu 26.04.1, 12 cores, 125 GiB; Rust 1.100.0 (clippy 0.1.100) and the
MSRV 1.94.1; PostgreSQL 18.7 (PGDG), connected over the Unix socket as a
role that is not a superuser. CI's order and feature sets, every crate, on
d37948c plus the transcription clippy fix:

sqlite    graphicle-graph   fmt, clippy (postgres,bedrock,transcription), 1255 passed, 3 ignored
sqlite    graphicle-memory  fmt, clippy, 177 passed, 1 ignored
sqlite    graphicle         fmt, clippy, clippy local-inference, 1106 passed, 3 ignored, doc tests 2 passed
postgres  graphicle         1106 passed, 3 ignored, doc tests 2 passed (593 store schemas)
msrv      1.94.1            cargo check of all three crates passes

The 3 ignored tests need Redis, Neo4j or Qdrant, as in CI. One PostgreSQL
test (graph_projection::a_head_set_recorded_by_an_older_engine_is_projected_again)
turns foreign keys off to imitate an older store, which on PostgreSQL sets
session_replication_role: the test role was granted that one parameter
(GRANT SET ON PARAMETER, PostgreSQL 15+). Graphicle itself does not need
it. macOS (Rust 1.99, SQLite) was green on all three crates as well.

Clippy 1.99 added chunks_exact_to_as_chunks, which flagged three existing
functions (embedding_cache::decode, query_file::sum, and in
graphicle-graph with transcription, the decoded-audio read of
transcribe_native.rs). All three now use as_chunks (stable since 1.88;
MSRV is 1.94.1). They read the same bytes in the same order, so stored
vectors, query-file checksums and decoded samples are unchanged, and the
embedding-cache and query-file tests pass.

Checklist

  • Every commit is signed off (git commit -s), as the
    Developer Certificate of Origin asks
    (CONTRIBUTING.md).
  • Tests and docs are updated where the behaviour changed.
  • If a Cargo.lock changed, I ran cargo deny check in that crate and
    refreshed its licence page, as CONTRIBUTING.md describes.
cargo deny check fails anyway, in all three crates. Bans, licences and sources pass; advisories fail. Upstream main (54ed0fb) fails the same three advisories, so CI’s deny job will be red on any PR right now. They’re new RustSec entries, not something this branch brought in:

RUSTSEC-2026-0544, libz-sys heap buffer overflow (a vulnerability). Fixed in 1.1.33; the locks have 1.1.31.
RUSTSEC-2026-0543 and RUSTSEC-2026-0555: chrono and chrono-tz are unmaintained. There’s no fixed version to move to.
  • Code taken or ported from another project is permissively licensed and
    credited in the crate's NOTICE.md.
  • If I changed a benchmark number or a claim in the docs, it links to its
    evidence in docs/benchmarks.md.

Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant