Repository navigation
Agent Memory Namespace for PostgreSQL - #45
Open
andreimerlescu wants to merge 3 commits into
Open
andreimerlescu wants to merge 3 commits into
andreimerlescu wants to merge 3 commits into
Conversation
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Private memory per credential (memory namespaces)
A credential made with a
memory_namespacereads the shared project (graph, sources, wiki) as any credential does, and reads and writes only its own namespace's memory: never shared (NULL) memory, never another namespace's. A credential without one behaves exactly as before. This lets one Graphicle instance serve many developers on one shared workspace (graphicle.dev)without their agents' memories mixing.
Apart from whose memory it reaches, a namespaced key gets exactly what a key without a namespace gets: dense retrieval, cached query vectors, Qdrant, observer notes, the same MCP tools and descriptions, and replies with the same fields. A test proves this field by field (below).
How isolation is enforced
Three layers, because no one of them is enough on its own:
MemoryScope(nowstore::scope, re-exported asmemory_scope) has private fields andpub(in crate::store)constructors.Principal's scope andself_reviewfields are private. The only way to get a scope isPrincipal::memory_scope(), and the only way to get a namespacedPrincipalis to authenticate as that credential. Every public memory helper takes&MemoryScope, never a namespace string:memory::{list_of_scoped, save_scoped, reviewed_state_scoped_of},structured_memory::{attach, close_replaced},claims::apply_memory_review,packs::import_scoped,work_state::start_section,store::event_scoped. The rawOption<&str>versions are private to their modules. Twocompile_faildoctests check that a scope or principal can't be forged from outsidestore(both fail with E0624/E0616 privacy errors, not incidental ones).Principal::shared(..)is the public constructor for tests and extensions, and it can only make a shared principal.memory_namespace::every_sql_statement_on_namespaced_tables_is_scopedreads every string literal incrates/graphicle/src. It fails on any statement that names a namespaced table withoutmemory_namespace, unless the statement is in an allowlist keyed by file and table with an exact count and a reason. Adding an unscoped statement fails the test until someone reviews it and updates the count. The scanner is a small Rust lexer, not a regex. The regex this branch started with paired quotes across'"'char literals, and so silently skipped the statements after line 539 ofdeletion.rs(20 statements on 7 namespaced tables). Those are now reviewed and listed, and a test pins the lexer on char literals, lifetimes, comments and raw strings.ids, claim ids, idempotency key, the words of its memory) through every
read path, over the store, the command API and MCP. Every reply is
searched for any trace of A, and for the string
memory_namespace.Errors for A's ids must equal errors for an unknown id, status and
message both.
a_namespaced_key_sees_what_a_shared_key_seesruns onehistory twice: in one store a shared agent with a reviewer, in another
namespace A next to namespace B (B writes, embeds, recalls and fills its
query cache between A's writes and reads). The history writes a reviewed
memory, work state and a claim, embeds, runs the observer, recalls with
dense retrieval, again from the cached query vector, with observer note
keys, and over Qdrant after the owner publishes, then reads 27 command
and MCP paths (context, memories, search, export, status, workflows,
pack, observer context, work state, history, claims, template,
recall_project,tools/list,initialize, ...). With ids, digests,clocks and timings masked, every reply of A equals the shared agent's,
and none carries B's text or id. A mock serves the embeddings, the
observer model and a Qdrant collection.
I checked the suite catches real leaks by planting them and re-running:
claims::currentignores the namespaceclaim.historyignores the namespaceclaim.historyignores the namespace and the memory filterclaim.history,shared claims)am_vectorswithout the namespace predicatememory_namespacecolumnmemory_namespacein a reply)recall with note keysfails)embedding_cache_namespaces(B's identical memory is a cache hit)Tables
Namespaced: a nullable
memory_namespacecolumn, indexed with the columnthat ties the row to its project. The list is
memory_scope::TABLES, whichthe migration, owner export/delete and the lint all share.
credentialsrepo_memories,repo_memory_revisionsam_work_stateam_claims,am_claim_events,am_entities,am_entity_linksrepo_eventsevent_scoped,job_event)am_jobs,am_call_reservations,am_model_callsam_model_cache,am_document_readsrepo_runs,repo_tool_eventsam_vectors(memory_namespace IS NOT DISTINCT FROM ? OR identity NOT LIKE 'memory:%'): the sources and its own memory, then only its own candidates' identities and fingerprints. Memory ids are unique per project, and the upsert refuses to take over a row of another namespaceam_query_cachegf_observations,gf_mental_models,gf_alias_proposals,am_alias_proposalsam_memory_namespaces(new)(workspace_id, name):self_review,created_at,last_read_at,last_write_atLooked at and left alone:
gf_episodes,gf_episode_labels,repo_pr_scratcham_passage_segments/files/termsam_observer_notes,am_observer_reflectionsTABLES, so the migration no longer adds itam_embedding_cachepassage:<model>:memory_namespace:<n>), sources and shared memory in the shared one as beforegf_symbol_pinsmemory_idwithON DELETE CASCADE; it follows its memoryam_reports,am_communities,am_preparation_stepsgf_served,gf_outcomes,gf_hintsrepo_evidence_linksmemory.rs)reviews,decisionsam_served_usageRead paths
All filter to the caller's scope in SQL, before any row is decoded,
counted or checked for staleness. Each is exercised by the behaviour test,
except where noted.
recall_project/context: snapshot memories, memory-source loading,the "has reviewed memory" graph-walk switch, and the session-start
section (work state and memories,
capture.rs)search_project_memory/memory.search(identity and lexical)project_memory_history/memory.history(includingknown_at)project_memory/memories,memory.export,memory.templatework_state_readand the work-state count boundrepository_status/status: memory counts, and the graph with itsclaim and entity nodes (
graph.rs)claim.history,interpretation.prepare(claim targets),prepare_memory_refreshworkflow_status/workflow.status,workflow.cancel,workflow.retry: another namespace's job id is now "Job unavailable",like an unknown id, instead of "denied". The owner still reaches every
job. Job dedup by input hash is scoped too.
wiki (
project_wiki.rs) andmuse_code.rsread memory throughmemory::list(scoped).prompt_context.rsreads no memory. These arescoped by construction and not separately probed: they need a model or a
local session.
workspace_map.rsandsymbol_links.rs: symbol links of memoriespack,packs::import_scoped(memory.import_pack)MCP
initializedense retrieval: the
embedjob, the recall/answer semantic arm, Qdrantpublication and ranking, and the local
contextdense arm (samepredicate; it needs an ONNX model, so only the ignored local-inference
tests run it)
observer notes:
observer_notesjobs,note_keys,observer_notesonanswers,
observer.contextWrites stamp the caller's scope:
memory.propose,note,memory.import,memory.import_pack,learning.review,learning.refresh.propose,interpretation.propose,work_state.write,memory.delete, job enqueue,model calls and caches (by job, in SQL), memory vectors and cached query
vectors.
Replies. Every reader that returns a namespaced row drops its
memory_namespacecolumn (memory_scope::conceal): memory listings (andso recall, search, export, pack, MCP), claims, and a job's model calls. A
memory revision no longer copies the namespace into its JSON (the column
holds it). A key's namespace is the key's, not part of what it reads.
Qdrant. Only the owner publishes, and owner keys are never namespaced,
so a publication holds what the owner's view holds current plus each
namespace's memory vectors. Each point carries a
partition(source,memory,memory:<n>). A reader asks for its own candidates' ids andaccepts a point only if its fingerprint matches and its partition is the
sources' or its own. Points of earlier releases have no partition and are
read as before until the next publication replaces them.
Review inside a namespace.
self_review: true, set by the owner whenthe namespace's first key is made, lets that namespace's keys that have
proposeapprove its own drafts (role_grants). The policy is immutable:a later key that restates it differently gets a 409. Grounding checks
(quotes of current sources, size limits,
<private>spans, credentialredaction) are unchanged. A namespace without
self_reviewcan't approve(tested). Shared reviewers can't see a namespace's drafts at all.
The switch for a later "shared plus mine" mode is
memory_scope::SHARED_FALLBACK_ENABLED(false). Nothing reads it yet.Owner operations
GET .../memory-namespaces/{n}/stats,GET .../export,POST .../import,POST .../delete(owner only):last_read_at,last_write_at. Arequest that needs only
readstamps the read time, and anything elsestamps the write time, so a recall is a read even though it runs on the
write path.
with a SHA-256 of the canonical JSON.
empty, and parents must exist in the workspace. It refuses a tampered
archive, another namespace's archive, unknown tables, invalid column
names and rowid overrides.
every table that must find zero rows, and the hash-chained receipt of
deletion.rs.Project deletion cascades through
repository_idand clears everynamespace's caches (tested over every table in
TABLES).memory.deleteby a namespaced key purges that namespace's cached results. It used to
purge none.
Downgrades
A program without this change ignores
memory_namespace: it wouldauthenticate a namespaced key as an ordinary key of its role and serve
namespaced memory as shared. So the first namespaced key a store gets
records store format 2 in
store_readers(READER_FORMATis now 2), and0.1.3 to 0.1.18 refuse such a store at open with "needs Graphicle 0.1.19 or
newer". Every open also records it for a store that has a namespace but
lacks the record (a copy, or a build of this branch from before). A store
that never had a namespace stays at format 1 and can still be downgraded.
Test:
a_store_with_a_namespace_refuses_programs_that_cannot_keep_it.Where this differs from the brief
The export format is a lossless table archive
(
graphicle.namespace.v1), not the formatmemory.import_packreads.import_packre-drafts and redacts what it imports, so an export → delete→ import round trip in that format could not give "the same recall answers
as before the export", which the brief's own test requires. The archive
keeps revisions, review state, claims and work state.
Earlier drafts of this branch kept namespaced keys out of dense vectors,
query embeddings and observer notes, and added a sentence to their memory
tools' descriptions. Both are gone: those features are partitioned (or, for
observer notes, shared like their sources), and every key lists the same
tools with the same descriptions.
Not changed, worth knowing
Captured agent sessions (
session.capture) are connector documents of theproject's agent-sessions connector, which only owner keys read. A namespaced
key's captured sessions are therefore readable by the owner, as every key's
are; they are sources, not memory, and this patch leaves them as they were.
Also in this patch
INSERT INTO t VALUES)into tables that gained a column now name their columns (
deletion.rs,backends.rs,product_workspace.rs,revocation.rs). No productioncode inserts positionally into these tables. An edition or extension that
does must name its columns.
self_reviewis bound as an integer (PostgreSQL won't take a boolean inan
INTEGERcolumn).lines from the earlier clippy fixes.
embedding_cache::decodeandquery_file::sumuseas_chunksforClippy 1.99 (no behaviour change).
Test runs
Ubuntu 26.04.1, 12 cores, 125 GiB; Rust 1.100.0 (clippy 0.1.100) and the
MSRV 1.94.1; PostgreSQL 18.7 (PGDG), connected over the Unix socket as a
role that is not a superuser. CI's order and feature sets, every crate, on
d37948cplus the transcription clippy fix:The 3 ignored tests need Redis, Neo4j or Qdrant, as in CI. One PostgreSQL
test (
graph_projection::a_head_set_recorded_by_an_older_engine_is_projected_again)turns foreign keys off to imitate an older store, which on PostgreSQL sets
session_replication_role: the test role was granted that one parameter(
GRANT SET ON PARAMETER, PostgreSQL 15+). Graphicle itself does not needit. macOS (Rust 1.99, SQLite) was green on all three crates as well.
Clippy 1.99 added
chunks_exact_to_as_chunks, which flagged three existingfunctions (
embedding_cache::decode,query_file::sum, and ingraphicle-graphwithtranscription, the decoded-audio read oftranscribe_native.rs). All three now useas_chunks(stable since 1.88;MSRV is 1.94.1). They read the same bytes in the same order, so stored
vectors, query-file checksums and decoded samples are unchanged, and the
embedding-cache and query-file tests pass.
Checklist
git commit -s), as theDeveloper Certificate of Origin asks
(CONTRIBUTING.md).
Cargo.lockchanged, I rancargo deny checkin that crate andrefreshed its licence page, as CONTRIBUTING.md describes.
credited in the crate's
NOTICE.md.evidence in
docs/benchmarks.md.