Migrate memories from other local memory systems into Engram

[jfrancoa]

Motivation

Anyone switching to Engram from another local memory system arrives with an existing store of accumulated memories and no way to bring it along. This adds a one-shot importer (/engram:migrate, bin/engram-migrate) with claude-mem as the first supported source. Bumps the plugin to 0.2.0.

Approach

• Adapter/engine split. A source adapter turns a foreign store into a stream of Records speaking a deliberately tiny, Engram-agnostic vocabulary of four kinds; the engine maps kinds to the group's actual topics (--topic-map for custom groups, validated against the live schema before anything is sent), resolves source project names to owner/repo via git-remote probing, batches per repo scope, and submits through the pre-extracted pipeline — the content was already LLM-summarized by the source, so no re-extraction. A new source is one adapter module plus a sources() entry.
• Planning is pure, execution is separate. Dry-run (the default) needs neither credentials nor the Engram SDK, so the whole migration can be inspected before anything leaves the machine. This forced a small refactor: core/__init__.py now lazy re-exports via PEP 562 and client.py imports the SDK inside get_client(), keeping SDK-free entry points working outside the plugin venv.
• Resumable by checkpoint, not by hope. ~/.engram/migrate/<source>.json records done/pending uids around every step (atomic writes); a re-run reconciles pending runs against server status and only sends what's missing.
• Curated by default. Transient per-session observation types (>80% of a real store, low recall value) ride behind --all.
• Skip, never mis-file. A project whose repo can't be resolved is excluded and reported (recoverable via --map) — a wrong repo_name would make memories unrecallable, which is worse than absent.
• Alternative path: --input conversation ingests through the extraction pipeline with real created_at date context; strictly chronological per the API contract, so an unfinished run aborts (resumably) rather than letting later days overtake it.
• Undo: --rollback deletes exactly what the migration created, via the server's per-run commit manifests.

Key areas for review

• plugin/core/migrate/engine.py execute / reconcile_pending — checkpoint discipline: pending is written before the wait, done/failed transitions after; verify no path can double-submit or lose uids
• plugin/core/migrate/engine.py rollback — deletes only memory ids listed in the migration's own run manifests (runs.get().committed_operations), so organically stored memories are untouchable by construction; checkpoint is kept when any manifest fetch fails
• plugin/core/__init__.py — the lazy re-export refactor sits under the existing hooks' import path; this is the only change touching live hook behavior
• plugin/core/migrate/__main__.py _batch_properties — required-scope handling: session_id auto-filled with a migration:<source> marker (found via live smoke: the add API rejects writes missing any group-required property), all other required props must be explicit
• plugin/core/migrate/claude_mem.py — read-only guarantee is at the sqlite level (mode=ro URI), not by convention

Risks and mitigations

• Mutating the source store: connection opened read-only at the sqlite level; a test asserts writes fail
• Duplicates on interrupt/re-run: checkpoint written around every submit/commit step with atomic replace; interrupted runs are reconciled against server status before planning, and their uids stay reserved while a run is still in flight. One inherent window remains: a crash between the server accepting an add and the pending entry being persisted resubmits that batch — closing it needs server-side idempotency keys
• Mis-filed (unrecallable) memories: unresolved projects are skipped and reported rather than guessed; topic mapping and required scope properties validated up front, failing before any batch is sent
• Rollback overreach: deletion enumerates only the server's per-run commit manifests for this migration's run ids; 404s tolerated for idempotence
• Hook regression from the lazy-import refactor: both hooks regression-tested against the refactored core; import resolution just moves to hook import time

Testing

• 12 unit tests (plugin/tests/test_migrate.py) against a synthetic claude-mem DB and a fake client: curation/composition (including legacy text and facts-only rows), read-only enforcement, batching and skip_uids, chronological conversation ordering, checkpoint roundtrip, pending reconciliation, execute commit-and-checkpoint flow, conversation input building, manifest-based rollback. Green on system python and the plugin venv.
• Real-data dry-run against an actual claude-mem store: 2179 items in 52 batches, 15/34 projects auto-resolved, skips reported with --map guidance.
• Full production runs of both paths against a real store (~10k observations, 1.2k summaries, 33 projects): pre-extracted (2531 items, 61 runs, 0 failures) → manifest rollback (1564 memories deleted exactly, native memories untouched) → conversation re-import (2830 records as 156 chronological conversations, 0 failures, resumed across three interruptions with no duplicates). Idempotency confirmed: a re-run sends nothing.

🤖 Generated with Claude Code

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[Copilot]

Pull request overview

Adds a one-shot migration tool to import memories from other local memory systems into Engram (initially claude-mem), with a dry-run-first workflow, resumable checkpointing, and optional rollback. It also refactors core imports so SDK-free entry points (migration dry-run and unit tests) can run outside the plugin venv.

Changes:

• Introduces migration adapter/engine/CLI (core.migrate) including repo resolution, batching, checkpointing, execution, and manifest-based rollback.
• Adds claude-mem SQLite adapter plus unit tests covering curation, planning, execution flow, reconciliation, and rollback.
• Updates docs/command metadata and bumps plugin version to 0.2.0; refactors core exports and client import to keep SDK imports lazy.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
README.md	Documents how to run migrations and explains key flags/modes.
plugin/tests/test_migrate.py	Adds migration unit tests (adapter behavior, planning, checkpoints, execution, rollback).
plugin/tests/init.py	Marks tests as a package for discovery/imports.
plugin/core/migrate/engine.py	Implements planning/execution, checkpointing, reconciliation, conversation mode, and rollback.
plugin/core/migrate/claude_mem.py	Implements the claude-mem read-only SQLite adapter and selection reporting.
plugin/core/migrate/main.py	Adds CLI entry point with dry-run default, execute/rollback flows, and schema/props validation.
plugin/core/migrate/init.py	Defines the adapter contract, Record, kinds, and adapter registry.
plugin/core/client.py	Makes Engram SDK import local to get_client() to support SDK-free code paths.
plugin/core/init.py	Switches to lazy re-exports via PEP 562 to avoid importing SDK at package import time.
plugin/commands/migrate.md	Adds Claude command guidance for safe dry-run-first migrations and confirmation gating.
plugin/bin/engram-migrate	Adds CLI launcher that derives plugin root/data env and runs migration inside the venv.
plugin/.claude-plugin/plugin.json	Bumps plugin version to 0.2.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (3)

plugin/core/migrate/main.py:51

• Same exit-code issue as above: sys.exit(<message>) here returns 1 even though this is a usage/validation error. Exiting with code 2 keeps the documented contract and aligns with argparse’s behavior.

        if not k.strip() or not v.strip():
            sys.exit(f"--{flag} {p!r}: name and value must be non-empty")

plugin/core/migrate/main.py:46

• _parse_kv uses sys.exit(<message>), which exits with status code 1. This contradicts the module’s documented exit codes (“2 usage error”) and makes bad --map/--topic-map/--property values indistinguishable from runtime failures. Emit the message on stderr and exit(2) instead.

This issue also appears on line 50 of the same file.

        if "=" not in p:
            sys.exit(f"--{flag} expects NAME=VALUE, got {p!r}")

plugin/core/client.py:92

• get_client() imports the Engram SDK before checking whether an API key exists. Hooks call get_client() even when memory is disabled; in environments without the SDK installed this will raise ModuleNotFoundError instead of returning None. Check ENGRAM_API_KEY first, and only then import the SDK (optionally raising a clearer error if the key is set but the SDK is missing).

def get_client():
    # SDK import stays local: everything else in this module (key/identity resolution, the
    # REST helper) is stdlib-only and must keep working where the SDK isn't installed.
    from engram import EngramClient

[jfrancoa]

Superseded by #7, which ships the same migration feature with the guided flow as a harness-independent skill instead of a slash command (Codex-portable). Keeping this open as the command-based fallback — only one of the two should merge.