Skip to content

feat(core): topical dedup context — build_dedup_context + WASM/PyO3 bindings (PRD-04 DEP-3, Part A) - #701

Open
p-diogo wants to merge 5 commits into
mainfrom
feat/dep-3-dedup-context-core
Open

p-diogo wants to merge 5 commits into
mainfrom
feat/dep-3-dedup-context-core

Conversation

@p-diogo

@p-diogo p-diogo commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Re-created from #669 on 2026-10-31. GitHub Support is removing the pull-request refs of every PR opened before the 2026-10-25 history rewrite (private-data purge), which would leave #669 without a diff. Branch and content are unchanged; the earlier discussion stays on #669.

What

PRD-04 DEP-3 (F2), Part A: core only (refs internal #537). This adds the new module rust/totalreclaw-core/src/dedup_context.rs. build_dedup_context(topical, pinned, recent, cap) -> String renders the "Existing memories" block the extraction prompt receives:

  • the header is byte-identical to today's inline block in Hermes, the OpenClaw plugin and MCP;
  • one [ID: <fact id>] <text> line per memory, pinned first, then topical, then recent;
  • every line-break character in an id or text (the str.splitlines set: LF, CR, VT, FF, FS/GS/RS, NEL, U+2028, U+2029) becomes one space, so a stored fact cannot forge an extra [ID: …] line;
  • duplicate fact ids keep the first occurrence;
  • at most cap lines (default 30); "" when nothing survives.

It only formats. Choosing what goes in each section (all pins, top-20 topical, 10 newest) is the caller's job, in Part B.

Bindings: WASM buildDedupContext(topicalJson, pinnedJson, recentJson, cap?), PyO3 build_dedup_context(topical_json, pinned_json, recent_json, cap=30) (JSON in, String out). WASM throws on a negative, fractional, NaN or infinite cap instead of coercing it.

Parity

tests/parity/fixtures/dedup-context-v1.json holds 16 cases. generate-dedup-context-v1.py writes it from an independent Python reference implementation. Three suites check it:

  • Rust: cargo test dedup_context (18 tests)
  • WASM: tests/parity/dedup-context-parity.test.ts (new step in CI job cross-language-parity)
  • PyO3: python/tests/test_dedup_context_parity.py (CI job python-tests)

Spec deviation

Enclave spec §4.6 sketches Vec<Claim>. The v0 Claim has no id, and each line must carry the on-chain fact id that the LLM echoes back as existingFactId. The element type is therefore DedupContextItem { id, text }.

Not in this PR

  • No version bump. The coordinator opens the core 2.7.0 line.
  • Part B (the Hermes adapter in agent/lifecycle.py, plus fetchers in operations.py / client.py) is not written yet. It needs a published core with build_dedup_context.
  • No client calls the new function yet, so merging this changes no runtime behaviour.

Review log

🤖 Generated with Claude Code

p-diogo and others added 3 commits September 30, 2026 18:01
…ure (PRD-04 DEP-3)

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
p-diogo and others added 2 commits November 3, 2026 00:08
Resolve conflicts in ci.yml (keep both the DEP-6 and DEP-3 parity steps)
and the core CHANGELOG ([Unreleased] above 2.6.1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bd3G1BkVMFSXkcdAQ7nFax
Review fixes for PRD-04 DEP-3 Part A (refs internal #537):

- Collapse every line-break character, not just CR and LF. U+2028,
  U+2029, NEL, VT, FF and FS/GS/RS (the str.splitlines set) could still
  put a forged "[ID: ...]" on its own line for any model or renderer that
  honours them.
- WASM buildDedupContext: cap is now optional (default 30) and must be a
  non-negative integer. As a usize, -1 silently meant "no cap" and
  NaN/undefined silently gave an empty block.
- PyO3 build_dedup_context: cap defaults to 30.
- Fixture generator strips Unicode White_Space explicitly, matching Rust
  str::trim (Python str.strip also strips U+001C..U+001F). Two new
  fixture cases cover both; the 14 existing vectors are unchanged.
- Module doc states the caller contract (section sizes are the caller's
  job; pins crowd out the rest at cap; strip embeddings before
  serialising).
- New tests: non-array section ValueError (PyO3), default cap, cap
  validation (WASM), id collapse before dedupe, 5,000-pin section stops
  at cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bd3G1BkVMFSXkcdAQ7nFax

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant