Repository navigation
feat(core): topical dedup context — build_dedup_context + WASM/PyO3 bindings (PRD-04 DEP-3, Part A) - #701
Open
p-diogo wants to merge 5 commits into
Open
feat(core): topical dedup context — build_dedup_context + WASM/PyO3 bindings (PRD-04 DEP-3, Part A)#701p-diogo wants to merge 5 commits into
p-diogo wants to merge 5 commits into
Conversation
…ure (PRD-04 DEP-3) Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Resolve conflicts in ci.yml (keep both the DEP-6 and DEP-3 parity steps) and the core CHANGELOG ([Unreleased] above 2.6.1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bd3G1BkVMFSXkcdAQ7nFax
Review fixes for PRD-04 DEP-3 Part A (refs internal #537): - Collapse every line-break character, not just CR and LF. U+2028, U+2029, NEL, VT, FF and FS/GS/RS (the str.splitlines set) could still put a forged "[ID: ...]" on its own line for any model or renderer that honours them. - WASM buildDedupContext: cap is now optional (default 30) and must be a non-negative integer. As a usize, -1 silently meant "no cap" and NaN/undefined silently gave an empty block. - PyO3 build_dedup_context: cap defaults to 30. - Fixture generator strips Unicode White_Space explicitly, matching Rust str::trim (Python str.strip also strips U+001C..U+001F). Two new fixture cases cover both; the 14 existing vectors are unchanged. - Module doc states the caller contract (section sizes are the caller's job; pins crowd out the rest at cap; strip embeddings before serialising). - New tests: non-array section ValueError (PyO3), default cap, cap validation (WASM), id collapse before dedupe, 5,000-pin section stops at cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bd3G1BkVMFSXkcdAQ7nFax
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PRD-04 DEP-3 (F2), Part A: core only (refs internal #537). This adds the new module
rust/totalreclaw-core/src/dedup_context.rs.build_dedup_context(topical, pinned, recent, cap) -> Stringrenders the "Existing memories" block the extraction prompt receives:[ID: <fact id>] <text>line per memory, pinned first, then topical, then recent;str.splitlinesset: LF, CR, VT, FF, FS/GS/RS, NEL, U+2028, U+2029) becomes one space, so a stored fact cannot forge an extra[ID: …]line;caplines (default 30);""when nothing survives.It only formats. Choosing what goes in each section (all pins, top-20 topical, 10 newest) is the caller's job, in Part B.
Bindings: WASM
buildDedupContext(topicalJson, pinnedJson, recentJson, cap?), PyO3build_dedup_context(topical_json, pinned_json, recent_json, cap=30)(JSON in, String out). WASM throws on a negative, fractional,NaNor infinitecapinstead of coercing it.Parity
tests/parity/fixtures/dedup-context-v1.jsonholds 16 cases.generate-dedup-context-v1.pywrites it from an independent Python reference implementation. Three suites check it:cargo test dedup_context(18 tests)tests/parity/dedup-context-parity.test.ts(new step in CI jobcross-language-parity)python/tests/test_dedup_context_parity.py(CI jobpython-tests)Spec deviation
Enclave spec §4.6 sketches
Vec<Claim>. The v0Claimhas no id, and each line must carry the on-chain fact id that the LLM echoes back asexistingFactId. The element type is thereforeDedupContextItem { id, text }.Not in this PR
agent/lifecycle.py, plus fetchers inoperations.py/client.py) is not written yet. It needs a published core withbuild_dedup_context.Review log
main(core 2.6.1, Python 2.5.3, fix(hermes): fast pre-turn recall; keep provider on after pairing #703, fix(python): end Harrier inputs with <eos> so semantic recall works #708). Conflicts only inci.ymland the core CHANGELOG; both sides kept.capfrom JS, PyO3 non-arrayValueErrortest, generator strip vs Rusttrim) and widened line collapsing to Unicode line breaks. The 14 original vectors are unchanged; two cases were added.🤖 Generated with Claude Code