fix(chunking): stable content-defined chunking with hash-based reuse planning

[MauryaQbit]

Fixes #1649.

Problem

On the self-hosted server, updating an existing document via the /v3/documents\ customId upsert path re-embeds nearly all chunks when the edit is anywhere but the tail: a one-paragraph mid-file edit to a 105 KB doc re-embedded 1007 of 1102 chunks (~21 min CPU on 2 vCPU with local embeddings). Append-only documents diff perfectly, which points at position-sensitive chunk boundaries defeating the per-chunk reuse check for mid-file edits: an early insertion/deletion shifts every downstream boundary, so every downstream chunk hash changes and reuse never hits.

Fix

New zero-dependency shared package @repo/chunking\ (\packages/chunking) implementing exactly the two mitigations suggested in the issue:

1. Content-defined chunking (\stableChunkText) — markdown headings and fenced code blocks are hard resync points (never merged across), and within a section boundaries fall on sentence/code-line units with FastCDC-style content-hash anchored cuts, so boundaries re-synchronize a few units after an edit instead of shifting to end-of-document. Boundaries always land on sentence/line edges; documented 2-sentence overlap preserved (restricted to the same section so sections stay independent).
2. Anchor-based resync (\planChunkReuse) — matches the new chunk list against stored chunks by content hash (FIFO per hash, O(n)) instead of by position, so unchanged chunks hit regardless of where they moved. Returns { reused, embedIndices }: carry over old embeddings, embed only \embedIndices.

Intended adoption on the upsert path: \chunks = stableChunkText(content)\ then \plan = planChunkReuse(storedChunks, chunks)\ and embed only \plan.embedIndices. Pure TS, no
ode:crypto, runs in Node/Bun/Workers.

Verification

14 new vitest tests (\packages/chunking/src/stable-chunking.test.ts, \�un run test\ in the package after install; verified here with vitest 3.2.4 — 14/14 pass). On a generated 45 KB / 90-chunk markdown doc:

scenario	stable chunker	naive fixed-size chunker
mid-file sentence replace	89 reused, 1 embeds	44 reused, 48 embed
append new section	90 reused, 1 embeds	—
delete a section	0 new embeds	—

Also covered: determinism, sequential positions + hash identity, sentence-boundary cuts, intact code-fence lines, duplicate-chunk FIFO pairing, position-independent matching, empty-input edge cases. \ sc --noEmit --strict\ (incl.
oUncheckedIndexedAccess\ + \exactOptionalPropertyTypes) and \�iome check\ are clean.

Note: the ingestion/embedding pipeline itself lives outside this monorepo, so this PR provides the algorithm + tested reference implementation for the pipeline to adopt; no existing behavior in this repo changes.

[MauryaQbit]

Update: pushed lockfile + CI wiring so this is merge-ready pending review.

Second commit (\9d22f5e)

• \�un.lock: registers the new \packages/chunking\ workspace (keeps \�un install --frozen-lockfile\ green).
• .github/workflows/ci.yml: runs chunking unit tests + type checking when \packages/chunking\ changes (same conditional pattern as tools/ai-sdk).

Verified locally with the exact CI toolchain (bun 1.3.6)

• \�un install --frozen-lockfile\ — no changes, passes
• \�un run --cwd packages/chunking test\ — 14/14 pass
• \�un run --cwd packages/chunking check-types\ — clean
• \�unx biome ci --changed --since=origin/main --no-errors-on-unmatched\ — clean

Note: the Actions runs for this fork PR show \�ction_required, so CI needs a maintainer to approve the workflow runs. Ready for review.