Skip to content

feat: retrieval ladder v0.5.0 (SQLite FTS5 + ripgrep) - #49

Merged
RichardHightower merged 1 commit into
mainfrom
feat/retrieval-ladder-0.5.0
Aug 30, 2026
Merged

feat: retrieval ladder v0.5.0 (SQLite FTS5 + ripgrep)#49
RichardHightower merged 1 commit into
mainfrom
feat/retrieval-ladder-0.5.0

Conversation

@RichardHightower

Copy link
Copy Markdown
Contributor

Replaces the JSON knowledge/.index/ inverted index (committed by accident in 0.4.x) with PKC's disposable retrieval ladder. Git + Markdown stays source of truth.

index present  →  use it
else rg on PATH  →  prefilter
else             →  pure Python scan

What landed

  • SQLite/FTS5 incremental index at knowledge/.dekc/index.sqlite (gitignored). Search and pack self-heal via mtime+size. build remains an alias for refresh --force so CI and existing skills keep working.
  • ripgrep prefilter (DEKC_RG_PATH / PKC_RG_PATH / OKF_RG_PATH). Optional. Never installed from a hook.
  • Score identity: Python scorer (title×10 / desc×5 / tags×4 / min(body,8)) ranks candidates. LIKE on the stored haystack is the default; --engine fts is the FTS5 MATCH opt-in (not score-identical).
  • Pack identity: undirected lineage (typed flow + SQL) for every visited node; extra non-lineage links[] still attach to the focus only. Pack always parses the current file for outbound SQL — rg -lF <path> cannot see a file that does not mention itself.
  • SQL edges store origin so incremental delete does not leak table-to-table edges.
  • Host manifests lockstep at 0.5.0 (.opencode/plugin/dekc.json and public/data/catalog.json were still 0.4.2).
  • Removes the committed sample-knowledge/.index/ JSON.

Design: docs/designs/retrieval-ladder.md.

Tests

  • tests/test_retrieval_ladder.py — rg identity, sqlite refresh/noop/touch/delete/drop, no JSON written, doctor path, sample revenue hits
  • existing tests/test_dekc.py (incl. version lockstep + plugin.json)
  • CI now globs scripts/dekc_*.py and asserts search engine plus index fts5/nodes

Leftovers (accepted)

  • dekc_pack.py --mermaid still calls build_graph (full scan)
  • Do not ignore the whole .dekc/ directory — .dekc/config.example.yml is committed

Replace JSON knowledge/.index/ with PKC's disposable ladder:
index present → rg prefilter → Python scan.

Git + Markdown stays source of truth. Search scores match a full scan
unless --engine fts. Pack lineage is undirected for every node; extra
non-lineage links stay on the focus. rg is optional and never installed
from a hook. knowledge/.dekc/index.sqlite is gitignored and self-heals
via mtime+size.
@RichardHightower
RichardHightower merged commit 4926577 into main Aug 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant