feat: add lightweight graph quality benchmarks

MANIFEST.in

@@ -14,3 +14,4 @@ recursive-include commands *.toml
 recursive-include skills *.md
 recursive-include docs *.md
 recursive-include deeprefine_skill/gemini_extension *
+recursive-include deeprefine_skill/benchmark_suites *.json *.jsonl *.md *.txt

README.md

@@ -162,6 +162,9 @@ Run from your KB project root.
 | `deeprefine apply --refresh-wiki --trace-file T --refinement-file F` | Apply actions and regenerate `graphify-out/wiki` from the refined graph; graph + Wiki are committed together |
 | `deeprefine apply --refresh-wiki --allow-low-confidence --trace-file T --refinement-file F` | Refresh the Wiki while explicitly overriding the LOW-confidence guard |
 | `deeprefine loop finish --trace-file T [--refinement-file F]` | Persist results and mark history refined |
+| `deeprefine benchmark prepare ...` | Prepare a deterministic smoke/Re-DocRED/2Wiki mini suite |
+| `deeprefine benchmark evaluate ...` | Compare baseline and candidate graphs without modifying either |
+| `deeprefine benchmark report ...` | Merge result JSON files into a README-ready Markdown table |
 
 ### Evidence-aware review and safe apply
 
@@ -186,6 +189,110 @@ When the knowledge base was created with Graphify Wiki output, use `--refresh-wi
 
 ---
 
+## Lightweight graph-quality benchmark
+
+`deeprefine benchmark` compares a Graphify baseline with a refined candidate
+without importing DeepRefine's FAISS/model stack. Default scoring is offline,
+deterministic, read-only, and uses only the Python standard library.
+
+It reports two complementary views:
+
+- **Intrinsic quality:** entity and strict `(head, relation, tail)`
+  precision/recall/F1 against an annotated graph, plus lightweight
+  GraphJudge-style G-BLEU and G-ROUGE.
+- **Downstream quality:** query-relevant evidence-edge recall, complete
+  multi-hop path rate, answer reachability, and before/after repair-regression
+  transitions. Optional prediction JSONL adds Hit@K, MRR, supporting-fact
+  Recall@K, answer EM, and token F1.
+
+The Wiki is a derived view of `graph.json`. Optional `--baseline-wiki` and
+`--candidate-wiki` arguments check `index.md`, local links, and orphan pages;
+those integrity checks are not treated as semantic quality scores.
+
+### Zero-cost smoke run
+
+The package includes a self-authored fixture solely to verify the evaluator:
+
+```bash
+deeprefine benchmark evaluate \
+  --suite synthetic-smoke-v1 \
+  --baseline-graph deeprefine_skill/benchmark_suites/synthetic-smoke-v1/baseline_graph.json \
+  --candidate-graph deeprefine_skill/benchmark_suites/synthetic-smoke-v1/candidate_graph.json \
+  --baseline-predictions deeprefine_skill/benchmark_suites/synthetic-smoke-v1/baseline_predictions.jsonl \
+  --candidate-predictions deeprefine_skill/benchmark_suites/synthetic-smoke-v1/candidate_predictions.jsonl \
+  --output-dir graphify-out/.deeprefine/benchmark/smoke
+```
+
+Expected self-check values:
+
+| Suite | Metric | Synthetic Before | Synthetic After | Δ |
+|---|---|---:|---:|---:|
+| synthetic-smoke-v1 | Entity F1 | 97.73% | 100.00% | +2.27 pp |
+| synthetic-smoke-v1 | Strict Triple F1 | 82.86% | 100.00% | +17.14 pp |
+| synthetic-smoke-v1 | Evidence Edge Recall | 47.92% | 100.00% | +52.08 pp |
+| synthetic-smoke-v1 | Complete Path Rate | 25.00% | 100.00% | +75.00 pp |
+| synthetic-smoke-v1 | Answer F1 | 25.00% | 100.00% | +75.00 pp |
+
+These values describe an intentionally damaged and repaired test graph. They
+are not a DeepRefine performance claim.
+
+### Prepare a real mini suite
+
+Raw third-party text is not bundled. Supply an official upstream JSON file:
+
+```bash
+deeprefine benchmark prepare \
+  --suite redocred-mini-v1 \
+  --profile readme \
+  --source /path/to/Re-DocRED/dev_revised.json \
+  --output-dir prepared/redocred-mini-v1
+
+deeprefine benchmark prepare \
+  --suite 2wiki-mini-v1 \
+  --profile readme \
+  --source /path/to/2wikimultihopqa.json \
+  --output-dir prepared/2wiki-mini-v1
+```
+
+`quick` selects 10 Re-DocRED documents or 16 balanced 2Wiki questions.
+`readme` selects 50 documents or 64 questions. Selection is deterministic, and
+`suite.lock.json` records source and generated-file SHA-256 hashes. Prepared
+corpora retain every selected 2Wiki question's distractor passages.
+
+After building and refining graphs from the prepared corpus, score the pair:
+
+```bash
+deeprefine benchmark evaluate \
+  --suite prepared/2wiki-mini-v1 \
+  --baseline-graph runs/before/graph.json \
+  --candidate-graph runs/after/graph.json \
+  --graphify-version 0.9.12 \
+  --deeprefine-version COMMIT_OR_MODEL \
+  --output-dir graphify-out/.deeprefine/benchmark/2wiki
+
+deeprefine benchmark report \
+  --result graphify-out/.deeprefine/benchmark/redocred/result.json \
+  --result graphify-out/.deeprefine/benchmark/2wiki/result.json \
+  --format markdown \
+  --output graphify-out/.deeprefine/benchmark/README-results.md
+```
+
+For optional G-BERTScore, install
+`pip install "deeprefine-cli[benchmark-semantic]"` and pass
+`--semantic-model roberta-large`. It is intentionally excluded from default
+CI because it downloads a large model and PyTorch runtime.
+
+2Wiki `evidences` are answer paths, not an exhaustive graph. Accordingly,
+2Wiki reports evidence recall/path coverage but never full-graph edge
+precision. This micro-benchmark supports regression checks and README
+demonstrations; it does not replace full GraphRAG evaluation. See
+[benchmark data attribution](docs/benchmark-data.md) for sources and licensing.
+Real Re-DocRED/2Wiki scores should only be added here together with their
+committed `result.json`, exact commands, versions, hashes, token counts, and
+cost.
+
+---
+
 ## Codex Integration
 
 <details>
@@ -563,7 +670,7 @@ deeprefine refine          # dry-run by default
 
 ```bash
 deeprefine --help
-# Expect: cursor, copilot, codex, claude, opencode, gemini, history, index, refine, review, apply, loop
+# Expect: cursor, copilot, codex, claude, opencode, gemini, benchmark, history, index, refine, review, apply, loop
 ```
 </details>
 

deeprefine_skill/benchmark_suites/README.md

@@ -0,0 +1,35 @@
+# Benchmark suites and third-party data
+
+The package ships only the self-authored `synthetic-smoke-v1` fixture. It is
+licensed under the same MIT license as DeepRefine-Skill and exists to test the
+evaluator; its scores are not a model-quality claim.
+
+Real mini suites are prepared locally from an upstream file supplied by the
+user. DeepRefine-Skill records the source SHA-256 and selected sample IDs, but
+does not redistribute the original text.
+
+## Re-DocRED
+
+- Paper: https://arxiv.org/abs/2205.12696
+- Repository: https://github.com/tonytan48/Re-DocRED
+- Expected input: the official revised dev JSON (`dev_revised.json`), with
+  `rel_info.json` in the same directory when human-readable relation names are
+  available.
+
+Review and comply with the dataset repository's current terms and the terms of
+its underlying Wikipedia/Wikidata content before use.
+
+## 2WikiMultiHopQA
+
+- Paper: https://arxiv.org/abs/2011.01060
+- Repository: https://github.com/Alab-NII/2wikimultihop
+- Expected input: an official JSON split, or the compatible
+  `DeepRefine/benchmark/2wikimultihopqa.json` file.
+
+The evaluator treats `evidences` as query-relevant reasoning paths only. They
+are not an exhaustive gold graph, so the 2Wiki suite never reports full-graph
+edge precision.
+
+Generated prepared-suite directories retain source paths, hashes, provenance,
+and paper links in `suite.json` and `suite.lock.json`. Those generated
+directories should not be redistributed without checking the upstream terms.

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/baseline_graph.json

@@ -0,0 +1,60 @@
+{
+  "directed": true,
+  "multigraph": true,
+  "nodes": [
+    {"aliases": ["M. Chen"], "id": "b01", "label": "Mira Chen", "source_file": "corpus\\doc-alpha.txt"},
+    {"aliases": ["Aurora"], "id": "b02", "label": "Aurora University", "source_file": "corpus\\doc-alpha.txt"},
+    {"id": "b03", "label": "Aster City", "source_file": "corpus\\doc-alpha.txt"},
+    {"aliases": ["Borealis"], "id": "b04", "label": "Borealis Republic", "source_file": "corpus\\doc-alpha.txt"},
+    {"id": "b05", "label": "1998", "source_file": "corpus\\doc-alpha.txt"},
+    {"id": "b06", "label": "Leo Hart", "source_file": "corpus\\doc-beta.txt"},
+    {"aliases": ["Quill"], "id": "b07", "label": "Quill Labs", "source_file": "corpus\\doc-beta.txt"},
+    {"aliases": ["Orion"], "id": "b08", "label": "Orion Database", "source_file": "corpus\\doc-beta.txt"},
+    {"id": "b09", "label": "Cedar City", "source_file": "corpus\\doc-beta.txt"},
+    {"id": "b10", "label": "2012", "source_file": "corpus\\doc-beta.txt"},
+    {"id": "b11", "label": "Mystery Lab", "source_file": "corpus\\doc-beta.txt"},
+    {"id": "b12", "label": "Nia Vale", "source_file": "corpus\\doc-gamma.txt"},
+    {"aliases": ["Silver"], "id": "b13", "label": "Silver Observatory", "source_file": "corpus\\doc-gamma.txt"},
+    {"aliases": ["Helios"], "id": "b14", "label": "Helios Comet", "source_file": "corpus\\doc-gamma.txt"},
+    {"id": "b15", "label": "Ridge Province", "source_file": "corpus\\doc-gamma.txt"},
+    {"id": "b16", "label": "2004", "source_file": "corpus\\doc-gamma.txt"},
+    {"id": "b17", "label": "Oren Pike", "source_file": "corpus\\doc-delta.txt"},
+    {"aliases": ["Bluebird"], "id": "b18", "label": "Bluebird Press", "source_file": "corpus\\doc-delta.txt"},
+    {"aliases": ["Ember"], "id": "b19", "label": "Ember Novel", "source_file": "corpus\\doc-delta.txt"},
+    {"id": "b20", "label": "Lumen Town", "source_file": "corpus\\doc-delta.txt"},
+    {"id": "b21", "label": "2018", "source_file": "corpus\\doc-delta.txt"}
+  ],
+  "edges": [
+    {"relation": "located_in", "source": "b02", "target": "b03"},
+    {"relation": "largest_city_of", "source": "b03", "target": "b04"},
+    {"relation": "founded_in", "source": "b02", "target": "b05"},
+    {"relation": "citizen_of", "source": "b01", "target": "b04"},
+    {"relation": "contains", "source": "b04", "target": "b03"},
+    {"relation": "born_in", "source": "b01", "target": "b03"},
+    {"relation": "enrolls", "source": "b02", "target": "b01"},
+    {"relation": "founded", "source": "b06", "target": "b07"},
+    {"relation": "built", "source": "b07", "target": "b08"},
+    {"relation": "headquartered_in", "source": "b07", "target": "b09"},
+    {"relation": "announced_in", "source": "b08", "target": "b10"},
+    {"relation": "lives_in", "source": "b06", "target": "b09"},
+    {"relation": "hosts", "source": "b09", "target": "b07"},
+    {"relation": "developed_by", "source": "b08", "target": "b07"},
+    {"relation": "created", "source": "b06", "target": "b08"},
+    {"relation": "owns", "source": "b11", "target": "b08"},
+    {"relation": "works_at", "source": "b12", "target": "b13"},
+    {"relation": "observed", "source": "b13", "target": "b14"},
+    {"relation": "located_in", "source": "b13", "target": "b15"},
+    {"relation": "discovered_in", "source": "b14", "target": "b16"},
+    {"relation": "observed", "source": "b12", "target": "b14"},
+    {"relation": "contains", "source": "b15", "target": "b13"},
+    {"relation": "observed_by", "source": "b14", "target": "b12"},
+    {"relation": "lives_in", "source": "b12", "target": "b15"},
+    {"relation": "wrote", "source": "b17", "target": "b19"},
+    {"relation": "published", "source": "b18", "target": "b19"},
+    {"relation": "published_in", "source": "b19", "target": "b21"},
+    {"relation": "lives_in", "source": "b17", "target": "b20"},
+    {"relation": "hosts", "source": "b20", "target": "b18"},
+    {"relation": "author", "source": "b19", "target": "b17"},
+    {"relation": "works_with", "source": "b17", "target": "b18"}
+  ]
+}

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/baseline_predictions.jsonl

@@ -0,0 +1,8 @@
+{"answer":"Borealis Republic","case_id":"qa-01","retrieved_nodes":["Borealis Republic","Aster City"],"retrieved_supporting_facts":[]}
+{"answer":"Aster City","case_id":"qa-02","retrieved_nodes":["Aster City","Borealis Republic"],"retrieved_supporting_facts":["doc-alpha#1"]}
+{"answer":"Quill Labs","case_id":"qa-03","retrieved_nodes":["Quill Labs","Cedar City"],"retrieved_supporting_facts":["doc-beta#0"]}
+{"answer":"2011","case_id":"qa-04","retrieved_nodes":["Cedar City","2012"],"retrieved_supporting_facts":[]}
+{"answer":"Silver Observatory","case_id":"qa-05","retrieved_nodes":["Silver Observatory","Ridge Province"],"retrieved_supporting_facts":["doc-gamma#0"]}
+{"answer":"Ridge Province","case_id":"qa-06","retrieved_nodes":["Ridge Province"],"retrieved_supporting_facts":["doc-gamma#2"]}
+{"answer":"Ember Novel","case_id":"qa-07","retrieved_nodes":["Ember Novel"],"retrieved_supporting_facts":["doc-delta#0"]}
+{"answer":"Bluebird Press","case_id":"qa-08","retrieved_nodes":["Bluebird Press","Lumen Town"],"retrieved_supporting_facts":["doc-delta#0"]}

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/candidate_graph.json

@@ -0,0 +1,60 @@
+{
+  "directed": true,
+  "multigraph": true,
+  "nodes": [
+    {"aliases": ["M. Chen"], "id": "c01", "label": "Mira Chen", "source_file": "corpus/doc-alpha.txt"},
+    {"aliases": ["Aurora"], "id": "c02", "label": "Aurora University", "source_file": "corpus/doc-alpha.txt"},
+    {"id": "c03", "label": "Aster City", "source_file": "corpus/doc-alpha.txt"},
+    {"aliases": ["Borealis"], "id": "c04", "label": "Borealis Republic", "source_file": "corpus/doc-alpha.txt"},
+    {"id": "c05", "label": "1998", "source_file": "corpus/doc-alpha.txt"},
+    {"id": "c06", "label": "Leo Hart", "source_file": "corpus/doc-beta.txt"},
+    {"aliases": ["Quill"], "id": "c07", "label": "Quill Labs", "source_file": "corpus/doc-beta.txt"},
+    {"aliases": ["Orion"], "id": "c08", "label": "Orion Database", "source_file": "corpus/doc-beta.txt"},
+    {"id": "c09", "label": "Cedar City", "source_file": "corpus/doc-beta.txt"},
+    {"id": "c10", "label": "2012", "source_file": "corpus/doc-beta.txt"},
+    {"id": "c11", "label": "Nia Vale", "source_file": "corpus/doc-gamma.txt"},
+    {"aliases": ["Silver"], "id": "c12", "label": "Silver Observatory", "source_file": "corpus/doc-gamma.txt"},
+    {"aliases": ["Helios"], "id": "c13", "label": "Helios Comet", "source_file": "corpus/doc-gamma.txt"},
+    {"id": "c14", "label": "Ridge Province", "source_file": "corpus/doc-gamma.txt"},
+    {"id": "c15", "label": "2004", "source_file": "corpus/doc-gamma.txt"},
+    {"id": "c16", "label": "Oren Pike", "source_file": "corpus/doc-delta.txt"},
+    {"aliases": ["Bluebird"], "id": "c17", "label": "Bluebird Press", "source_file": "corpus/doc-delta.txt"},
+    {"aliases": ["Ember"], "id": "c18", "label": "Ember Novel", "source_file": "corpus/doc-delta.txt"},
+    {"id": "c19", "label": "Lumen Town", "source_file": "corpus/doc-delta.txt"},
+    {"id": "c20", "label": "2018", "source_file": "corpus/doc-delta.txt"}
+  ],
+  "links": [
+    {"relation": "studied_at", "source": "c01", "target": "c02"},
+    {"relation": "located_in", "source": "c02", "target": "c03"},
+    {"relation": "capital_of", "source": "c03", "target": "c04"},
+    {"relation": "founded_in", "source": "c02", "target": "c05"},
+    {"relation": "citizen_of", "source": "c01", "target": "c04"},
+    {"relation": "contains", "source": "c04", "target": "c03"},
+    {"relation": "born_in", "source": "c01", "target": "c03"},
+    {"relation": "enrolls", "source": "c02", "target": "c01"},
+    {"relation": "founded", "source": "c06", "target": "c07"},
+    {"relation": "created", "source": "c07", "target": "c08"},
+    {"relation": "headquartered_at", "source": "c07", "target": "c09"},
+    {"relation": "released_in", "source": "c08", "target": "c10"},
+    {"relation": "lives_in", "source": "c06", "target": "c09"},
+    {"relation": "hosts", "source": "c09", "target": "c07"},
+    {"relation": "developed_by", "source": "c08", "target": "c07"},
+    {"relation": "created", "source": "c06", "target": "c08"},
+    {"relation": "works_at", "source": "c11", "target": "c12"},
+    {"relation": "discovered", "source": "c12", "target": "c13"},
+    {"relation": "located_in", "source": "c12", "target": "c14"},
+    {"relation": "discovered_in", "source": "c13", "target": "c15"},
+    {"relation": "observed", "source": "c11", "target": "c13"},
+    {"relation": "contains", "source": "c14", "target": "c12"},
+    {"relation": "observed_by", "source": "c13", "target": "c11"},
+    {"relation": "lives_in", "source": "c11", "target": "c14"},
+    {"relation": "wrote", "source": "c16", "target": "c18"},
+    {"relation": "published", "source": "c17", "target": "c18"},
+    {"relation": "located_in", "source": "c17", "target": "c19"},
+    {"relation": "published_in", "source": "c18", "target": "c20"},
+    {"relation": "lives_in", "source": "c16", "target": "c19"},
+    {"relation": "hosts", "source": "c19", "target": "c17"},
+    {"relation": "author", "source": "c18", "target": "c16"},
+    {"relation": "works_with", "source": "c16", "target": "c17"}
+  ]
+}

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/candidate_predictions.jsonl

@@ -0,0 +1,8 @@
+{"answer":"Aurora University","case_id":"qa-01","retrieved_nodes":["Aurora University"],"retrieved_supporting_facts":["doc-alpha#0"]}
+{"answer":"Borealis Republic","case_id":"qa-02","retrieved_nodes":["Borealis Republic","Aster City","Aurora University"],"retrieved_supporting_facts":["doc-alpha#0","doc-alpha#1","doc-alpha#2"]}
+{"answer":"Orion Database","case_id":"qa-03","retrieved_nodes":["Orion Database","Quill Labs"],"retrieved_supporting_facts":["doc-beta#0","doc-beta#1"]}
+{"answer":"2012","case_id":"qa-04","retrieved_nodes":["2012"],"retrieved_supporting_facts":["doc-beta#2"]}
+{"answer":"Helios Comet","case_id":"qa-05","retrieved_nodes":["Helios Comet","Silver Observatory"],"retrieved_supporting_facts":["doc-gamma#0","doc-gamma#1"]}
+{"answer":"Ridge Province","case_id":"qa-06","retrieved_nodes":["Ridge Province"],"retrieved_supporting_facts":["doc-gamma#2"]}
+{"answer":"Ember Novel","case_id":"qa-07","retrieved_nodes":["Ember Novel"],"retrieved_supporting_facts":["doc-delta#0"]}
+{"answer":"Lumen Town","case_id":"qa-08","retrieved_nodes":["Lumen Town","Bluebird Press"],"retrieved_supporting_facts":["doc-delta#0","doc-delta#1"]}

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/corpus/doc-alpha.txt

@@ -0,0 +1,3 @@
+Mira Chen studied at Aurora University and was born in Aster City.
+Aurora University is located in Aster City and was founded in 1998.
+Aster City is the capital of the Borealis Republic.

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/corpus/doc-beta.txt

@@ -0,0 +1,3 @@
+Leo Hart founded Quill Labs in Cedar City.
+Quill Labs created the Orion Database and is headquartered in Cedar City.
+Orion Database was released in 2012.

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/corpus/doc-delta.txt

@@ -0,0 +1,3 @@
+Oren Pike wrote Ember Novel and works with Bluebird Press.
+Bluebird Press published Ember Novel in 2018 and is located in Lumen Town.
+Oren Pike lives in Lumen Town.

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/corpus/doc-gamma.txt

@@ -0,0 +1,3 @@
+Nia Vale works at Silver Observatory in Ridge Province.
+Silver Observatory discovered Helios Comet in 2004.
+Nia Vale observed Helios Comet and lives in Ridge Province.

deeprefine_skill/benchmark_suites/synthetic-smoke-v1/queries.jsonl

@@ -0,0 +1,8 @@
+{"case_id":"qa-01","question":"Where did Mira Chen study?"}
+{"case_id":"qa-02","question":"Which country contains the university where Mira studied?"}
+{"case_id":"qa-03","question":"What database was created by the company Leo founded?"}
+{"case_id":"qa-04","question":"When was Orion Database released?"}
+{"case_id":"qa-05","question":"What comet was discovered by Nia's observatory?"}
+{"case_id":"qa-06","question":"Where is Silver Observatory located?"}
+{"case_id":"qa-07","question":"What novel did Oren Pike write?"}
+{"case_id":"qa-08","question":"Where is the publisher that works with Oren located?"}