WIP: OIE mapping pipeline + OWASP eval harness (not merge-ready)

[northdpole]

Status

Draft. Not ready for merging. Visibility snapshot of the living OIE librarian/harvester pipeline and the eval data the harnesses actually run on. Do not treat CI green/red as a merge signal. Do not write these decisions into the live OpenCRE graph.

B2 gold now loads from application.utils.mapping_fixtures (merged in #1098). Duplicate AISVS/API/LLM/Top10/K8s JSON under scripts/oie_owasp_eval/fixtures/b2_gold/ is gone; only the agentic stub stays there. This PR is still WIP for the full OIE pipeline — gold-loader follow-up is in, the rest is not merge-ready.

Summary

• Post-GSoC librarian/harvester/orchestrator work
• OWASP eval harness scripts under scripts/oie_owasp_eval/
• Harness testing data at scripts/oie_owasp_eval/fixtures/:
  • b2_gold/ — agentic-AI stub only (owasp_agentic_ai_stub.json); all other families use mapping_fixtures.load_owasp_mapping_fixture
  • b2_sources/ — section text the B2 harness scores against
  • reports/ — latest B1 / B2 / Module C accuracy JSON
• Librarian golden set remains at application/tests/librarian/fixtures/golden_dataset.json

How it is tested today

Vertical	What it measures	Latest snapshot
Librarian golden (evaluate_librarian.py)	C.0 boundary, C.0.5 explicit ids, C.1 recall@20, C.2 top-1, C.3 ECE, C.4 auto-link vs review	Retrieval ~98% recall@20; top-1 ~75%; decision accuracy at τ=0.80 ~57%; review recall 5/5
B1 (score_b1_opencre_api.py)	ASVS chunk / CheatSheet sheet vs live opencre.org CRE gold	33/37 (89%), gate >60% pass
B2 (run_b2_pr_mappings.py)	New-family section grain: ≥1 of union(rerank top-2 ∪ vector top-2) ∈ gold cre_ids	26/62 (42%), gate >60% fail

B2 by family (gated report b2_accuracy_report_gated.json): LLM Top 10 100%, K8s 2022 67%, Top 10 / API / K8s 2025 40%, AISVS 17%, Agentic stub 0% (no hub Links, expected weak).

The ~50% figure in Slack is the same story as conservative auto-link / B2 transfer, not “the pipeline is half broken.” Wrong auto-links are the dangerous direction; close calls go to review.

Reproduce

# B2 against mapping_fixtures gold + in-repo sources
PYTHONPATH=. python scripts/oie_owasp_eval/run_b2_pr_mappings.py --score-only --run-id <id>

# Librarian golden (hermetic)
python scripts/evaluate_librarian.py \
  --dataset application/tests/librarian/fixtures/golden_dataset.json

Test plan

• Mentors / contributors: read the reports and comment on ranking, corpus (CRE descriptions), and B2 misses — not “please merge”
• B2 gold for AISVS/API/LLM/Top10/K8s comes from mapping_fixtures; agentic stub remains local
• Before any real merge: split, rebase onto main, drop WIP leftovers, re-run B1/B2

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}