Skip to content

chore(bench): add cbm-vs-graphify benchmark suite#628

Closed
yfjin-png wants to merge 1 commit into
DeusData:mainfrom
yfjin-png:chore/add-benchmark-suite
Closed

chore(bench): add cbm-vs-graphify benchmark suite#628
yfjin-png wants to merge 1 commit into
DeusData:mainfrom
yfjin-png:chore/add-benchmark-suite

Conversation

@yfjin-png

Copy link
Copy Markdown

Summary

codebase-memory (cbm) と graphify を比較するベンチマーク一式を bench/ 配下に追加します。

内容

  • スクリプト: build_oracle.py, generate_qbank.py, compare_layer1.py, measure_layer3.py, analyze_layer2.py, qa_workflow.js, headtohead.sh
  • 結果: oracle.json, layer2_full.json, qbank.jsonl, summary.json ほか (bench/cbm-vs-graphify/results/)
  • レポート: REPORT_cbm_vs_graphify.md, BASELINE.md, README.md

19 files changed, 2015 insertions(+).

Notes

ドキュメント/ベンチマークの追加のみで、製品コードへの変更はありません。

@DeusData

Copy link
Copy Markdown
Owner

Huge thanks for opening this PR and for the work you put into it.

The maintainer shop is currently full, so this may sit for a bit before it gets a proper review. We will come back to this as soon as possible with real feedback; I wanted to make sure it did not sit unacknowledged in the meantime.

@DeusData DeusData added enhancement New feature or request stability/performance Server crashes, OOM, hangs, high CPU/memory priority/backlog Valuable contribution, lower scheduling urgency; review when maintainer capacity opens. labels Jun 29, 2026
@DeusData

DeusData commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Thank you for putting this together — head-to-head benchmarking against graphify is genuinely useful signal. We're going to close this one, for a couple of practical reasons rather than the idea: the DCO check is failing (sign-off email mismatch), and the suite bakes in machine-specific state — hardcoded absolute paths (/Users/charlesqin/…, /Users/yf.jin/.local/bin/…) and ~1900 lines of committed result JSON — so it isn't reproducible for anyone else as-is. If you'd like to bring it back as portable tooling (parameterized binary paths + repo root, results out of git, DCO signed-off), we'd be glad to take another look. Appreciate the effort and the competitive analysis!

@DeusData DeusData closed this Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request priority/backlog Valuable contribution, lower scheduling urgency; review when maintainer capacity opens. stability/performance Server crashes, OOM, hangs, high CPU/memory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants