Skip to content

feat: add lightweight graph quality benchmarks - #13

Open
Someday-czy wants to merge 1 commit into
HKUST-KnowComp:mainfrom
Someday-czy:feat/lightweight-graph-benchmarks
Open

feat: add lightweight graph quality benchmarks#13
Someday-czy wants to merge 1 commit into
HKUST-KnowComp:mainfrom
Someday-czy:feat/lightweight-graph-benchmarks

Conversation

@Someday-czy

Copy link
Copy Markdown
Contributor

Summary

  • Add deeprefine benchmark prepare, evaluate, and report CLI commands.
  • Add deterministic intrinsic and downstream graph-quality metrics.
  • Add Re-DocRED-mini and 2Wiki-mini preparation adapters.
  • Add a self-authored synthetic test fixture for offline metric and CLI verification.
  • Add package data, attribution documentation, and benchmark tests.

Validation

  • python -m pytest -q — 49 tests passed.
  • python -m deeprefine_skill.cli benchmark --help
  • python -m build

Scope

  • Default evaluation uses the Python standard library and does not require a GPU or LLM API.
  • The synthetic suite is only an expected fixture output for testing metric calculations and CLI behavior.
  • No real Re-DocRED or 2Wiki performance result is claimed in this PR.
  • Third-party raw datasets are not redistributed.
  • G-BERTScore is optional and remains outside the default installation and CI path.

Notes

  • evaluate reads saved baseline and candidate graphs without modifying them.
  • Optional prediction JSONL files enable retrieval and answer metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant