Skip to content

feat(evaluator): bundle metrics for plugin execution#23

Closed
SandyChapman wants to merge 3 commits into
mainfrom
metric-bundling-cloudpickle/schapman
Closed

feat(evaluator): bundle metrics for plugin execution#23
SandyChapman wants to merge 3 commits into
mainfrom
metric-bundling-cloudpickle/schapman

Conversation

@SandyChapman
Copy link
Copy Markdown
Contributor

Summary

  • add MetricBundle and cloudpickle bundling primitives to the evaluator SDK
  • make evaluator plugin backend bundle runtime metrics before remote submission
  • hydrate bundles in plugin execution so the backend has no hard-coded metric type knowledge
  • publish remote evaluator aggregate, row-score, and artifact results through Jobs result storage
  • add local and remote smoke scripts covering built-in, custom protocol, and multi-metric execution

Validation

  • uv run pre-commit run -a
  • uv run --frozen pytest plugins/nemo-evaluator/tests/test_evaluate_job.py plugins/nemo-evaluator/tests/test_sdk_job_resources.py -q
  • uv run --frozen scripts/run_evaluator_local_bundle_path.py
  • uv run --frozen scripts/run_evaluator_remote_bundle_path.py

@SandyChapman SandyChapman force-pushed the metric-protocol-contract/schapman branch 4 times, most recently from 14efb5a to 5fcb878 Compare May 22, 2026 18:05
@SandyChapman SandyChapman force-pushed the metric-bundling-cloudpickle/schapman branch 3 times, most recently from 6f020db to 92f3566 Compare May 25, 2026 12:36
@SandyChapman SandyChapman changed the base branch from metric-protocol-contract/schapman to main May 25, 2026 12:36
@SandyChapman SandyChapman force-pushed the metric-bundling-cloudpickle/schapman branch from 92f3566 to 81de0ba Compare May 25, 2026 12:37
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 25, 2026

Suite Lines Covered Line Rate Branch Rate
Unit Tests 18392/24362 75.5% 61.8%
Integration Tests 11830/23144 51.1% 26.2%

Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
@SandyChapman SandyChapman force-pushed the metric-bundling-cloudpickle/schapman branch from 81de0ba to 9ff99d7 Compare May 25, 2026 15:23
@SandyChapman
Copy link
Copy Markdown
Contributor Author

Closing this older draft in favor of #46, which carries the resolver-based metric bundling work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant