Skip to content

Add severity-weighted anytime risk discovery - #6

Open
BurhanAbdullah wants to merge 1 commit into
Power-Agent:mainfrom
BurhanAbdullah:renewable-grid-benchmark
Open

Add severity-weighted anytime risk discovery#6
BurhanAbdullah wants to merge 1 commit into
Power-Agent:mainfrom
BurhanAbdullah:renewable-grid-benchmark

Conversation

@BurhanAbdullah

@BurhanAbdullah BurhanAbdullah commented Aug 11, 2026

Copy link
Copy Markdown

Summary

This PR adds evaluator-side severity-weighted anytime risk-discovery metrics to the Level 2 steady-state N-2 benchmark.

What changed

  • Adds anytime_risk_auc to quantify the timeliness of risk discovery.
  • Adds severity-weighted risk-discovery metrics at 25%, 50%, 75%, and 100% of the validation budget.
  • Uses hidden oracle severity only within the evaluator; oracle information is not exposed to agents.
  • Preserves the validation trajectory so the evaluator can measure when severe contingencies are discovered.
  • Integrates the metrics into the existing score_agent() evaluation pipeline.
  • Adds regression and integration tests covering early vs. late discovery, validation-budget limits, zero-risk and empty cases, metric bounds, and score_agent() integration.
  • Documents the new metrics in the main README and the Level 2 benchmark README.

Validation

  • python -m unittest discover -s tests -v — 6/6 passed.
  • python -m py_compile poweragentbench/steady_state_agentic.py — passed.
  • python -m py_compile tests/test_anytime_risk.py — passed.
  • git diff --check — passed.
  • Evaluated across validation budgets of 10, 20, 40, and 80.
  • Tested on the IEEE 39-bus Case39 benchmark.

Motivation

Existing benchmark metrics evaluate final discovery, evidence quality, safety, mitigation, action cost, and workflow compliance. They do not directly capture when an agent discovers severe hidden contingencies within a limited validation budget.

These metrics add that missing temporal dimension by measuring the cumulative severity discovered as the validation budget is consumed.

The proposed metrics therefore complement the existing benchmark metrics rather than replacing them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant