Persist retry diagnostics in rollout outputs

[RitwijParmar]

Summary

This adds persisted retry diagnostics for eval rollouts when max_retries is used. Today retries are visible in logs, but saved results.jsonl does not show whether a rollout needed retries, which retryable error triggered them, or whether retries were exhausted. That makes it hard to debug transient provider and sandbox instability after the run has finished.

The PR adds a retry block to rollout state and saved output only when a retry actually happens. The block records attempts, retry count, exhaustion status, elapsed retry time, and per-attempt error summaries. It works for both single rollout and grouped rollout retry paths.

Related to #1607 and the retry visibility part of #1563.

What changed

• Track retry events inside maybe_retry for exception-based failures and retryable state["error"] failures
• Attach retry summaries to single-state and grouped-state results
• Persist the retry field through state_to_output
• Add RetryEvent and RetryData typed output contracts
• Document the saved retry block in eval docs and output reference
• Cover successful retry, exhausted retry, grouped retry, and output serialization

Tests

• uv run --python 3.12 pytest tests/test_environment.py::TestMaybeRetry tests/test_save_utils.py::TestSavingResults::test_states_to_outputs_includes_retry_metadata -q
• uv run --python 3.12 ruff check verifiers/utils/async_utils.py verifiers/utils/save_utils.py verifiers/types.py tests/test_environment.py tests/test_save_utils.py
• uv run --python 3.12 python -m py_compile verifiers/utils/async_utils.py verifiers/utils/save_utils.py verifiers/types.py tests/test_environment.py tests/test_save_utils.py
• git diff --check

Note: the first local pytest attempt under Python 3.13 hit a macOS system policy denial while importing the pandas native extension. I reran the focused tests under Python 3.12, which is part of the project CI matrix.

---

Note

Low Risk
Adds optional serialized metadata on an existing retry path; state_to_output now rejects non-JSON-serializable retry values.

Overview
When evals use --max-retries, saved rollout outputs in results.jsonl now include a retry block whenever a rollout actually retried, so transient infra failures are visible after the run—not only in logs.

maybe_retry records per-attempt error summaries (including retryable state["error"] and grouped rollouts, with state_index when one member of a group triggers retry), then attaches a summary with attempts, exhaustion, elapsed time, and events. state_to_output persists retry into RolloutOutput, with new RetryData / RetryEvent types and docs/skill guidance for post-run debugging.

^{Reviewed by Cursor Bugbot for commit 129847c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Persist retry diagnostics in rollout outputs

• Introduces RetryData and RetryEvent TypedDicts in verifiers/types.py to represent retry metadata, including attempt count, exhaustion status, elapsed time, and per-attempt error summaries.
• Refactors maybe_retry in async_utils.py to track retry lifecycle and attach a retry block to returned states on both successful retries and exhaustion; for list results, each event records the state_index of the triggering error.
• Updates state_to_output in save_utils.py to pass through the retry field from state to the serialized RolloutOutput, raising ValueError if not JSON-serializable.

^{Macroscope summarized 129847c.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 17030c0. Configure here.}