2/3 Add the redesigned VeRO system (vero/)

[varunursekar]

Second of three stacked PRs splitting #41. Base: pr1-legacy-relocate.

Adds the v0.5 system at the repo root: vero/ (core + harbor eval sidecar, build compiler, runtime/observability), vero-tasks/, runs/. Root README points to legacy/. Pure adds (197 files) + README/.gitignore.

Review after 1/3. 3/3 adds harness-engineering-bench/ on top.

🤖 Generated with Claude Code

Greptile Summary

This PR introduces the complete v0.5 VeRO system (197 new files) covering the core optimization engine, Harbor eval sidecar, build compiler, and runtime/observability stack. The change is a pure addition with no modifications to existing files.

• Evaluation engine + budget ledger: The EvaluationEngine and BudgetLedger implement correct cancellation-safe patterns via a new _drain_write helper that centralises the previously hand-rolled shield/rollback logic; the reserve, refund, and engine exception handlers are all consistent.
• Harbor sidecar: The CanonicalVerifier, EvaluationSidecar, and InferenceGateway are well-structured with atomic writes, timing-safe token comparison, and a principle-of-least-privilege agent/admin boundary. _safe_extract_tar now applies filter=\"data\" matching extract_harbor_session_archive.
• run_result_to_messages in vero/src/vero/utils/tokens.py: The helper reduces every output or content list to its first element; in a Responses API message that carries parallel tool calls, all but the first call are silently discarded.

Confidence Score: 4/5

Safe to merge; the one functional concern is confined to a utility helper used in agent conversation history and does not affect the evaluation engine, budget ledger, or Harbor security boundary.

The evaluation engine, budget ledger, sidecar auth, and Harbor compiler are carefully implemented and the cancellation-safety issues from the previous review cycle have been addressed. The only confirmed defect is in run_result_to_messages, which silently drops all but the first content block from a multi-block Responses API output item — a scenario that arises whenever an agent makes parallel tool calls.

Files Needing Attention: vero/src/vero/utils/tokens.py — the process_item helper truncates multi-block output/content lists

Important Files Changed

Filename	Overview
vero/src/vero/utils/tokens.py	run_result_to_messages silently truncates multi-block output/content to first element, dropping parallel tool calls
vero/src/vero/evaluation/engine.py	Authorization, budget reservation/refund, and cancellation-safe shield patterns are all correctly implemented; previous review issues are resolved
vero/src/vero/evaluation/store/budget.py	New _drain_write helper centralizes the reserve/refund cancellation-safe write pattern; CancelledError propagation is correct in both reserve and refund paths
vero/src/vero/sidecar/auth.py	Admin token generation uses secrets module, atomic write with strict permissions (0o400/0o700), and timing-safe comparison; well-implemented
vero/src/vero/sidecar/verifier.py	CanonicalVerifier candidate selection, re-scoring, and finalization logic is well-structured with idempotent durable writes; baseline floor semantics are correctly handled
vero/src/vero/harbor/build/compiler.py	_safe_extract_tar now includes filter="data" matching session.py's defensive posture; previous review concern is resolved
vero/src/vero/gateway/inference.py	Inference gateway with token authentication, per-scope budgets, and cancellation-safe semaphore + lock pattern; streaming SSE usage extraction handles both Chat and Responses API formats

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
vero/src/vero/utils/tokens.py:22-26
**Multi-block output silently dropped to first item**

When an item has more than one entry in its `output` or `content` list — for example, a Responses API message that contains both a text block and one or more parallel tool-call blocks — `process_item` replaces the entire list with just the first element. Every tool call after the first is silently discarded. When the result is later passed as conversation history to another model invocation, the model will not see that those tool calls were ever made, which will cause the agent to mis-attribute or duplicate tool calls on the next turn.

The test `test_run_result_to_messages_uses_first_content_item` confirms this is the current behavior, but parallel tool-use is a common pattern in the Responses API. If truncation to the first block is intentional (e.g. this function is only ever called in contexts where multi-block output cannot arise), the docstring should state that precondition explicitly.

_{Reviews (31): Last reviewed commit: "Tell the optimizer its results are persi..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​orjson@​3.11.9
	pypi/​wandb@​0.28.0
	pypi/​litellm@​1.92.0
	pypi/​openai@​2.46.0
	pypi/​pydantic@​2.13.4
	pypi/​fastapi@​0.139.0

View full report

[shehabyasser-scale]

Really nice split. Breaking #41 into relocate-legacy / add-system / add-benchmarks makes it reviewable again (Greptile passes each now), keeps the old stack intact under legacy/, and reads as clean pure-adds. I went through the system tree closely against the legacy Harbor stack.

Two things I'd flagged on the earlier v0.5 snapshot are already handled here, thanks for that:

• Baseline floor fails safe now (verifier.py:383, raises NoCandidateError if the seed can't be re-scored rather than shipping it unverified).
• k-anonymity floor defaults to 5 (build/config.py:29, and resolved to 5 under AGGREGATE in evaluation/models.py:707).
• The in-process .veroaccess ACL being gone is fine: it's replaced by the privilege-drop + root-owned session dir isolation. Might be worth a one-line note in the PR that this was intentional.

One integrity concern I'd want resolved before this is relied on for competitive selection:

Infra-classified cases are excluded from the aggregate rather than scored at failure_score. An all-dead "infrastructure" case returns CaseStatus.ERROR (backend.py:1083-1105, the comment there literally says "excluded from the aggregate") and is dropped from informative_scores, so the mean divides by the informative count only (backend.py:1324-1330). The transient-infra classification matches the candidate process's own exception type name + message against a regex allowlist (error_taxonomy.py:131-159: rate.?limit|...|time(?:d.?)?out|connection|...). Budget and auth are detected out of band (good), but transient-infra is not.

If a candidate-attributable failure can reach that path (a candidate that raises ConnectionError, or emits a "timeout"/"connection" message, on a hard case), that case drops out of its own denominator and inflates the mean over the rest. That's the one-sided-selection lever the legacy code deliberately kept off: legacy always zero-filled dead attempts and documented infra retry as trusted-candidate-only. Here there's no trusted-only gate and infrastructure_max_attempts defaults to 3 (on).

Could you confirm whether the routing prevents a candidate-raised exception from being classified transient-infra? If it doesn't, I think the fix is:

1. Score candidate-attributable all-dead cases at failure_score (restore the zero-fill invariant); restrict exclusion to harness-produced signals (coverage gaps, gateway-ledger budget events).
2. Default infrastructure_max_attempts to 1, and gate retry > 1 to trusted / finalization evaluations only.

One more (high), in scale-vero-tasks: a single non-finite metric poisons the whole report. runner.py:154 writes score/metrics unguarded, so a NaN/inf from one case makes vero reject the report and collapse the entire run to the failure value, discarding all per-case data. Suggest enforcing finiteness at the TaskResult boundary.

Smaller, non-blocking follow-ups:

• Whole-run retry replaces rather than merges groups (backend.py:1222), so with aggregate_attempts: best it's a best-of-3 amplifier tripped by one missing task.
• A candidate self-timeout is classed transient-infra and excluded (backend.py:1041-1044) instead of scored at failure_score.
• The legacy operator ERROR alarm and the partly-zero-filled-mean WARNING are gone (raw n_dead_infra/n_clean counts survive as metrics, but there's no grep-able signal in the logs).
• No per-case infra_retry audit marker, so a score that survived N re-rolls looks identical to a first-attempt score.
• DGM / Evolutionary counters + seeded RNG reset on resume (strategy.py:241), silently shifting the search distribution.

Happy to pair on the aggregate/retry piece if that's useful.

[greptile-apps]

Double-cancellation race leaks budget reservation

When asyncio.timeout fires it internally cancels the task, and the resulting CancelledError is correctly caught at line 191. The handler then calls await asyncio.shield(self._persist_failure(...)). If quiesce_agent_evaluations delivers a second task.cancel() while that shielded await is in progress, Python raises a new CancelledError from inside the except asyncio.CancelledError: block before raise EvaluationCancelledError(...) is reached. The engine only catches EvaluationCancelledError and EvaluationExecutionError; a raw CancelledError bypasses both handlers, so neither _record nor budget_ledger.refund is called — the reservation is permanently leaked.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/evaluation/evaluator.py
Line: 191-208

Comment:
**Double-cancellation race leaks budget reservation**

When `asyncio.timeout` fires it internally cancels the task, and the resulting `CancelledError` is correctly caught at line 191. The handler then calls `await asyncio.shield(self._persist_failure(...))`. If `quiesce_agent_evaluations` delivers a second `task.cancel()` while that shielded await is in progress, Python raises a new `CancelledError` *from inside* the `except asyncio.CancelledError:` block before `raise EvaluationCancelledError(...)` is reached. The engine only catches `EvaluationCancelledError` and `EvaluationExecutionError`; a raw `CancelledError` bypasses both handlers, so neither `_record` nor `budget_ledger.refund` is called — the reservation is permanently leaked.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

_persist_failure unshielded in TimeoutError and Exception handlers

The CancelledError handler above (line 193) correctly wraps _persist_failure in asyncio.shield. The TimeoutError and Exception handlers do not. When asyncio.timeout fires, it absorbs the timeout's internal cancellation and re-raises as TimeoutError — but if quiesce_agent_evaluations has also called task.cancel(), that external cancellation is still pending. The first await inside the unshielded _persist_failure (typically await store.save(...) → asyncio.to_thread) will deliver that cancellation. The CancelledError propagates past the raise EvaluationExecutionError(...) line, exits Evaluator.evaluate as a raw CancelledError, bypasses both EvaluationCancelledError and EvaluationExecutionError handlers in the engine, and leaves the budget reservation permanently charged. The same race applies to the except Exception: path.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/evaluation/evaluator.py
Line: 209-238

Comment:
**`_persist_failure` unshielded in `TimeoutError` and `Exception` handlers**

The `CancelledError` handler above (line 193) correctly wraps `_persist_failure` in `asyncio.shield`. The `TimeoutError` and `Exception` handlers do not. When `asyncio.timeout` fires, it absorbs the timeout's internal cancellation and re-raises as `TimeoutError` — but if `quiesce_agent_evaluations` has also called `task.cancel()`, that external cancellation is still pending. The first `await` inside the unshielded `_persist_failure` (typically `await store.save(...)` → `asyncio.to_thread`) will deliver that cancellation. The `CancelledError` propagates past the `raise EvaluationExecutionError(...)` line, exits `Evaluator.evaluate` as a raw `CancelledError`, bypasses both `EvaluationCancelledError` and `EvaluationExecutionError` handlers in the engine, and leaves the budget reservation permanently charged. The same race applies to the `except Exception:` path.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

Multi-block output silently dropped to first item

When an item has more than one entry in its output or content list — for example, a Responses API message that contains both a text block and one or more parallel tool-call blocks — process_item replaces the entire list with just the first element. Every tool call after the first is silently discarded. When the result is later passed as conversation history to another model invocation, the model will not see that those tool calls were ever made, which will cause the agent to mis-attribute or duplicate tool calls on the next turn.

The test test_run_result_to_messages_uses_first_content_item confirms this is the current behavior, but parallel tool-use is a common pattern in the Responses API. If truncation to the first block is intentional (e.g. this function is only ever called in contexts where multi-block output cannot arise), the docstring should state that precondition explicitly.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/utils/tokens.py
Line: 22-26

Comment:
**Multi-block output silently dropped to first item**

When an item has more than one entry in its `output` or `content` list — for example, a Responses API message that contains both a text block and one or more parallel tool-call blocks — `process_item` replaces the entire list with just the first element. Every tool call after the first is silently discarded. When the result is later passed as conversation history to another model invocation, the model will not see that those tool calls were ever made, which will cause the agent to mis-attribute or duplicate tool calls on the next turn.

The test `test_run_result_to_messages_uses_first_content_item` confirms this is the current behavior, but parallel tool-use is a common pattern in the Responses API. If truncation to the first block is intentional (e.g. this function is only ever called in contexts where multi-block output cannot arise), the docstring should state that precondition explicitly.

How can I resolve this? If you propose a fix, please make it concise.