fix: accept Claude native trajectories

[vincentkoc]

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

• add claude-code to the real-trajectory harness allowlist
• assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

• coverage: 4/4
• score: 0.75
• exact passes: 3
• trajectory complete: true
• trajectory status: real for all four tasks
• infrastructure failures: 0
• exclusion reason: exploratory_subset

Tests

• python -m pytest -q passes locally (444 passed, 5 skipped)
• python -m ruff check clawbench app.py scripts tests passes locally
• live Claude Code native run aggregates as real harness events

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 28, 2026, 10:02 PM ET / August 29, 2026, 02:02 UTC.

ClawSweeper review

What this changes

The PR marks Claude Code stream-json trajectories as real harness events and adds coverage that every pinned native harness receives that classification.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

Keep open for maintainer review: current main still excludes Claude Code from real-trajectory eligibility, while this narrow fix changes the benchmark’s evidence-comparability policy and must be rebased before it can merge.

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is a focused, source-supported correction with a focused test; rebase resolution and a maintainer evidence-policy decision remain before merge.
Proof confidence	🌊 off-meta tidepool	Not applicable: The captured PR body reports an after-fix four-task Claude Code native run whose aggregation marked every trajectory real; because the author is a repository MEMBER and this change has no authority-boundary trigger, the external-contributor proof gate does not apply.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The captured PR body reports an after-fix four-task Claude Code native run whose aggregation marked every trajectory real; because the author is a repository MEMBER and this change has no authority-boundary trigger, the external-contributor proof gate does not apply.
Evidence reviewed	7 items	Current main still omits the requested eligibility: Current main includes Claude Code in the pinned native harness matrix but its real-trajectory allowlist contains only OpenClaw, Hermes, and Codex, so this requested behavior is not already implemented. Claude Code has a production trajectory path: The native runtime parses Claude Code stream-json into a trajectory, validates a successful terminal event and model identity, and returns a real trajectory status when those checks pass. Aggregation makes the allowlist outcome material: Native aggregation requires a real-harness trajectory mode and real status for every required row; otherwise it assigns trajectory_unavailable.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native runner captures each harness’s execution trace and writes a normalized trajectory record. Aggregation uses the recorded trajectory mode and status to decide whether a completed run can contribute to native benchmark results.

flowchart LR
  A[Native harness output] --> B[Claude Code stream-json parser]
  B --> C[Normalized trajectory record]
  C --> D[Aggregation eligibility policy]
  D --> E[Native benchmark result]

Loading

Decision needed

Question	Recommendation
Should validated Claude Code stream-json trajectories be treated as evidence-comparable real harness events for native benchmark eligibility?	Approve after rebase: Resolve the current-main conflicts and accept validated Claude Code stream-json records as real native evidence.

Why: The implementation is small and current source confirms the classification gap, but admitting a trace format into benchmark eligibility is a policy decision about result comparability rather than a purely mechanical repair.

Before merge

• Resolve merge risk (P1) - The verified three-way merge conflicts with current main in both changed files, so this exact head cannot land without a rebase or manual conflict resolution.
• Resolve merge risk (P1) - Classifying reconstructed Claude Code stream-json records as comparable real harness evidence changes which native results can be eligible; maintainers must explicitly accept that evidence standard.
• Complete next step (P2) - A maintainer must decide the evidence-comparability policy and resolve the current-main conflict; neither is a safe automated repair choice.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	2 files; production 1 line changed, tests 12 lines added	The implementation is narrowly scoped, with the remaining merge blocker concentrated in the same two files.

Merge-risk options

Maintainer options:

1. Rebase and confirm evidence policy (recommended)
   Resolve the verified conflicts, rerun the focused native trajectory checks, and obtain explicit approval for Claude Code trace comparability before merge.
2. Retain the existing exclusion
   Pause this policy change if reconstructed stream-json traces should not yet be eligible for native comparison.

Technical review

Best possible solution:

Rebase the two focused hunks onto current main, preserve the Claude Code eligibility correction, and land it only after maintainers affirm that the validated stream-json reconstruction meets the benchmark’s real-evidence standard.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main writes validated Claude Code trajectories but omits Claude Code from the real-trajectory allowlist that aggregation requires for eligibility. The submitted body also reports the intended after-fix native control-run outcome.

Is this the best way to solve the issue?

Unclear pending policy approval: adding Claude Code is the narrowest code repair, but maintainers must first decide whether its reconstructed stream-json trace meets the same benchmark-evidence standard as accepted harnesses.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 7e117cba3bd8.

Labels

Label justifications:

• P2: This corrects a research-facing native-result eligibility path with limited user-facing blast radius.
• merge-risk: 🚨 other: The change alters the benchmark policy that determines whether a completed Claude Code trace may count as real native evidence.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The captured PR body reports an after-fix four-task Claude Code native run whose aggregation marked every trajectory real; because the author is a repository MEMBER and this change has no authority-boundary trigger, the external-contributor proof gate does not apply.

Evidence

What I checked:

• Current main still omits the requested eligibility: Current main includes Claude Code in the pinned native harness matrix but its real-trajectory allowlist contains only OpenClaw, Hermes, and Codex, so this requested behavior is not already implemented. (scripts/native_eval/models.py:102, 7e117cba3bd8)
• Claude Code has a production trajectory path: The native runtime parses Claude Code stream-json into a trajectory, validates a successful terminal event and model identity, and returns a real trajectory status when those checks pass. (scripts/native_eval/harness_trajectories.py:321, 7e117cba3bd8)
• Aggregation makes the allowlist outcome material: Native aggregation requires a real-harness trajectory mode and real status for every required row; otherwise it assigns trajectory_unavailable. (scripts/native_eval/aggregate.py:907, ac1c974c1274)
• Claude Code protocol is an affirmative dependency signal: The native harness directly executes Claude Code with its stream-json output format, which is the protocol parsed by the changed policy path. (scripts/native_eval/harnesses.py:876, ac1c974c1274)
• Current-main merge conflict: A three-way merge using the verified base, current main, and this exact head conflicts in both the allowlist file and native-runner test file; the branch requires a rebase or manual conflict resolution. (tests/test_native_eval_runner.py:65, ac1c974c1274)
• Feature-history provenance: The native evaluation surface was introduced and subsequently developed through commits by Vincent Koc, while current-main native-evaluation fixes were recently integrated by Peter Steinberger. (scripts/native_eval/models.py:102, 7e117cba3bd8)

Likely related people:

• vincentkoc: Introduced and developed the native evaluation, trajectory, and aggregation surfaces on current history, including the base commit immediately preceding this PR. (role: native-evaluation feature owner; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/models.py, scripts/native_eval/harness_trajectories.py, scripts/native_eval/aggregate.py)
• steipete: Recently integrated current-main native-evaluation validity and reasoning-effort changes adjacent to the merge conflict. (role: recent native-evaluation contributor; confidence: medium; commits: 01211da61b64, 7e117cba3bd8; files: scripts/native_eval/aggregate.py, scripts/native_eval/models.py, tests/test_native_eval_runner.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Rebase the two changed files onto current main and rerun the focused native-evaluation tests.
• Obtain explicit maintainer approval that validated Claude Code stream-json reconstruction is comparable real benchmark evidence.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (65 earlier review cycles; latest 8 shown)

• reviewed 2026-08-09T15:19:11.621Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T19:45:59.678Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T22:01:21.887Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-11T23:10:51.821Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-12T01:22:50.827Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T10:11:27.098Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T11:15:30.965Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-21T23:55:18.903Z sha ac1c974 :: needs maintainer review before merge. :: none