Skip to content

Rebuttal rerun: real turns, task-level splits, fixed controls, persistence analysis - #20

Open
andre15silva wants to merge 34 commits into
mainfrom
rebuttal-rerun-turnfix
Open

andre15silva wants to merge 34 commits into
mainfrom
rebuttal-rerun-turnfix

Conversation

@andre15silva

Copy link
Copy Markdown
Member

Rebuttal rerun: real agent turns, task-level splits, fixed controls, persistence analysis

Reruns every step-dependent experiment after the turn-label fix (#18, 37bdf12) and fixes four problems found along the way. Full write-up with all tables: reports/rebuttal_rerun.md.

Code changes

  • Shuffled-label control scored on real test labels (fa41d05). Both --shuffle-labels modes also shuffled y_test, which pins test AUC at ~0.5 regardless of what the probe learned. Now only train/val labels are shuffled. Adds a regression test for both shuffle units.
  • Locked results/weights merge in run_final (05b8f24). Per-layer jobs of the same run read-modify-wrote shared results.pt/weights.pt without coordination: they could read a half-written file or silently drop another job's layer. Now uses an fcntl lock (works across nodes on /proj) and atomic renames. Adds a multi-process test.
  • Split by task on both benchmarks; --group-by removed (ff6b081). project grouping meant repository on Verified (12 groups, a single repository for validation) but base commit on Pro, so the benchmarks were split on different units. All runs of a task now share one group.
  • Paper figures (b1fa6e9). After-edit versions of the AUC summary, transfer and per-layer plots. The token-shuffle control is drawn as dash-dot curves in the layer plots and as markers on the summary bars.
  • Analysis scripts and report (500c619, 4828a49): label/turn checks, k=0 identity check, persistence analysis with a task-level bootstrap, report tables, and regenerated paper/*_table.tex.

Results (details and CIs in the report)

  • Current-state probes (k=0): 0.69–0.84 best-layer test AUC on held-out tasks, mostly at or above the paper.
  • Lookahead: labels rarely change at short horizons (under 6% of tokens flip within 3 turns), so the persistence baseline is strong (0.65–0.98 AUC at k=1).
    • The probe beats it clearly for reduces_failing (both models on Pro; Qwen on Verified at k = 5–25), Laguna Pro correct (k = 5–15), and Laguna Verified compiles (k ≥ 5).
    • It is clearly worse for compiles on Pro.
    • Elsewhere the two can't be distinguished.
  • After-edit targets: lookahead AUC has a median of 0.62, with 14 of 128 cells below 0.5.
  • Controls:
    • Fixed token shuffle: 0.51–0.55.
    • Trajectory shuffle: 0.56–0.78 for correct, has_regressions and reduces_failing. Much of the signal is tied to position in the trajectory.
  • Qwen Pro layer 0 (the embedding layer) reaches 0.75 on compiles and 0.84 on has_regressions, so those two labels are partly predictable from token identity alone.

Checks

  • Full test suite passes (141 tests at ff6b081; new tests for the shuffle, merge lock and task grouping).
  • k=0 identity check: y, sample_id and the edit index are identical to the pre-fix snapshots; group_id equals the task ID.
  • Completeness audit: every result file has all 5 layers (4,336 items), and the legacy results are complete too.

Not in git (too large, or gitignored): results/ (old results kept as *_legacyturns), cache/, snapshots/, paper/figures/.

🤖 Generated with Claude Code

https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX

andre15silva and others added 30 commits September 28, 2026 15:19
Both --shuffle-labels modes shuffled y_test along with y_train/y_val, which
pins test AUC at ~0.5 regardless of what the probe learned and makes the
control uninformative. Shuffle only train and val (val drives early stopping
on the null labels) and evaluate against the real test labels. Adds a
regression test covering token and trajectory shuffles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Per-layer final jobs of the same run read, update and rewrite shared
results.pt/weights.pt without coordination, so a job could read a
half-written file (EOFError) or silently drop another job's layer. Do the
read-merge-write under an fcntl lock (works across nodes on /proj) and write
atomically via rename. Adds a multi-process test with realistic payloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
--group-by project cut the group id at its last '-'. On SWE-bench Verified
that gave the repository (12 groups, a single repository for validation); on
SWE-bench Pro it only dropped the version suffix, merging tasks that share a
base commit. The two benchmarks were therefore split on different units.

Use the task (instance) id as the group, shared by all runs of a task, so
every run of a task lands in the same split on both benchmarks, and remove
the option to group by repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Label/turn checks, k=0 identity check, persistence analysis with task-level
bootstrap, and report tables for the rerun with real agent turns and
task-level splits. Report: reports/rebuttal_rerun.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
run_paper_figures.py now also writes after-edit versions of the AUC summary
barplot, the transfer barplot and the per-layer AUC plots (from the
*_after_edit eval runs), and draws each series' token-shuffled control
(*_pooled_shuffled, or its after-edit eval) as a dash-dot curve in the layer
plots and as a marker on the summary bars.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TmWo2FP8pBbmrH5DaGARcz
…eport

auc_table.tex, calibration_table.tex and transfer_table.tex now come from the
rerun with real agent turns, task-level splits and new sweeps; the Shuffled
column is the fixed token-shuffle control scored on real test labels.
The report gains a precise definition of the after-edit filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Checks that y_original (label[t]) and the lookahead target
(turn_label[step_idx + k]) agree with each other and with the k=0 turn
labels in every label file and window-50 shift cache: no mismatches.
Adds the per-k fraction of tokens/trajectories with y_{t+k} != y_t and a
turn-level persistence view. The low Pro persistence AUC at window 50
comes from very few positive turns plus token weighting, and the
window-50 probe advantages on Pro do not replicate at window 15; the
report summary now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUQN3pUiFsQaGDFBbyw
E2: the k=0 probe's scores as a forecast of y_{t+k}. E16: within y_t=0/1,
the k-trained probe against a trajectory-shuffle probe trained per k (168
new runs at the validation-best layer, mirroring each real probe's submit
line) and two position baselines (causal: turn and token index; non-causal:
rel_pos). E17: k=0 selectivity against token and trajectory shuffles, layers
chosen on validation against real labels. Task-level bootstrap CIs, paired
differences.

Report: layer 0 is the output of the first transformer block (extractors use
hidden_states[L + 1]), not the embedding layer; the persistence tables
bootstrap tasks, not "instances"; sentences about how runs were made now say
whether they describe the submitted (legacy) runs or the rerun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval-only stratification of the within-y_t=0 result: pre-edit tokens
(y_{t+1}=0), eventual outcome (last-turn label y_T), both, and y_T=0, with
the y_T oracle AUC, per-stratum token/positive/task counts and an
underpowered flag (<20 test tasks with a positive). Reuses the E16 scorers at
the same layers (lookahead scoring refactored into reusable functions; S0
reproduces E16 exactly). Also compares y_T for `correct` with the
final-patch evaluation, describes what tokens make up a turn, and notes the
low power of the y_t=1 stratum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Every Qwen trajectory (Verified and Pro) was tokenized with the "role: content"
fallback in _tokenize_trajectory, and extraction runs on those token_ids, so
the Qwen hidden states were computed without the chat template and without
reasoning. Documented as found, not fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Keeps what is known (scope, likely cause, reasoning handling, Laguna asymmetry,
re-extraction pitfalls and cost estimates) for a future rebuttal; not fixed by
decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval-only: the rerun's k=0 probes on Pro at their validation-best layers
(as in E17), AUC per language (repo_language) and per repo, stratified
same-language / same-repo AUC, language-only and repo-only baselines,
task-bootstrap CIs, and the Verified-trained compiles probe on Pro by
language. Figures in reports/rerun/figures/e11_*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Fig B now shows the repo-only baseline; in-plot titles removed; filled
markers without the per-cell power legend (every cell is underpowered); fig C
labels both pooled lines and stacks the two models at single-column width;
fig A at single-column width.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Same as E11 on SWE-bench Verified (Python-only, so repository strata only):
pooled vs same-repo AUC, repo-only baseline, per-repo AUC and counts, and
the submitted runs' repository-split AUC for context (stored results; test
repos astropy/django/sphinx). Figure e11b_verified_compiles_by_repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Brief P-FIG: tab:auc_roc with a trajectory-shuffle column, tab:transfer,
the summary barplot with token- and trajectory-shuffle markers, the layer
grid, the transfer barplot, the lookahead figures with oracle persistence and
the E2 k=0 curve, and the new horizon figures (E16, y_t = 0). Paths mirror the
paper repo. _weighted_mean now drops single-class bins; only Qwen's
Verified->Pro transfer rows change. The lookahead reference curves come from
scripts/rebuttal_paper_lookahead_refs.py (32 CPU jobs, under a minute each).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… (SEL-CI)

paper_rebuttal_candidate/transfer: transfer probes re-scored with one pooled
AUC over the target's task-split test tokens. Token sets and the binned values
are checked against the in-dist rows and the current tab:transfer. Includes
paired task-bootstrap CIs for transfer - in-dist, and a candidate table and
barplot. paper_rebuttal_candidate/selectivity: real vs trajectory- and
token-shuffle probes at their own best test layers, with paired CIs. Eval
only, about 10 CPU core-hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Adds --layer-rule/--layer and --transfer-layer-rule/--transfer-layer to
rebuttal_per_language.py (asserted against the stored test AUC and T-POOL), and
--root/--out/--show-layer to the E11 figure script. Outputs go to
paper_rebuttal_candidate/lang: Pro and Verified at test-best layers, Qwen/Pro
compiles and has_regressions also at the validation-best layer, and the three
appendix figures. Eval only, about 1 CPU core-hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Counts tasks, trajectories and hidden states per split for k=0, k_max=15 and
k_max=50 on the rerun, checked against the stored n_train/n_val/n_test and
the lookahead figure labels. Rebuilds the submitted (legacy) populations from
snapshots/ to explain the 1.1M vs 145k in the submitted Fig 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Adds --no-strat to rebuttal_per_language_figures.py; figures from the
E11-TEST data in paper_rebuttal_candidate/lang/figures_nostrat/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Checks that all 80 task-split sweeps ran 20 finished trials and that every
rerun training job used the matching sweep's best config; regenerates
tab:probe-hparams at layer 21 into paper_rebuttal_candidate/hparams/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval only: the rerun's k=0 pooled weights on the first turn after each edit, dropping edits whose
command also ran something (Python file-rewriting scripts kept; strict variant drops them too).
Pooled AUC with 95% task-bootstrap CI at the tab:auc_roc best layer; candidate tab:after_edit_k0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… run (E9, candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…(FIG3, candidate)

Token shuffle becomes a black-filled circle (was a black tick); --fig3 writes only
this figure to paper_rebuttal_candidate/fig3/. No bars within 0.01 AUC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…edit (EI, candidate)

New linear probes at the k=0 probe's validation-best layer, weight decay chosen on
validation; baselines persistence, k=0 probe, position, edit counts, trajectory
shuffle, prompt-only, y_T oracle; populations all / y_t=0 / y_t=0 and y_T=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
andre15silva and others added 4 commits November 16, 2026 12:20
… text rule (BON, candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…ctories (PRUNE, candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…action baseline (USE, candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… in BON and USE (VB, candidate)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant