Repository navigation
Rebuttal rerun: real turns, task-level splits, fixed controls, persistence analysis - #20
Open
andre15silva wants to merge 34 commits into
Open
andre15silva wants to merge 34 commits into
andre15silva wants to merge 34 commits into
Conversation
Both --shuffle-labels modes shuffled y_test along with y_train/y_val, which pins test AUC at ~0.5 regardless of what the probe learned and makes the control uninformative. Shuffle only train and val (val drives early stopping on the null labels) and evaluate against the real test labels. Adds a regression test covering token and trajectory shuffles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Per-layer final jobs of the same run read, update and rewrite shared results.pt/weights.pt without coordination, so a job could read a half-written file (EOFError) or silently drop another job's layer. Do the read-merge-write under an fcntl lock (works across nodes on /proj) and write atomically via rename. Adds a multi-process test with realistic payloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
--group-by project cut the group id at its last '-'. On SWE-bench Verified that gave the repository (12 groups, a single repository for validation); on SWE-bench Pro it only dropped the version suffix, merging tasks that share a base commit. The two benchmarks were therefore split on different units. Use the task (instance) id as the group, shared by all runs of a task, so every run of a task lands in the same split on both benchmarks, and remove the option to group by repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Label/turn checks, k=0 identity check, persistence analysis with task-level bootstrap, and report tables for the rerun with real agent turns and task-level splits. Report: reports/rebuttal_rerun.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
run_paper_figures.py now also writes after-edit versions of the AUC summary barplot, the transfer barplot and the per-layer AUC plots (from the *_after_edit eval runs), and draws each series' token-shuffled control (*_pooled_shuffled, or its after-edit eval) as a dash-dot curve in the layer plots and as a marker on the summary bars. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TmWo2FP8pBbmrH5DaGARcz
…eport auc_table.tex, calibration_table.tex and transfer_table.tex now come from the rerun with real agent turns, task-level splits and new sweeps; the Shuffled column is the fixed token-shuffle control scored on real test labels. The report gains a precise definition of the after-edit filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX
Checks that y_original (label[t]) and the lookahead target
(turn_label[step_idx + k]) agree with each other and with the k=0 turn
labels in every label file and window-50 shift cache: no mismatches.
Adds the per-k fraction of tokens/trajectories with y_{t+k} != y_t and a
turn-level persistence view. The low Pro persistence AUC at window 50
comes from very few positive turns plus token weighting, and the
window-50 probe advantages on Pro do not replicate at window 15; the
report summary now says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUQN3pUiFsQaGDFBbyw
E2: the k=0 probe's scores as a forecast of y_{t+k}. E16: within y_t=0/1,
the k-trained probe against a trajectory-shuffle probe trained per k (168
new runs at the validation-best layer, mirroring each real probe's submit
line) and two position baselines (causal: turn and token index; non-causal:
rel_pos). E17: k=0 selectivity against token and trajectory shuffles, layers
chosen on validation against real labels. Task-level bootstrap CIs, paired
differences.
Report: layer 0 is the output of the first transformer block (extractors use
hidden_states[L + 1]), not the embedding layer; the persistence tables
bootstrap tasks, not "instances"; sentences about how runs were made now say
whether they describe the submitted (legacy) runs or the rerun.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval-only stratification of the within-y_t=0 result: pre-edit tokens
(y_{t+1}=0), eventual outcome (last-turn label y_T), both, and y_T=0, with
the y_T oracle AUC, per-stratum token/positive/task counts and an
underpowered flag (<20 test tasks with a positive). Reuses the E16 scorers at
the same layers (lookahead scoring refactored into reusable functions; S0
reproduces E16 exactly). Also compares y_T for `correct` with the
final-patch evaluation, describes what tokens make up a turn, and notes the
low power of the y_t=1 stratum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Every Qwen trajectory (Verified and Pro) was tokenized with the "role: content" fallback in _tokenize_trajectory, and extraction runs on those token_ids, so the Qwen hidden states were computed without the chat template and without reasoning. Documented as found, not fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Keeps what is known (scope, likely cause, reasoning handling, Laguna asymmetry, re-extraction pitfalls and cost estimates) for a future rebuttal; not fixed by decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval-only: the rerun's k=0 probes on Pro at their validation-best layers (as in E17), AUC per language (repo_language) and per repo, stratified same-language / same-repo AUC, language-only and repo-only baselines, task-bootstrap CIs, and the Verified-trained compiles probe on Pro by language. Figures in reports/rerun/figures/e11_*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Fig B now shows the repo-only baseline; in-plot titles removed; filled markers without the per-cell power legend (every cell is underpowered); fig C labels both pooled lines and stacks the two models at single-column width; fig A at single-column width. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Same as E11 on SWE-bench Verified (Python-only, so repository strata only): pooled vs same-repo AUC, repo-only baseline, per-repo AUC and counts, and the submitted runs' repository-split AUC for context (stored results; test repos astropy/django/sphinx). Figure e11b_verified_compiles_by_repo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Brief P-FIG: tab:auc_roc with a trajectory-shuffle column, tab:transfer, the summary barplot with token- and trajectory-shuffle markers, the layer grid, the transfer barplot, the lookahead figures with oracle persistence and the E2 k=0 curve, and the new horizon figures (E16, y_t = 0). Paths mirror the paper repo. _weighted_mean now drops single-class bins; only Qwen's Verified->Pro transfer rows change. The lookahead reference curves come from scripts/rebuttal_paper_lookahead_refs.py (32 CPU jobs, under a minute each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… (SEL-CI) paper_rebuttal_candidate/transfer: transfer probes re-scored with one pooled AUC over the target's task-split test tokens. Token sets and the binned values are checked against the in-dist rows and the current tab:transfer. Includes paired task-bootstrap CIs for transfer - in-dist, and a candidate table and barplot. paper_rebuttal_candidate/selectivity: real vs trajectory- and token-shuffle probes at their own best test layers, with paired CIs. Eval only, about 10 CPU core-hours. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Adds --layer-rule/--layer and --transfer-layer-rule/--transfer-layer to rebuttal_per_language.py (asserted against the stored test AUC and T-POOL), and --root/--out/--show-layer to the E11 figure script. Outputs go to paper_rebuttal_candidate/lang: Pro and Verified at test-best layers, Qwen/Pro compiles and has_regressions also at the validation-best layer, and the three appendix figures. Eval only, about 1 CPU core-hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Counts tasks, trajectories and hidden states per split for k=0, k_max=15 and k_max=50 on the rerun, checked against the stored n_train/n_val/n_test and the lookahead figure labels. Rebuilds the submitted (legacy) populations from snapshots/ to explain the 1.1M vs 145k in the submitted Fig 6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Adds --no-strat to rebuttal_per_language_figures.py; figures from the E11-TEST data in paper_rebuttal_candidate/lang/figures_nostrat/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Checks that all 80 task-split sweeps ran 20 finished trials and that every rerun training job used the matching sweep's best config; regenerates tab:probe-hparams at layer 21 into paper_rebuttal_candidate/hparams/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Eval only: the rerun's k=0 pooled weights on the first turn after each edit, dropping edits whose command also ran something (Python file-rewriting scripts kept; strict variant drops them too). Pooled AUC with 95% task-bootstrap CI at the tab:auc_roc best layer; candidate tab:after_edit_k0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… run (E9, candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…(FIG3, candidate) Token shuffle becomes a black-filled circle (was a black tick); --fig3 writes only this figure to paper_rebuttal_candidate/fig3/. No bars within 0.01 AUC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…edit (EI, candidate) New linear probes at the k=0 probe's validation-best layer, weight decay chosen on validation; baselines persistence, k=0 probe, position, edit counts, trajectory shuffle, prompt-only, y_T oracle; populations all / y_t=0 / y_t=0 and y_T=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…ate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…idate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… text rule (BON, candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…ctories (PRUNE, candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
…action baseline (USE, candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
… in BON and USE (VB, candidate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fbTUN6hnCgxz8r6vTGkkk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuttal rerun: real agent turns, task-level splits, fixed controls, persistence analysis
Reruns every step-dependent experiment after the turn-label fix (#18,
37bdf12) and fixes four problems found along the way. Full write-up with all tables:reports/rebuttal_rerun.md.Code changes
fa41d05). Both--shuffle-labelsmodes also shuffledy_test, which pins test AUC at ~0.5 regardless of what the probe learned. Now only train/val labels are shuffled. Adds a regression test for both shuffle units.run_final(05b8f24). Per-layer jobs of the same run read-modify-wrote sharedresults.pt/weights.ptwithout coordination: they could read a half-written file or silently drop another job's layer. Now uses anfcntllock (works across nodes on/proj) and atomic renames. Adds a multi-process test.--group-byremoved (ff6b081).projectgrouping meant repository on Verified (12 groups, a single repository for validation) but base commit on Pro, so the benchmarks were split on different units. All runs of a task now share one group.b1fa6e9). After-edit versions of the AUC summary, transfer and per-layer plots. The token-shuffle control is drawn as dash-dot curves in the layer plots and as markers on the summary bars.500c619,4828a49): label/turn checks, k=0 identity check, persistence analysis with a task-level bootstrap, report tables, and regeneratedpaper/*_table.tex.Results (details and CIs in the report)
Checks
ff6b081; new tests for the shuffle, merge lock and task grouping).y,sample_idand the edit index are identical to the pre-fix snapshots;group_idequals the task ID.Not in git (too large, or gitignored):
results/(old results kept as*_legacyturns),cache/,snapshots/,paper/figures/.🤖 Generated with Claude Code
https://claude.ai/code/session_011yqn6W82r4UrrQZPxEhZaX