Skip to content

feat(search): Refit the attention model on our own games, split by game (+8 ±8, unresolved at the cap) - #266

Closed
aywrite wants to merge 1 commit into
masterfrom
search/attention-refit
Closed

aywrite wants to merge 1 commit into
masterfrom
search/attention-refit

Conversation

@aywrite

@aywrite aywrite commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

Played twice and unresolved both times: +8 ±8 over 4,000 games, then +1 ±16 over 1,000 on the rebased head. Opened so the measurement can be read where the code is; whether it merges is a separate decision.

The thirteen ATTENTION_* integers were fitted on the bench's eighteen roots with train and holdout split by position. This vector was fitted on a depth 8 ledger over 1,428 positions from 1,340 of our own games, the halves split by source game, with scripts/fit_attention.py --group-by (#261). On rows the gates can reach it reads a held out auc of 0.7674 ± 0.0093 against 0.7468 for the old vector on the same rows, +0.0206 ± 0.0029 paired.

The two thresholds move with the vector. Each was frozen on the training half to dead the same share of depth 4+ rows as the old one, then read on the other half: −4401 deads 86.74% against −4637's 86.52%, −6324 skips 37.73% against −7954's 37.64%. The schedule (LATE_MOVE_THRESHOLD, DEEP_REDUCTION, the replay) is unchanged. Bench 4,105,923 against master's 4,102,609.

The match

Registered before the first batch: this commit against master 4b1b80f, sprt [0, 10] at 10+0.1 on 8moves_v3, 500 game batches carried as pair counts, cap 4,000.

batch run pooled LLR
1 35097085635 +15 ±22 0.81
2 35099778872 +9 ±16 0.55
3 35102556925 +7 ±13 0.56
4 35105385931 +4 ±11 −0.40
5 35108328421 +4 ±10 −0.36
6 35111303763 +6 ±9 0.66
7 35114256601 +7 ±9 1.20
8 35117197566 +8 ±8 1.67

Pooled pairs by the candidate's score: 154, 446, 723, 510, 167. Neither bound reached at the cap, so the registration says not shipped. An auc gap of 0.02 is worth about eight elo give or take eight, and a [0, 10] test of this size cannot tell that from nothing. A longer run is a new registration.

Notes

  • The games above were played as 97061c1 on 4b1b80f. This head is that commit rebased onto a18334d on request, after the match: the constants moved to late_move.rs in the meantime, the integers are the same, and the bench and the swap fixture's budget moved with the tree.
  • The rebase moved the two suites, which the first push missed and ci caught. The tactical suite finds 236 of 300 against 237: WAC.120 is the position spent (this vector plays h1g1 at depth six and does not find g5g6 through depth nine), written into ACCEPTED_LOSSES with that reason and running out at 0.6.0. The strategic suite scores 99,670 of 149,703 against 100,176, half a percent down.
  • A 1,000 game sprt [0, 10] on this head against a18334d (run 35210387073): +1 ±16, LLR −0.55, inconclusive, pairs 35, 119, 198, 103, 45. Consistent with the 4,000 games above and with an effect of a few elo either side of nothing. Nothing here supports merging.
  • Both cargo test runs, fmt, clippy and pre-commit pass. The tests pinning the score arithmetic carry the new integers; the swap fixture's node budget moves because the swap is at depth nine on this tree.
  • Not for merge under the registration it was played on. If it is to be played again, the pooled pairs above can be carried forward or a fresh registration can start clean.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Speed against a18334d9

Both sides built and run on this runner in this job, the way
scripts/speed.sh measures a perf commit. The change is between
medians, and the spread beside it is what the change has to be
read against. The node counts are the search's: they move when
the search does, and a speed change leaves them alone.

round     base nps  candidate nps
    1      8201707        8307616
    2      8225221        8441436
    3      8457609        8068405
    4      8430844        8430205
    5      8405969        8373265
    6      8498712        8378289
    7      8477743        8234094
    8      8344979        8424773
    9      8462494        8363509

             nodes    time  median nps  fastest nps
base       4102609  0.49 s     8430844      8498712
candidate  4105923  0.49 s     8373265      8441436
change       +0.1%   +0.8%       -0.7%        -0.7%

the node counts differ, so this is a search change as well as a speed
one. nps only says what a node costs. Time to depth is what the change is
worth at this depth, and whether the new tree is the right one is for
games to say.

Speed: -0.7% (bench nps, 9 interleaved rounds vs a18334d9, spread 4.5%)

@aywrite
aywrite force-pushed the search/attention-refit branch from 97061c1 to dc23564 Compare September 17, 2026 10:10
The thirteen ATTENTION_ integers were fitted on 2026-09-06 over 193,143
rows from the bench's eighteen roots, train and holdout split by
position. Rows from one root share a parent node and most of a feature
vector, so that holdout had seen the training half's neighbours, and its
0.927 was read on the rows the gate cannot reach as much as on the rows it
can. On fresh rows the same vector reads 0.7468 at depth four and up.

This vector was fitted by `scripts/fit_attention.py --group-by` over a
depth 8 ledger recorded one root at a time on 1,428 positions sampled from
1,340 of our own games, 13,697,021 scouts with a 20% row sample fitted,
the halves split by source game. On its held out half it reads 0.7674
+/- 0.0093 at depth four and up, +0.0206 +/- 0.0029 over the old vector
paired on the same rows. A split by position on the same ledger reads
0.7723 +/- 0.0038: on a corpus this wide the position split does not
inflate the figure, it understates the error.

The two thresholds move with the vector. Each is the integer that deads
the same share of the ledger's depth four and up rows, scouts and skipped
together, as the old threshold did, matched on the training half and read
on the other: -4401 deads 86.74% against -4637's 86.52%, at 0.324%
attention against 0.344%; -6324 skips 37.73% against -7954's 37.64%, at
0.076% against 0.091%. The same share of rows is not the same tree, so
the nodes move. The schedule does not: LATE_MOVE_THRESHOLD, DEEP_REDUCTION
and the replay are as they were.

An auc is an offline reading, so this was played before it landed: as
97061c1 on master 4b1b80f, sprt [0, 10] at 10+0.1 on 8moves_v3, eight
batches of 500 games carried as pair counts. +8 +/-8 over 4,000 games,
likelihood ratio 1.67 of 2.94, unresolved at the registered cap, which
is not a pass. This is that commit rebased onto master after the match,
the constants having moved to late_move.rs in the meantime; the integers
are the same and the tree under them is master's.

The tests pinning the score arithmetic carry the new integers, the
threshold tests solve alpha and beta at eight a point and one a point
rather than seven and four, and the swap fixture's node budget moves to
2,350,000 to sit inside depth ten's swap on this tree.

The two suites move with the gates. The tactical suite finds 236 of 300
at depth six against 237: WAC.120 is the position spent, where this
vector plays h1g1 and does not find g5g6 through depth nine, and it is
written into ACCEPTED_LOSSES with that reason, running out at 0.6.0. The
strategic suite scores 99,670 of 149,703 against 100,176, half a percent
down, spread over the themes rather than sitting in one.

Bench: 4105923
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@aywrite
aywrite force-pushed the search/attention-refit branch from dc23564 to ae03c53 Compare September 17, 2026 10:24
@aywrite aywrite closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant