Repository navigation
feat(eval): Refit the eight mobility weights on the whole archive - #247
Merged
Merged
Conversation
The tuner's holds were a ladder downward. Each term earned a hold as it was fitted and a fit of the newest term named every hold below it, so the shelter was fitted under two holds and the pawn structure under three. A refit of a term already in the tree cannot be expressed that way: every other term is then older than the fit rather than newer, and the ones above it have no flag. `--hold-pawn` is the upper end. A mobility refit now passes `--hold-tables --hold-shelter --hold-pawn` and moves its eight weights alone, which is what lets a match read the eight and not the twenty four. Which holds an arm passes follows from the one term it means to move, and not from where that term sits in the vector. The reason for the rule is on the record and cost a reading once already: `1b0862a` found half of the king safety fit's apparent gain to be a mobility refit that no hold had stopped. A mobility refit under the old flags would leave the pawn structure free and repeat that with the terms the other way round. Scripts and documentation only. The engine is untouched and the bench is the one master carries. Bench: 4089373 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WgBrtJt4pEXoB2WjNBmVrd
… rows `Corpus` refused any named sealed pair that reached no row of the extraction, on the grounds that the group was then smaller than the file said. That conflated two things and refused the harmless one. A pair the corpus holds no game for is a seal drawn against another archive. Nothing can be read under it, because the reading would be over games a reader cannot name, so it is still refused and the message now says which fault it is. A pair the corpus does hold whose positions all went elsewhere is ordinary once the archive is large. Every position those two games saw was either claimed by a game with a lower key, which is what keeps a position in one group, or filtered out of the extraction as in check, unsettled or drawn. The pair is sealed and contributes nothing. That is a number to report, and it is reported where the group sizes are, so it cannot pass unseen. The 2026-09-13 mobility corpus has four such pairs in 4,750, which is what found this. The pawn structure fit had none and so never reached it. Bench: 4089373 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mobility was fitted once, in 2026-09-11, over 1,812 games. Six of its eight
weights rounded to nothing and the term shipped as a rook count, with
`SCORED_KINDS` skipping the other three kinds at every leaf. This refit reads
43,675 games and prices all four.
knight 4 / 1 bishop 6 / 3
rook 4 / 4 queen 1 / 7
259 artifacts across 54 runs, whose 5,694,775 post-book plies gave 5,355,792
positions and 2,461,322 quiet rows. The 768 table entries, the six material
values, the fourteen shelter weights and the sixteen pawn structure ones were
all held, so these eight are the only thing that moved. That needed
`--hold-pawn`, which the commit below adds: the holds were a ladder downward,
one per term as it was fitted, and a refit of a term already in the tree has
every other term older than it rather than newer.
The selection group reads -0.000798 against a standard error of 0.000113 at a
design factor of 3.8. The sealed group, 4,747 pairs and 9,452 games drawn from
seventeen strength runs that no fit had read and opened once after the vector
was frozen, reads -0.000510 against 0.000107 at 4.1. Both are outside their
intervals; they are 1.85 standard errors apart, so the selection group
overstates this fit by about a third. The king safety fit's two agreed to 0.68
and the pawn structure fit's to 0.15.
The ridge is zero and here that is not the trap it was for pawn structure: the
vector quantizes identically at zero, 1e-8 and 1e-7, the largest weight is 7,
and every one of the eight slots carries a coefficient in 37% to 70% of the
training rows. There is no thin direction for an unregularised fit to hide in.
The learning curve is flat from an eighth of the training pairs: 0.086962 at
1,593 pairs against 0.086879 at 12,743. The corpus stopped being the
constraint on this term some time ago.
`specs/mobility-cost.md` predicted this vector from a preliminary fit taken
before the pawn structure term existed, at `[5,6,5,1]` and `[0,2,2,7]`. It came
out `[4,6,4,1]` and `[1,3,4,7]`. Both terms read open lines and the question
was whether the newer one had taken the older one's pay; it has not.
## What it costs, which is the other half of the arm
Every weight is now non-zero, so `SCORED_KINDS` has nothing to skip and all
four kinds are counted at every leaf again. Callgrind over the bench with
`mobility_with` forced out of line:
the rook alone, as it shipped 229,040,403 6.00% of the run
four counted, old weights 423,840,322 10.56% tree identical
four priced, this vector 528,497,023 13.11%
The middle row is the counting alone: weights still at zero cannot move a
score, so the tree is the same 4,089,373 nodes and the row is what three more
kinds cost to count. The term is 1.85 times what it was to count and 2.31
times what it was to count and price.
The run is 3,819,405,653 instructions before and 4,030,086,289 after, which is
+5.52%, and the tree shrinks 5.39% to 3,868,987 nodes, so the cost a node is
+11.5%. `specs/mobility.md` allows a leaf term 5% of instructions. This term
has never met that: 15.6% when first fitted, 9.6% once the zero-weight kinds
were skipped, 6.00% today, 13.11% here. The refit more than doubles the
distance from the line.
## What the instruments say
The strategic suite rises 96,532 to 99,938 and eleven of its fifteen themes
rise with it. Open Files and Diagonals is up 562, which is the theme this term
is most nearly about and the one the original mobility arm predicted would
move. AKPC is up 1,080 and Pawn Play in the Center 983. Simplification is down
357, Center Control 349.
The tactical suite falls 231 to 224. That is the tripwire and it had never
moved by more than one before. It is not a gate and it is not noise at that
size.
The two are the trade this arm was asking about, and neither settled it.
The match did, in one batch of 500 at 10+0.1 against the commit below: **+76
with a 95% interval of +50 to +101**, the log likelihood ratio reaching 4.15
against a bound of 2.94, pentanomial 11 / 41 / 78 / 70 / 50. So the term buys
the 5.52% and the seven tactical positions several times over. The first
mobility fit took +12 over 4,000 games; this one takes +76 over 500, on the
same term, from nothing but a corpus twenty four times the size.
## Four tests the refit perturbed
None of them is a node count pin and none was weakened to fit.
`the_mate_distance_survives_a_deeper_warm_search` searched depth four to warm
the table and asserted a mate in two. Whether a given depth finds that mate is
not monotone in the depth and never was: the old weights miss it at three and
find it from four, these find it at three and miss it at four, and every other
depth from three to seven finds it under both. The property in the test's name
is that the distance does not move when the same position is searched again
deeper, so it now asks that of every depth from three to six and requires that
more than two of them see it.
`the_pruning_looks_at_less_of_the_tree` asserted that adding late move pruning
searches fewer nodes. The skip does not have that property: it changes the
tree, and a changed tree changes what the table holds and what the ordering
does with it. Over the bench's eighteen positions at that depth the old
weights never made the skip cost nodes and these make it cost them on three,
this position among them. It is renamed for what it checks, that the switch
reaches the search, and asserts the trees differ.
`a_sample_is_replayed_on_a_cold_table` needs two depths whose answers disagree,
and says so in its own first assertion. Five and two came to the same -13 here.
Five and three disagree under both vectors.
`a_run_records_rows_that_hold_together` needs a run holding all three scout
outcomes. At depth five these weights leave no fail high at any rate. Depth six
at one event in five has all three and is a smaller run than the old one.
The uci session's node budget moves with the tree, which its comment says. The
swap is at depth ten, which now ends at 2,884,594 nodes against the depth
before it at 1,267,976.
Bench: 3868987
Elo: +76 ±25 (sprt [0, 10] passed, 500 games, 10+0.1, vs 2df11cb)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Speed against
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mobility was fitted once, over 1,812 games. Six of its eight weights rounded
to nothing, so
SCORED_KINDSstopped counting three of the four kinds at theleaf and the term shipped as a rook count. This refits it on 43,675 games with
every other term held, and prices all four.
+76 ±25 elo, sprt [0, 10] passed at LLR 4.15 against a bound of 2.94,
settled in the first batch of 500. The baseline is the commit below the
refit, which is the same tree with the old weights, so the difference measured
is the eight weights and nothing else.
Three commits, each measurable on its own.
--hold-pawn. The holds were a ladder downward, one per term as it wasfitted. A refit of a term already in the tree has every other term older than
it rather than newer, and the terms above it had no flag. Without this the
sixteen pawn structure weights would have moved beside the eight and the match
could not say which it measured, which is what
1b0862afound the last time ahold was missing.
A tuner fix.
Corpusrefused any named sealed pair that reached no row ofthe extraction. That conflated a seal drawn against the wrong archive, which
nothing can be read under, with a pair whose positions all went to a
lower-keyed game or were filtered out of the extraction, which is ordinary
once the archive is large. The first is still refused, the second is counted
and reported. No engine code.
The refit. 259 artifacts over 54 runs, 5,355,792 positions, 2,461,322
quiet rows, with the tables, the material, the shelter and the pawn structure
all held.
What it costs, which is the other half of the arm
Every weight is non-zero now, so
SCORED_KINDShas nothing to skip and allfour kinds are counted at every leaf again. Callgrind over the bench with
mobility_withforced out of line:The middle row is the counting alone, with the tree identical because weights
still at zero cannot move a score. The term is 1.85 times what it was to count
and 2.31 times to count and price. The run goes +5.52% in instructions while
the tree shrinks 5.39%, so a node costs +11.5%.
specs/mobility.mdallows a leaf term 5%. Mobility has never met it and thismore than doubles the distance. The games say that is worth paying, which
makes the ceiling the thing under question rather than the term.
What the instruments say
Eleven of the strategic suite's fifteen themes rise. Open Files and Diagonals
is up 562, which is the theme this term is most nearly about and the one the
original mobility arm predicted would move.
The tactical suite falls seven. That is the tripwire and it had never moved by
more than one across any earlier evaluation arm. It is not a gate, and the
match was run knowing it; +76 buys it.
The fit
Selection reads -0.000798 ±0.000113 and the sealed group, 4,749 pairs from
seventeen runs nothing had read, reads -0.000510 ±0.000107. Both outside
their intervals, 1.85 standard errors apart, so selection overstates this fit
by about a third; king safety's two agreed to 0.68 and pawn structure's to
0.15.
The ridge is zero and that is not the trap it was for pawn structure: the
vector quantizes identically at zero, 1e-8 and 1e-7, the largest weight is 7,
and every slot carries a coefficient in 37% to 70% of the training rows.
The learning curve is flat from an eighth of the training pairs. The corpus
stopped being this term's constraint some time ago.
Four tests moved that are not node count pins
None was weakened to fit, and each says something about the refit.
A known mate in two is missed at depth four and found at three, where the old
weights miss it at three and find it at four. Whether a depth finds that mate
was never monotone; the hole moved rather than appeared. The test asserted it
at one depth and now asserts the invariant in its own name, that the distance
does not change, across four depths.
Adding late move pruning now costs nodes on three of the bench's eighteen
positions, where under the old weights it cost nodes on none. That test
asserted the skip searches less, which is not a property the skip has, and now
asserts what its comment says it is for: that the switch reaches the search.
The residual replay needs two depths whose answers disagree, and says so in
its own first assertion; five and two came to the same answer and five and
three disagree under both vectors. The reduction ledger needs a run holding
all three scout outcomes, and at depth five these weights leave no fail high
at any rate.
The uci session's node budget moves with the tree, which its comment says.
The finding worth keeping
The first mobility fit read 1,812 games, priced one piece and took +12 ±8 over
4,000 games. This one read 43,675, priced four and took +76 ±25 over 500.
Nothing about the term changed between them.
So
SCORED_KINDSwas never an optimisation. It was a measurement of howlittle the old corpus knew, and it had three quarters of a term switched off
at the leaf for want of evidence rather than want of value.
Rebased onto
6a2c5c7in a scratch copy before opening this: it appliesclean, the bench is the same 3,868,987 and both suites and both pin sets hold.
🤖 Generated with Claude Code