Skip to content

V67: early stopping for the boosting, measured and not built - #167

Merged
dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0
Nov 13, 2026
Merged

dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0

Conversation

@dapiced

@dapiced dapiced commented Nov 13, 2026

Copy link
Copy Markdown
Owner

What

F9, opened by V66: the glass box had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30. The obvious fix is early stopping. This wave measured the fix before writing it, and the measure says no. Nothing changes in training: no figure of any run moves and no golden reference is regenerated.

The trap seen before measuring

The validation rows choose the family. A boosting that also chooses its number of rounds on them would be scored on rows it tuned itself on. So three stops were compared against the 120 fixed rounds:

  • A: at the lowest error on the run's validation rows;
  • B: at the lowest error on 15 or 20 % of the training rows held out inside the boosting, keeping that smaller model;
  • C: the round found by B, applied to the model fitted on all the training rows.

Measured

Six demos, ten splits each, a quarter of the rows kept away from everything. Stop A against 120 rounds, on those rows:

Demo Accuracy or RMSE Log loss
titanic +0.4 point (better 5, worse 4) better on 10 of 10
fraud no change slightly worse
iris no change slightly worse
reviews 4.0 points lower (worse 6, better 2) better on 10 of 10
mpg no change no change
energy RMSE 3.5 % lower (better 9, worse 1) same measure
  • Fewer rounds give better probabilities, not better predictions. The curve V66 showed is a log loss in classification, which is why it turned early while the predicted classes held.
  • The one gain is on energy, where the stop is often round 0 (the mean): the boosting does no better than the baseline there and is the family chosen on validation on none of the ten splits.
  • B and C did no better than A on five demos of six.
  • The trap was real: with A, the boosting's validation score rose above its private score by 0.9 accuracy point on titanic and 6 on reviews, beyond what it already did at round 120.

What ships

  • The measure, written in the header of train/gbdt.ts, where the next reader will look for early stopping.
  • The glass box's sentence: it said the later rounds « did not help » and let the reader conclude that fewer rounds would predict better. It now says what was measured.
  • The how-to page « replay a prediction » and the limits page, in both languages, carry the table.
  • README (one sentence on the glass box row), PLAN row V67, F9 closed, changelog, version 1.67.0.

Not in this wave

A stop tied to the ranking metric when the user ranks on log loss (the measure does favour fewer rounds there on two demos of four, but the validation trap has to be solved first); tuning depth or learning rate; the network or the forest.

Checks

  • Lint, typecheck (apart from the xlsx types unreachable in my sandbox), format: clean.
  • Unit: 1289 passed locally, 5 failed, all in spreadsheet.test.ts (the xlsx tarball cannot be fetched in my sandbox; CI is the reference). No test added: no behaviour changed, one assertion follows the new sentence.
  • E2E: 132 passed locally after a fresh build, 1 failed (lab.spec.ts:31, the Excel upload, same cause).

Generated by Claude Code

V66 showed the boosting's validation error lowest well before its last
round on 17 runs of 30. Before building early stopping, it was measured:
six demos, ten splits each, a quarter of the rows kept away from
everything, the 120 fixed rounds against a stop at the lowest validation
error, a stop on rows held out inside the training rows, and that stop
followed by a refit.

On the metric the leaderboard ranks on, stopping early changes nothing
on four demos, loses 4 accuracy points on reviews, and helps only on
energy, where the boosting does no better than the mean. It improves the
log loss, not the predictions. A stop chosen on the validation rows also
flatters the boosting on the rows that choose the family.

Nothing changes in training. The measure is recorded in gbdt.ts and
PLAN.md, the glass box no longer suggests that fewer rounds would
predict better, and the docs carry the table in both languages. F9 is
closed. Version 1.67.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AyBsUet8WybTG986ubroar
@dapiced
dapiced marked this pull request as ready for review November 13, 2026 19:19
@dapiced
dapiced merged commit 2470561 into main Nov 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants