Skip to content

Fit the STK-15 canary under the Gemini output ceiling again, and keep the usage of a truncated run #1707

Description

@braboj

Defect

The STK-15 canary stops on MAX_TOKENS again, with thinking disabled, so the release cut has no live canary. And the report of such a run carries no token figures: _gemini in tests/providers.py raises on a non-STOP finish before it returns usage, so the runner's Tokens line never renders on the error path, and the report cannot say how far over the ceiling the run went.

Measured

2026-09-12, during the v2.90.0 cut. Provider gemini, model gemini-2.5-flash, thinking_budget 0, MAX_TOKENS 65536.

Tree Result Elapsed Tokens in report
b1c9681 (templates identical to the cut commit 1569f17: git diff --stat -- templates/ is empty) ERR: stopped on FinishReason.MAX_TOKENS, not STOP 257.6s none

Report: tests/reports/2026-09-12T12-19-40-e2e-b1c9681.md (gitignored).

The python-lib chain, as generated/stack-python-lib.md:

Tree Bytes
v2.89.0 463,990
c8eb111, the merge of #1686 that closed #1676 469,660
1569f17, the v2.90.0 cut 482,706

The comment beside the thinking-disabled call records the same prompt completing at 11% of the ceiling when #1676 was fixed. This run is the first live canary since that fix, and the first at a chain past 470K bytes.

Not measured

How many output tokens the truncated run produced; the runner drops the figure. Whether the growth (base-oop in the core tier, #1696, is the largest single addition) or the model accounts for the difference.

Expected

  • The canary completes under the ceiling at the current chain, or the ceiling is raised where the model allows it, or the canary's output model changes; the v3.0 plan already names the fork's canary fitting under the ceiling as a Phase 1 exit.
  • A run stopped on MAX_TOKENS still reports its usage, so the next such report says by how much.

Found during the v2.90.0 cut; recorded in the cut's pull request and not fixed inside it, per the PLAYBOOK.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High — must fix before next milestonebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions