Repository navigation
feat: cut the cost of scode-build-goal overseer runs - #527
Merged
Merged
Conversation
In scode-build-goal's overseer mode, a strong overseer session hands a goal to a cheap executor session, typically Haiku, which it starts through farhelm (the tool that launches and supervises agent sessions) and steers by reading the executor's transcript. The first real run, a ten-PR goal, cost about $42 at API list prices, and most of that paid for re-reading context rather than for work. The executor never compacted: its context grew to 919K of its 1M-token window, and because Haiku bills any call whose prompt exceeds 100K tokens at five times its base rate, re-reading cached context made up about 90% of its cost. The opus overseer had been started in the session that planned the goal, so each of its looks also carried about 300K tokens of unrelated history. A haiku executor is now launched with Claude Code's auto-compact window set to 125K tokens. On Claude Code 2.1.325, compaction happens about 31K short of the window (measured at 100K and 130K windows), so it lands at roughly 94K, leaving some margin under Haiku's price step. The alternative was a fresh executor session per PR, but that ties a session's lifetime to the size of a PR, drops all context between PRs, and needs a farhelm launch for each one, which cannot yet happen fully unattended. Other models have no such price step and keep the default window. The setting travels as a farhelm launch wrapper, and farhelm holds a wrapper for the user's approval unless the host allows YOLO launches without asking, so an unattended haiku run now needs that host permission even when the overseer is not running with permissions bypassed. The skill's questions and closing instructions also tell the user to start the overseer in a fresh session, or to compact the session first. Each PR's review saw only that PR, so in that run nothing ever checked a PR together with the later PR that consumes its API. The overseer's final report now asks whether to run one review over the whole stack. It does not run one by default, since whether a stack warrants it is the user's call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In scode-build-goal's overseer mode, a strong overseer session hands a goal to a cheap executor session, typically
Haiku, which it starts through farhelm (the tool that launches and supervises agent sessions) and steers by reading the
executor's transcript. The first real run, a ten-PR goal, cost about $42 at API list prices, and most of that paid for
re-reading context rather than for work. The executor never compacted: its context grew to 919K of its 1M-token window,
and because Haiku bills any call whose prompt exceeds 100K tokens at five times its base rate, re-reading cached context
made up about 90% of its cost. The opus overseer had been started in the session that planned the goal, so each of its
looks also carried about 300K tokens of unrelated history.
A haiku executor is now launched with Claude Code's auto-compact window set to 125K tokens. On Claude Code 2.1.325,
compaction happens about 31K short of the window (measured at 100K and 130K windows), so it lands at roughly 94K,
leaving some margin under Haiku's price step. The alternative was a fresh executor session per PR, but that ties a
session's lifetime to the size of a PR, drops all context between PRs, and needs a farhelm launch for each one, which
cannot yet happen fully unattended. Other models have no such price step and keep the default window. The setting
travels as a farhelm launch wrapper, and farhelm holds a wrapper for the user's approval unless the host allows YOLO
launches without asking, so an unattended haiku run now needs that host permission even when the overseer is not running with
permissions bypassed. The skill's questions and closing instructions also tell the user to start the overseer in a
fresh session, or to compact the session first.
Each PR's review saw only that PR, so in that run nothing ever checked a PR together with the later PR that consumes
its API. The overseer's final report now asks whether to run one review over the whole stack. It does not run one by
default, since whether a stack warrants it is the user's call.