Skip to content

fix(core): tell Code Mode models built-in tools stay top-level - #57627

Open
rekram1-node wants to merge 2 commits into
devfrom
codemode-builtins-wording
Open

rekram1-node wants to merge 2 commits into
devfrom
codemode-builtins-wording

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Issue for this PR

Fixes #53623

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Problem. Weaker models (Gemma 4, MiMo v2.6 Flash) write await tools.read(...), tools.shell(...) or tools.glob(...) inside execute. The Code Mode prompt opens with "Use the execute tool to call the tools listed below" under ## Available tools, and nothing says the regular tools are still called directly.

Change. Only the Code Mode prompt wording in packages/core/src/codemode/instructions.ts:

 # Code Mode

-Use the `execute` tool to call the tools listed below. They cannot be called directly, and neither can `search`. Both only work inside code you pass to `execute`.
+Code Mode adds the extra tools listed below, which you call from JavaScript inside the `execute` tool. Your regular tools (such as `read`, `edit`, `shell` and `grep`) are separate: keep calling them directly as usual. They do not exist inside `execute`.
+
+The tools listed below cannot be called directly, and neither can `search`. Both only work inside code you pass to `execute`.

-The catalog is partial. Inside `execute`, use `search(...)` to find a tool, then call it by the `path` in the result. `search` is synchronous. ...
+The catalog is partial. Before calling a tool that is not listed below, find it with the global `search(...)` function (not `tools.search`). Do not guess tool names. `search` is synchronous. ... Each result's `path` is the complete call expression: write it out as code, for example `await tools.ns.tool_name({ ... })`. Do not use it as a key such as `tools[path]`.

-## Available tools
+## Code Mode tools (call only inside `execute`)

Evidence. From the codemode-routing eval (https://github.com/anomalyco/evals/pull/18, run on dev aadd4d8) with only this wording change, share of trials that call a built-in inside execute:

model catalog before after
Gemma 4 31B no MCP 46% 0%
Gemma 4 31B small catalog 20% 0%
Gemma 4 26B small catalog 17% 2%
MiMo v2.6 Flash no MCP 13% 0%

Mean tokens on the small catalog dropped from 136k to 26k (Gemma 4 31B) and 180k to 33k (Gemma 4 26B), mostly from fewer runaway loops.

Cache effect. The Code Mode block is part of the instruction baseline, which is re-rendered from the stored catalog on every request. Existing Code Mode sessions get the new text on their next request after deploy and miss the prompt cache once for that prefix.

Out of scope. Gemma 4 26B with an 84-tool partial catalog still guesses tool names instead of searching, and the runtime error for unknown paths is unchanged.

How did you verify your code works?

  • bun typecheck in packages/core: pass
  • bun test test/codemode in packages/core: 23 pass
  • The eval runs above

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks right overall. On the base, the new test fails with the old Unknown tool 'read'. error and passes with this PR. codemode tests (1416) and core codemode tests (23) pass, and both packages typecheck. I couldn't check that the models themselves behave differently: that needs a real provider, and the simulated model only replays scripted calls.

Public API: the published @opencode/codemode package changes behaviour. Unknown tool paths whose first segment is one of the hardcoded names now fail with an opencode-specific message instead of the fuzzy "Did you mean" suggestion. Nothing is added or removed from its API, and opencode's SDK, server, config and plugin API are unchanged.

One concern, inline: the hardcoded list of built-in tool names in packages/codemode would be better passed in from core, using the tools core keeps out of Code Mode (codemode: false).

Related: #51949, #55087

Comment thread packages/codemode/src/tool-runtime.ts Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Code Mode" instructions are confusing Gemma-4-31B

1 participant