Skip to content

fix(usage): record verified Claude ACP per-prompt counters - #1048

Merged
xintaofei merged 6 commits into
spacering-net:mainfrom
Adam-Dalloul:fix/explicit-acp-turn-usage
Oct 27, 2026
Merged

xintaofei merged 6 commits into
spacering-net:mainfrom
Adam-Dalloul:fix/explicit-acp-turn-usage

Conversation

@Adam-Dalloul

@Adam-Dalloul Adam-Dalloul commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Custom agents using the official Claude ACP adapter currently lose the usage returned with completed prompts. Record those per-prompt counters with the turn so they remain available after reopening the conversation.

This is the recording-only split from #1029. Native history recovery remains a separate design discussion in #1047.

  • Capture the counter policy from the actual launch definition. Only the official @agentclientprotocol/claude-agent-acp executable is accepted here; display names and custom IDs do not select accounting semantics.
  • Keep Command Code's cumulative/cache-inclusive response counters and unknown adapters out of per-turn accounting. TRAE retains its native enrichment, and ZCode's existing turn metrics remain authoritative.
  • Preserve reported spend for a completed prompt that emitted no text or tools. A footer cannot attach that spend to an older prompt.
  • Preserve ACP message IDs while batching transcript chunks. Distinct messages with otherwise identical metadata remain distinct.
  • Bump the fact projection version and expose stale facts through sync status so unchanged conversations are rebuilt too.

The official Claude response reports main-loop usage, not a complete billing ledger. This does not recover historical native files, subagent/autonomous/compaction spend, or stopped/cancelled prompts, even when their later responses include usage. It does not infer token counts from context occupancy or combine native and ACP totals.

Validation: production server/MCP/CLI compile and all-target Clippy pass. Focused tests cover the real typed ACP prompt-response recording path, unsupported cumulative counters, ZCode overlap, paid empty turns, message IDs, transcript projection, and token usage sync. The relevant test groups passed (4 contract/projection, 2 protocol recording, 28 transcript, 42 ACP parser, 46 ZCode, and 52 token/service cases; groups overlap).

An additional isolated HTTP fixture exercised the ordinary custom-agent launch and real usage report: verified Claude counters persisted after reopening, an unrecognized producer was excluded, and both missing and old projection markers triggered an incremental request to rebuild unchanged conversations. The ACP responses were deterministic fixtures; no provider generation or real sign-in was performed.

…ition

The manual add form stores only the package spec: the launch command is
derived from it and the version label falls back to `custom`. Check that
shape through `build_meta`, so a change to either derivation cannot
silently stop recording usage for these agents.
Document what version 9 changes in the version list the earlier bumps
use, and why 8 is skipped. Also restore the alphabetical order of the
`acp` modules and rewrap the dependency note on the two unstable ACP
features.
claude-agent-acp relays a message typed in another client of the
session (Remote Control) as a `user_message_chunk`, which can arrive
while codeg's prompt is still open. It closes the reply so far, so when
nothing followed it the turn end found no open part, no steer, and an
earlier reply in the turn, and its usage went nowhere: the prompt's
whole recorded spend was dropped.

A turn end with no open part now puts its usage and model on the last
reply of its own prompt in every case, as it already did after a
trailing steer. ZCode's turn metrics, which ride on the same turn end,
gain the same fix, which the schema 9 note now covers.
A turn end records the model selector's current value. For
claude-agent-acp that is an alias such as `opus` or `default`, so every
turn whose prompt-response usage is now recorded showed up under the
alias: the by-model breakdown listed it verbatim and the API-price
estimate counted its tokens as unpriced.

Since 0.71.0 the adapter's response splits the prompt's own tally by
model id in `_meta.quota.model_usage`, leaving out work the agent
started on its own and models with nothing to report. When exactly one
model did all of that work, the turn is named by it. When several did,
the main loop's model is unknown and the selector value stands.

The loop test harness can now answer a prompt with a whole response,
not only its usage.
A custom agent on the official Claude adapter runs Claude Code itself. When
it shares Claude Code's config directory, each of its sessions is also a
native Claude Code session under the same id, and importing local sessions
added a second, Claude Code conversation for it. Now that codeg records the
adapter's per-prompt usage, that session was counted twice in Token Usage.

The import now leaves such a session out, neither inserting it nor bringing
back a deleted copy, when a live conversation of a custom agent whose current
registration selects the adapter's counters holds the session, or continues
it after the adapter could not load it again; the import picker lists it as
already imported. Sessions of other adapters, whose usage codeg does not
record, import as before. Conversations imported before this change are left
as they are. After a plan approval that clears the context, the adapter
carries on in a new Claude Code session that no conversation holds, so that
session still imports.
@xintaofei

Copy link
Copy Markdown
Collaborator

@Adam-Dalloul thanks for splitting this out of #1029. It's the recording-only piece we hoped for, and several parts of it are careful work:

  • The counter rule comes from the launch definition. It checks the npx package and command, so a custom id or display name can't select it, and a running process keeps the rule it was launched with.
  • Counters with other meanings stay out. The prompt-response usage of Command Code, TRAE and unknown adapters isn't recorded.
  • A paid prompt with no text or tools keeps its usage, and a footer still can't attach to an older prompt.

I went through it and pushed five commits on top of your 009250a to this branch (a fast-forward, no force-push). If you push more changes, pulling first (git pull --ff-only) keeps them on the branch. In short:

  1. a4efe84b4 fix: a turn's usage was dropped when a message relayed from another client closed the reply.
  2. b2841d3c8 fix: recorded turns were usually named by a model selector alias (opus, default), which can't be priced.
  3. d6cb45b7f fix: a session could be counted twice when the custom agent shares Claude Code's config directory and local sessions are imported.
  4. c0ba35d4d test: a manually added definition is recognized as the official adapter.
  5. 2d934da15 docs: the schema 9 note and module order.

Details below, then three smaller notes on the rest of the PR. None of the notes block it.

1. Usage dropped when a relayed message closes the reply

claude-agent-acp forwards a message someone types in another client of the session (Remote Control, from claude.ai or the mobile app) as a user_message_chunk (acp-agent.ts#L8019-L8043, v0.97.0, the version codeg pins). It can arrive while codeg's prompt is still open, and in the acp_native projection it closes the reply so far. If no agent output followed it before the turn end, and the turn hadn't been steered, the turn end found no open part and nothing to fall back on, so its usage and model went nowhere: the prompt's whole recorded spend was dropped.

Now, when no part is open at the turn end and the prompt has a reply, the turn end's usage and model go on the prompt's last reply whether or not the turn was steered; before, only a steered turn did that (acp_native.rs#L535-L556). A prompt with no reply at all still takes your empty-turn path. ZCode's turn metrics ride on the same turn end, so the fix applies to them as well. Your schema 9 bump covers the change (I extended its note), so there's no new bump.

2. The recorded model is usually the selector's alias

Before this commit, a turn end recorded the model selector's current value. For claude-agent-acp that's usually one of the picker's aliases, such as opus or default, so the by-model breakdown listed the alias, and the API-price estimate found no price for it and counted those tokens as unpriced.

The prompt response's _meta.quota.model_usage splits the turn's spend by model id (acp-agent.ts#L12458-L12485). Those rows usually also count the prompt's sub-agents and internal calls such as compaction, which usage leaves out; the first result after a session is resumed lists only its main-loop usage (acp-agent.ts#L12631-L12659). Either way, they leave out work the agent starts on its own (acp-agent.ts#L7097-L7120). So when exactly one model there has non-zero tokens, the recorded usage was spent on that model, and the turn is now named by it (reported_usage.rs#L64-L97, connection.rs#L14372-L14386). When several have, the main loop's model can't be told apart, and the selector value stays.

3. Double count with a shared Claude config directory

A custom agent on the official adapter runs Claude Code itself, and for an ordinary new session the adapter uses the ACP session id as Claude's own session id (acp-agent.ts#L11467-L11483, #L11907-L11910). So when the custom agent resolves to the same Claude config directory as the built-in Claude Code (say, neither sets its own CLAUDE_CONFIG_DIR), its sessions are also native Claude Code sessions under the same ids, apart from the clear-context case below. Importing local sessions (the folder import or the import picker) then added a second, Claude Code conversation for such a session, because the import only de-duplicates by session id and agent (import_service.rs#L531-L535). That duplicate isn't new, but with this PR it double counts: Token Usage counts the session once from the native file under Claude Code, and again from the recorded turns under the custom agent.

The import now leaves such a session out (import_service.rs#L537-L546): it adds no Claude Code row and doesn't restore a deleted one. The picker lists the session as already imported (conversations.rs#L828-L834). A session is left out when a live conversation of a custom agent holds it and that agent's current registration selects the adapter's counters by your for_distribution rule. A session that conversation continues is left out too: when codeg can't load a session again, it carries the conversation on in a new one, and the earlier session stays on disk under its own id (import_service.rs#L335-L373). Sessions held by custom agents on other adapters import as before, since codeg doesn't record those adapters' prompt-response usage.

What it doesn't cover:

  • Duplicates imported before this change are left as they are, and a live one is still refreshed. Deleting the Claude Code copy is enough: as long as the custom agent's conversation holds the session, the import won't bring it back.
  • A plan approved with "clear context". The adapter carries on in a fresh Claude session under a random id (clear-context-coordinator.ts#L132-L135, acp-agent.ts#L11907-L11910), which no conversation holds. That session still imports, so the work after the approval can still be counted twice.
  • Definitions that changed. Only the agent's current registration is checked. If its definition was removed, or now points at another adapter, the import no longer treats the conversation's session as held, so the session can be imported as Claude Code. The conversation itself and the usage it recorded stay as they were.
  • Spend only the native file has. The import skips by session id, so for a session that wasn't imported as Claude Code before this change, what only its native file records isn't counted anywhere: turns the custom agent ran before this PR (they recorded no usage), turns run on the same session outside codeg, and spend the adapter's prompt counters leave out, such as sub-agents.

4. The other two commits

  • c0ba35d: a definition from the manual add form can carry just the package spec: the command is then derived from the package, and an empty version label falls back to custom. The test runs that shape through build_meta and for_distribution (reported_usage.rs#L177-L224), so a change to either derivation can't silently stop recording for these agents.
  • 2d934da: schema 9 is now in the /// * N — version list, with why 8 is skipped (token_usage.rs#L147-L153); reported_usage is back in alphabetical order in acp/mod.rs; the Cargo.toml note on the two unstable ACP features is rewrapped.

Smaller notes (non-blocking)

  • Stopped prompts. The description lists "interrupted prompts that did not return usage" as out of scope. That reads as if a stopped prompt's usage were kept when the adapter returns it, but it isn't. claude-agent-acp answers a cancelled prompt with the usage it has tallied for that turn (acp-agent.ts#L12382-L12421). When you press Stop, codeg records no turn end (connection.rs#L14695-L14704), keeps awaiting the response in the background, and drops it (connection.rs#L14779-L14789). Recording it would be a change of its own, since the cancel path skips the turn end on purpose (the comment there says why), so I'd just reword the description.
  • The version label check. for_distribution also requires version_or_tag(version) (reported_usage.rs#L34-L51). That version is the definition's label. What gets installed comes from package, or from a version picked in settings (acp.rs#L170-L183), and the launch runs cmd. A saved definition's label is already limited to [A-Za-z0-9._-], with custom when it's empty (custom_registry.rs#L369-L376, #L441-L444, checked on save at custom_agent_service.rs#L97). So the check can't fail for a definition codeg loads, and the npm:other label case in the test (reported_usage.rs#L145-L149) describes a definition codeg won't save.
    The comment's "never … a different executable installed alongside it" also promises more than the rule can check. codeg looks the command up by name: PATH, then ~/.local/bin, then the npm global prefix (acp.rs#L402-L419, connection.rs#L2126-L2133). If a different ACP program named claude-agent-acp is found first, codeg launches it, and any usage its prompt responses report is recorded under the official adapter's rule. Identifying the declared definition is fine; I'd narrow the comment and either drop the label check or say what it's for.
  • The launch wiring has no committed test. build_agent picks the mode and spawn_agent_connection writes it into the session state (connection.rs#L2051-L2054, #L2695), where the default is Unsupported (session_state.rs#L864). The loop tests set prompt_usage_mode by hand (connection.rs#L42276-L42277), and the manager test that launches a custom agent uses a binary definition and doesn't look at the mode. Your description mentions an isolated HTTP fixture that ran the launch, but it isn't in the PR, so if that write were lost, recording would stop and no test in the repo would fail.

Tests

  • Server lib: 5,971 passed. Desktop: 6,283 lib tests plus the integration tests passed. Clippy with -D warnings is clean for desktop (all targets), server and codeg-mcp.
  • Each fix has tests, and I checked that they fail when the fix is undone or weakened.
  • As of today, the branch merges into main without conflicts.

If you see any of this differently, I'm happy to change it.

@xintaofei
xintaofei merged commit 08fbdda into spacering-net:main Oct 27, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants