Provider usage-limit / quota errors are not classified as environment faults — the loop burns retries and defers healthy stories

[jackmcintyre]

bmad-loop version: 0.9.0 (repo e955c17)
Adapter: opencode-http (zai-coding-plan/glm-5.2 for dev; claude/opus for review)
OS: macOS (darwin 25.5.0) · opencode: 1.18.4
Impact: 3 stories deferred, ~12 hours of wall-clock burned, on specs that were fine.

---

Summary

When the dev provider returns a hard usage-limit / quota error, bmad-loop does not recognise it as an environment fault. It records env_fault: false, spends max_dev_attempts on the story, defers it, and moves to the next story — which hits the identical wall. In my run this converted a single 5-hour provider quota window into three deferred stories and seven dead sessions, each of which sat for its full session_timeout_min consuming zero tokens.

The story specs were not at fault. The first story of the same run completed and merged cleanly on the same adapter and model; it was simply the one that exhausted the quota.

What happened

Run of 5 stories. Story 1 succeeded, burned 2,517,158 raw tokens in its dev session, and exhausted the provider's 5-hour rolling budget. Everything after it failed:

session	status	tokens	wall-clock
story-1 dev-1	completed	2,517,158	— (merged fine)
story-2 dev-1	timeout	649,118	90.1 min
story-2 dev-2	timeout	517,249	90.0 min
story-3 dev-1	timeout	0	103.2 min
story-3 dev-2	timeout	0	110.0 min
story-4 dev-1	timeout	0	124.4 min
story-4 dev-2	timeout	0	105.2 min
story-5 dev-1	timeout	26,778	100.2 min

Every one of the 22 provider errors in the run logs is the same line:

AI_APICallError: Usage limit reached for 5 hour. Your limit will reset at 2026-07-26 22:49:46

Later sessions degraded to a second form — the provider dropping the connection rather than returning a clean 429:

AI_APICallError: Cannot connect to API: The socket connection was closed unexpectedly.

A representative dead session, start to finish (~6 KB of log for a 90-minute session):

15:45:39.732  stream providerID=zai-coding-plan modelID=glm-5.2
15:45:39.736  llm runtime selected  llm.runtime=ai-sdk
16:03:07      cleanup prune=7.days          <- 17 min in, no output
17:07:18      cleanup prune=7.days          <- 82 min in, no output
17:23:10      cancel session.id=...         <- bmad-loop's timeout firing
17:23:10      ERROR process ... error=Aborted

The journal records the decision path (abridged):

{"kind":"session-end","task_id":"...-dev-1","status":"timeout","tokens":0}
{"kind":"dev-decision","attempt":1,"session_status":"timeout","action":"retry","reason":"dev session timeout","env_fault":false}
{"kind":"session-end","task_id":"...-dev-2","status":"timeout","tokens":0}
{"kind":"dev-decision","attempt":2,"session_status":"timeout","action":"defer","reason":"dev session timeout","env_fault":false}
{"kind":"story-deferred","reason":"dev session timeout"}

env_fault: false is the bug. A provider quota is the definition of an environment fault: nothing about the story, the spec, or the worktree would change the outcome, and retrying is guaranteed to fail until the quota window rolls.

Root cause

Two independent gaps.

1. OpencodeHttpAdapter inherits the base env-fault hook, which is a no-op

To be precise: the plumbing is not missing. run() in src/bmad_loop/adapters/base.py:123 has always called _classify_env_fault for every adapter, so the hook fired on all seven of these sessions. It is the base implementation (base.py:145) that does nothing:

# src/bmad_loop/adapters/base.py:145
def _classify_env_fault(self, handle, spec, result) -> SessionResult:
    ...
    return result   # identity — same shape as _post_kill_reconcile

The only override shipped in #194 lives on GenericAdapter:

• src/bmad_loop/adapters/generic.py:746 — _classify_env_fault()
• src/bmad_loop/adapters/generic.py:773 — _env_fault_evidence()
• src/bmad_loop/adapters/generic.py:73 — ENV_FAULT_STATUSES = frozenset({"timeout", "stalled", "crashed"})

And the two adapters are siblings, not parent and child:

# src/bmad_loop/adapters/generic.py:249
class GenericAdapter(_ResultFileMixin, CodingCLIAdapter):

# src/bmad_loop/adapters/opencode_http.py:273
class OpencodeHttpAdapter(_ResultFileMixin, CodingCLIAdapter):

opencode_http.py contains zero occurrences of env_fault and does not import generic, so it never overrides the hook — it runs the identity version and returns the result untouched. An un-overridden hook, not absent wiring. ENV_FAULT_STATUSES already includes "timeout", so these sessions would have been eligible had an override been in place.

2. No profile has a quota / rate-limit pattern — including claude.toml

env_fault_patterns is a per-profile regex list (src/bmad_loop/adapters/profile.py:186). Of the six shipped profiles in src/bmad_loop/data/profiles/, exactly one defines any patterns — claude.toml:

env_fault_patterns = [
  "API Error.*(Unable to connect|Connection ?(error|refused|reset|timed ?out)|ConnectionRefused|ECONNREFUSED|ETIMEDOUT|ENOTFOUND|EAI_AGAIN)",
]

antigravity.toml, codex.toml, copilot.toml, gemini.toml and opencode.toml define none.

This is the more important half. That pattern is connection-class only — it would not match Usage limit reached, a 429, or a quota message on any adapter. So even a user on the claude adapter, where the machinery does run, is unprotected against this failure mode. Anthropic subscription plans have their own usage limits, so switching adapters does not avoid the bug; it just changes which provider trips it.

3. (Minor, possibly separate) sessions overran session_timeout_min

session_timeout_min = 90, but four sessions ran 103–124 minutes. Likely related to a 12-minute pre-flight hang seen at server boot in the same logs:

15:33:10  opencode server listening
15:45:33  ERROR "Failed to fetch models.dev" cause=Cause([Fail(TimeoutError)])

If the timeout clock arms only after the session is established, a slow/hanging bootstrap is unbounded. Worth confirming; it is not the main issue but it widened the damage.

Expected behaviour

A provider quota / rate-limit error should be classified env_fault = true, which routes to the existing pause path (src/bmad_loop/escalation.py:100, env_fault_pause_reason() at :76) rather than consuming max_dev_attempts. The run should pause and stay resumable, not defer work that has nothing wrong with it.

Ideally the loop would also not proceed to the next story after an env-fault defer — story N+1 will hit the same wall, and in this run it did, three times over.

Suggested fix

1. Hoist env-fault classification into shared code. Move _classify_env_fault / _env_fault_evidence / ENV_FAULT_STATUSES out of GenericAdapter into a mixin (or into _ResultFileMixin, which both adapters already use) and call it from OpencodeHttpAdapter.run() at the same point in teardown. This is the structural fix — any future sibling adapter otherwise repeats the omission.

2. Add quota/rate-limit patterns to every profile — but anchored, not as bare keywords. This is the part that is easy to get wrong. The scanned log is logs/<task_id>.log, which on the tmux adapters is a pane capture that includes the model's own output. A bare rate limit, quota or 429 therefore also matches a perfectly healthy session that merely works on a rate-limiting story — fixture data, banner copy, test names, assertions and docs all print that vocabulary. The loop would pause the run on a story with nothing wrong with it, which is the same bug inverted and no cheaper. Even in this run, the only 429 in story-2 dev-1 is the millisecond field of a timestamp (10:59:24.429Z) in an INFO line about formatting a file.

   So each pattern must require an error-shaped anchor and a cause on the same line — the shape claude.toml already uses (API Error.*<cause>). Its cause list is what is too narrow, not its structure. Anchors that actually appear in real output: AI_APICallError:, a level=ERROR log field, ✗ Model call failed:, ERROR: exceeded retry limit, last status:. Causes to join to them:

   • Usage limit reached
   • 429 Too Many Requests / rate_limit_error / RateLimitError
   • insufficient_quota / quota exceeded / RESOURCE_EXHAUSTED
   • socket connection was closed unexpectedly (providers commonly drop the socket instead of returning 429)

   Extend claude.toml the same way — its current pattern is correctly anchored but connection-class only.

3. Consider a distinct verdict for "retry cannot help". A connection blip is worth one retry; a stated quota window with a reset timestamp is not. The provider even supplies the reset time in the message, so the loop could in principle pause until then. At minimum, an env-fault defer should stop the run rather than marching into the next story.

4. Surface a zero-token timeout loudly. A session that hits session_timeout_min having consumed 0 tokens is almost never a hard story — it is nearly always infrastructure. That is a cheap, adapter-agnostic heuristic and would have caught this on the first occurrence rather than the seventh.

   Nothing like this exists today; the loop has no token/activity signal in its verdict at all. It matters more than it looks, because it is the only suggestion here that covers the sessions patterns cannot reach: three of the seven dead sessions logged no provider error of any kind — not a quota line, not a socket drop, nothing but the timeout abort. No pattern list, however well anchored, would classify those. Patterns handle the sessions that got an error back; the token signal is what handles the ones that got nothing back.

Evidence / where to look

All paths below are on my machine, in a private project repo — happy to supply any of it on request. bmad-loop diagnose <run-id> produces the sanitized dump if you would prefer that form; say the word and I will attach it.

Run directory:

<project>/.bmad-loop/runs/20260726-194932-d7e4/
├── ATTENTION            # the three "story deferred: ... dev session timeout" lines
├── journal.jsonl        # session-start/end, dev-decision (env_fault:false), story-deferred
├── state.json
├── logs/
│   ├── <story-1>-dev-1.log     248.9K  — the healthy session, for contrast (128 streams, 0 errors)
│   ├── <story-2>-dev-1.log     225.6K  — timed out with *zero* provider errors; its one ERROR line is the timeout abort
│   ├── <story-2>-dev-2.log     296.5K  — 160 streams, 12 errors; the first quota errors in the run appear here
│   ├── <story-3>-dev-1.log      14.4K  — 13 streams, 13 errors
│   ├── <story-3>-dev-2.log       6.2K  — the fully-dead session quoted above
│   ├── <story-4>-dev-1.log       7.3K
│   └── <story-5>-dev-1.log       8.5K
└── failed/<story-2>/changes.patch   80.7K — preserved partial work

Useful greps:

grep -h "Usage limit reached" logs/*.log | wc -l      # 22
grep -h "Cannot connect to API" logs/*.log            # the socket-drop form
grep -c "llm runtime selected" logs/<session>.log     # stream count: 128 healthy vs 2 dead
python3 -c "import json; [print(json.loads(l)) for l in open('journal.jsonl') if 'dev-decision' in l]"

The stream-count-vs-error-count ratio is the clearest single signal of the degradation:

story-1 dev-1   128 streams,  0 errors   <- healthy
story-2 dev-1   110 streams,  1 error    <- the one error is the timeout abort, not the provider
story-2 dev-2   160 streams, 12 errors   <- provider errors start here
story-3 dev-1    13 streams, 13 errors   <- collapsing
story-4 dev-1     2 streams,  2 errors   <- dead

Workaround

Switch [adapter.dev] off the quota-limited provider. Note this only moves the exposure — per gap 2 above, the claude profile's patterns do not match quota errors either, so a subscription usage limit would reproduce the same failure.

[coderabbitai]

🔗 Related PRs

#77 - feat(stories): spec-to-loop engine, adapter, CLI + HITL checkpoints (Phase 2 + 3) [merged]
#95 - feat(adapters): post-kill reconcile rescues finished stalled/timeout sessions [merged]
#175 - fix(adapters,engine): timely + observable session-timeout teardown [merged]
#272 - fix(#194): pause on coding-CLI transport failures instead of burning story/sweep attempts [merged]
#284 - feat: auto-resolve deferred-work entries a story declares (closes #234) [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!