docs(devlog): record why the root workflow budget expires a long session (#4546) - #4653
Conversation
…ion (#4546) A Codex session dispatching subagents was refused across three unrelated providers with a 429 that reads as a provider rate limit. The refusal was this proxy: workflowSendCeilingReached compares a per-root send count that only ever grows, keyed on x-codex-parent-thread-id, so for Codex the cap is a session expiry rather than a fan-out guard. A probe carrying the session id was refused while a probe carrying a fresh root id was served, and restarting the proxy served both. The unit records the diagnosis and plans two layers: windowed ceilings so a rate is bounded rather than a lifetime, and a refusal an operator can read, name and clear without restarting. Verification posture: local suite, typecheck, install and build NOT run by explicit instruction. Hosted CI at the exact final head is the only proof. Pushed with --no-verify.
…efusal (#4546) One body, one upstream, two answers separated only by the claimed root id. That single check rules out the provider, the account and the model, and it is what the next person should run before spending hours on a status page. Also records that a restart erases the evidence, which is why the obvious remedy hides the cause. Verification posture: local suite, typecheck, install and build NOT run by explicit instruction. Hosted CI at the exact final head is the only proof.
|
✅ Deterministic PR hygiene checks passed. |
github-actions
Bot
added
the
documentation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe PR adds three planning documents. They describe a per-root lifetime budget defect, a timestamped windowing plan, and a management API plan for legible, operator-controlled refusals. No implementation code changes are included. ChangesWorkflow budget controls
Estimated code review effort: 1 (Trivial) | ~5 minutes Change: Other · Severity of issue fixed: Medium ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
리뷰 · 우선순위 71 / 80설명 이 PR은 #4546 비용 가드 스택에서 소스 코드는 건드리지 않고, 운영 중에 실제로 겪은 「긴 Codex 세션이 갑자기 전 제공자 429로 죽는」 현상의 원인을 기록하고, 다음 두 층(wfb 창 단위 천장, wfc 읽기 쉬운 거절)의 로드맵만 진단은 현재 checkout과 맞습니다. 비싸게 만든 두 점도 코드와 일치합니다. 거절은 방향(평생 합계 → 창 안 개수, 거절에 천장 이름·프록시 출처, 관리면으로 한 루트 읽기/지우기)은 #4546이 「예산을 제안이 아니라 경계로」 지키려던 것과 같은 축입니다. 숫자만 키우는 실패 모드를 명시적으로 금지한 것도 맞습니다. 다만 이 PR 자체는 구현이 아니라 계획서이므로, 머지 가치는 「다음 구현 PR이 이 단위를 기준으로 갈라지게」 만드는 쪽에 있습니다. 경로 메인테이너의 판단이 필요한 지점
너의 추천 이 댓글은 grok-bot이 작성했습니다 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 095f45b2e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # any root the process has not seen: served | ||
| curl -s -o /dev/null -w '%{http_code}\n' -N -X POST http://127.0.0.1:10100/v1/responses \ | ||
| -H 'Content-Type: application/json' -H 'Accept: text/event-stream' \ | ||
| -H "x-codex-parent-thread-id: probe-$(date +%s)" -d "$BODY" |
There was a problem hiding this comment.
Publishing this _plan unit exposes an unfixed spend-guard bypass: the command tells a data-plane client how to evade an exhausted root by minting a new x-codex-parent-thread-id, while the planned windowing and management fixes have not shipped. Repository policy explicitly places bypass reasoning and reproduction steps for unfixed defects in scratch rather than tracked devlog; remove this unit from the commit and keep it under .tmp/ until the fix is public.
AGENTS.md reference: AGENTS.md:L124-L130
Useful? React with 👍 / 👎.
| Two answers from one proxy, one body and one upstream, separated only by which | ||
| root the request claims. That is what rules out the provider, the account and the | ||
| model in a single step, and it is the check to run first the next time a fan-out | ||
| starts failing for no visible reason. |
There was a problem hiding this comment.
In a Codex Pool installation with multiple stored accounts, these probes do not hold the upstream constant: codexPoolAffinityKey derives conversation affinity from these headers (src/codex/auth-context.ts:112-117), and account resolution consumes that affinity key (src/codex/auth-context.ts:935-943). Replacing the parent ID can therefore select a different account, so a 429/200 result does not rule out the account or upstream as claimed; pin an account/route or corroborate the synthetic terminal source before presenting this as a definitive diagnostic.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
1 participant
Summary
A Codex session spent hours dispatching subagents and every dispatch failed, across three unrelated providers, with a 429 that reads as a provider rate limit. It was not the model, the account or the upstream — the refusal came from this proxy, and this unit records why and what to do about it.
workflowSendCeilingReachedcompares a per-root send count againstmaxPhysicalSends, and that count only ever grows.state.childrenis a Set that is only ever added to, its own doc comment reading "Distinct children one root may ever create". The root id isx-codex-parent-thread-id, which for Codex is the session. So for a long-lived session the cap is not a fan-out guard, it is an expiry: a session that made 256 sends in its first hour is refused for the rest of the process even if it sends nothing for a day.The justification in the code says a per-request cap "cannot bound a fan-out that sends once per child seven hundred times". That is a burst concern, and a burst is bounded by a rate. A lifetime total cannot tell seven hundred sends in a minute from two hundred and fifty-six spread over four hours, and it refuses both.
Two things made this expensive rather than merely annoying. The refusal is a 429 from a proxy that also forwards provider 429s, with nothing on the record saying which one it was — the same defect #4639 fixed for the synthetic 503. And the only way out is restarting the proxy:
resetWorkflowBudgetsForTestis named for its audience andworkflowBudgetSnapshothas no caller outside the module, so the state that decided the refusal cannot be read or cleared from outside the process.The unit plans two layers:
010makes the send and distinct-child ceilings windowed so a rate is bounded rather than a lifetime, and020makes the refusal name the ceiling that fired, marks it as proxy-origin on the record, and gives an operator a bounded, recorded way to read and clear one root. It also records the two-probe reproduction, which separates a proxy refusal from a provider one in a single step.Docs only. No source behaviour changes in this pull request.
Verification
Not run, by explicit instruction: the local suite,
bun run typecheck,bun install, and any build. The only proof for this unit is hosted CI at the exact final head SHA; this push used--no-verify.Nothing in the build, typecheck or test path reads from
devlog/, so this change cannot affect them;privacy:scandoes read it and the unit contains no credentials, account identifiers or unfixed-defect reproduction beyond a rate ceiling that is already public in the source.Checklist
Summary by CodeRabbit