Skip to content

feat(bridge): retry transient upstream failures with visible reconnect rows - #3

Open
dpolishuk wants to merge 3 commits into
feat/off-peakfrom
feat/upstream-turn-retry
Open

dpolishuk wants to merge 3 commits into
feat/off-peakfrom
feat/upstream-turn-retry

Conversation

@dpolishuk

Copy link
Copy Markdown
Owner

Stacked on #2 (needs the off-peak session markers it guards on).

When the model provider fails a turn with a transient upstream condition — the Z.ai 5xx outage on 2026-10-25 failed every turn with "Provider returned a server error." — the bridge now retries the turn instead of surfacing the error immediately.

  • Retry: holds the bb turn open and re-runs it via the native v4 retryTurn control (the CLI's own Retry action: rewind to the last user input, no message duplication). Up to 2 automatic retries per turn (~15s, ~45s), then explicit failure with the original native error.
  • Visible: every hold emits a bb provider.error reconnect row (willRetry: true), and native apiRetry ladder state on the event envelope is surfaced deduplicated ("ZCode retry 3/11 in 5 s").
  • Conservative: never retries a turn that already ran tools (re-run would repeat side effects), never applies to idle-time ticket runs, interrupt cancels a pending hold, duplicate turn.failed cannot settle a held turn, older CLIs without v4 controls fail explicitly.

Verified facts about the native protocol (pinned in knowledge/log.md): turn.failed carries code/type = model_request_failed without retryable; retryTurn targets the conversation tip row with actions.canRetry; bb renders provider.error + willRetry: true as a reconnect row.

8 new regressions in tests/retry.test.ts; full suite 220 green; typecheck/format/marketplace/build pass.

…t rows

A turn that fails with model_request_failed/model_request_timeout (5xx
server error, provider overload, retryable network failure) no longer
fails the bb turn immediately. The bridge holds the turn open, emits a
provider.error reconnect row (willRetry), and re-runs the turn through
the native v4 retryTurn control - the same rewind-to-last-input action
as ZCode Desktop's Retry button, without duplicating the message. The
retried native turn streams into the same bb turn.

- Budget: two automatic retries per turn (~15s, ~45s waits; the native
  CLI has already exhausted its internal 11-attempt ladder), then the
  turn fails explicitly with the original error.
- Guards: turns that already ran tools are never retried (a re-run
  would repeat side effects), idle-time ticket runs keep their own
  requeue path, interrupting the thread cancels a pending hold, and a
  duplicate turn.failed cannot settle a held turn.
- Visibility: native apiRetry state on the event envelope (attempt,
  max, delay, HTTP status) is surfaced deduplicated as reconnect rows
  while the CLI climbs its internal ladder.

Docs: configuration.md section, CHANGELOG unreleased entry, knowledge
base invariants/log. Tests: tests/retry.test.ts (8 regressions).
Copilot AI balanced review requested due to automatic review settings October 26, 2026 10:37
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: dd3b0eed-78cd-4fb8-9c57-6ca459d817bc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T14:21:39.268706Z 1e420e4 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 385645e9f2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/bridge.ts Outdated
Comment thread src/bridge.ts

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change adds intricate asynchronous turn-lifecycle state machinery (holds, timers, interrupt races, native retryTurn) whose correctness depends on SDK grammar and timing behavior that cannot be fully verified here, so it warrants final human review.

1 open issue
What changed in this PR

This PR makes the bridge resilient to transient upstream provider failures. Previously any turn.failed with a transient condition (the 2026-10-25 Z.ai 5xx outage) surfaced immediately as a failed turn. Now the bridge holds the bb turn open and re-runs it via ZCode's native v4 retryTurn control, keeping each attempt visible through provider.error reconnect rows. It fits into the existing session/turn lifecycle in src/bridge.ts, reusing the History.command channel and the Timeline tool-activity signal.

Changes:

  • Adds a bounded, conservative automatic-retry path (2 retries at ~15 s/45 s): only for transient codes, only for turns that ran no tools, never for idle-time (off-peak) runs; interrupts and duplicate turn.failed events are handled.
  • Surfaces native apiRetry ladder state from the event envelope as deduplicated provider.error reconnect rows.
  • Tracks per-turn tool activity in Timeline (toolActivity/noteToolActivity) so tool-bearing turns are never retried.

Per the repository's knowledge-maintenance guideline, this PR updates the knowledge base alongside the behavior change: knowledge/log.md, knowledge/architecture/session-invariants.md, plus user docs (docs/configuration.md) and CHANGELOG.md.

File Description
src/​bridge.ts Core retry logic: transient classification, hold/retry loop via retryTurn, exhaustion failure, apiRetry surfacing, retry-budget reset.
src/​timeline.ts Adds ranTools tracking with toolActivity getter and noteToolActivity() to gate retries on tool side effects.
tests/​retry.test.ts 8 new regression tests covering retry success, exhaustion, tool/idle exclusions, interrupt, timeout, and apiRetry rows.
docs/​configuration.md Documents the upstream-failure retry behavior for users.
knowledge/​architecture/​session-invariants.md Records the retry invariant and adds a hard-bounds table row.
knowledge/​log.md Pins verified native-protocol facts behind the retry implementation.
CHANGELOG.md Unreleased entry describing the new retry behavior.

Review effort: Balanced

Comment thread tests/retry.test.ts
…retry guard

Codex P2 fixes on 385645e:

- Bind the retry-start watcher to its original hold (bridge.ts:910): when a
  retried turn starts and fails again between two watcher polls, the next
  hold is installed before the first watcher notices. The watcher loop and
  failRetriedTurn now compare session.retry against the captured hold
  object, so a superseded watcher stands down instead of expiring against
  the newer hold's longer delay and failing the turn with a false
  "did not start the retried turn".
- Handle already-streamed text before rewinding (bridge.ts:790): retryTurn
  rewinds the native conversation, but the bb timeline cannot retract the
  failed attempt's streamed text - a retry after partial output would leave
  the abandoned answer next to (or concatenated with) the replacement.
  Timeline now tracks streamActivity, and canHoldForRetry excludes turns
  with any streamed text or reasoning; they fail explicitly instead.

Regressions: fast-second-failure watcher race completes the turn; streamed
partial answer fails without retryTurn. 222 tests green.
@dpolishuk

Copy link
Copy Markdown
Owner Author

Review round 1 fixes pushed in d95fcc9:

  • Hold-identity watcher (Codex P2 @ bridge.ts:910): the retry-start watcher and failRetriedTurn now compare session.retry against the captured hold object. When a retried turn starts and fails again between two polls, the superseded watcher stands down instead of expiring against the newer hold's longer delay. Regression: fast-second-failure race now completes the turn with no false "did not start the retried turn".
  • Streamed-output guard (Codex P2 @ bridge.ts:790): Timeline tracks streamActivity (any text/reasoning delta) and canHoldForRetry excludes such turns — a retry after partial output would leave the abandoned answer in the bb transcript. These turns now fail explicitly with "already streamed output or ran tools". Regression added.

222 tests green, typecheck/format pass.

Copilot review: every thread/delta this suite produces - including the
new provider.error reconnect rows - is now checked against
threadDeltaNotificationParamsSchema in the harness write hook, with
explicit grammarFailures assertions on the retry-success,
retry-exhaustion, and apiRetry-surfacing regressions.
@dpolishuk

Copy link
Copy Markdown
Owner Author

Copilot's grammar-coverage point fixed in 1e420e4: the harness now validates every emitted thread/delta (including the new provider.error reconnect rows) against threadDeltaNotificationParamsSchema, with explicit assertions on the retry-success, retry-exhaustion and apiRetry-surfacing regressions. 222 tests green. @codex review

@dpolishuk

Copy link
Copy Markdown
Owner Author

@codex review — re-pinging the pass on 1e420e4, the previous run appears stuck.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 1e420e4e93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants