Fail a boot index that cannot render within minutes, and report it as terminal - #6076
Open
backspace wants to merge 5 commits into
Open
Fail a boot index that cannot render within minutes, and report it as terminal#6076backspace wants to merge 5 commits into
backspace wants to merge 5 commits into
Conversation
A render that times out while its page is idle — main thread responsive, no script running, nothing fetching, no module import or document load in flight — was waiting on nothing, and nothing about the next file changes that. In a host CI shard whose prerender pages could not reach the icons server, every base module render timed out that way at about 72 s apiece; the 267-file pass would have needed five hours, the job would have been rejected at its 3600 s limit with nothing written, and until then nothing said what was wrong. The from-scratch visit loop now counts consecutive idle timeouts, recognised from the diagnostics the prerender server captured as the timer fired, and after three throws past the per-file error isolation with a message naming the files and the likely cause: an origin the page needs is unreachable from the browser. The job is rejected within minutes carrying that reason. Incremental passes are unchanged, since their rows are the only record of a user's write. INDEX_IDLE_RENDER_TIMEOUT_ABORT_AFTER overrides the threshold; 0 disables the guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A brand-new realm awaits its first from-scratch index at startup, and when that job fails the failure is logged and swallowed: startup completes, the index lane is clear, and `_readiness-check` answers 200 over an empty index. A caller waiting on the realm — a publish, the CI realm wait — is handed a realm that serves nothing, and every test that follows fails for a reason two steps removed from the cause. The realm now keeps the failure the awaited boot index returned and answers readiness with 503 `X-Boxel-Not-Ready: index-failed`, the failure in the body and no `Retry-After`, since no amount of polling cures it; a later full index that completes clears the state. The publish flow's poll and the CI wait task both stop on that stage and report the reason. A test drives a realm whose every render times out idle through startup and checks that the job gave up after three visits, was rejected naming the idle timeouts, and that readiness reports the terminal stage with the failure in its body. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2fbe6e9e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Consecutive idle render timeouts are not proof of a broken stack on their own. The visit order groups a definition's instances together, so one card type that hangs on an untracked promise or timer times out idle instance after instance, and the pass would have abandoned the whole realm over one bad card. Nor were the idle signals themselves airtight: absent diagnostics counted as zero, so a timeout whose CDP sampling or in-page hook had failed read as idle. When the streak reaches the threshold the loop now renders a canary — the base card-api module, which every realm depends on and none owns — on the pass's own affinity and loader epoch. If it renders, the timeouts are the cards' own: the streak resets and the pass continues, recording them as the files' errors. If it cannot render either, the pass gives up as before, naming the canary's failure alongside the files. Idleness now requires each outside-the-page signal — the responsiveness probe, the CPU sample, the CDP request list — to be present and idle; the in-page counters disqualify when present and non-empty and say nothing when absent. The readiness test exercises both outcomes: a failing canary ends the pass after three or four of six visits and rejects the job naming both causes, and a rendering canary lets the pass visit every file, complete, and leave the realm ready. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The terminal readiness stage was held in a field set when the awaited boot index failed and cleared only in Realm.fullIndex. The realm's own _reindex and _full-reindex endpoints go through startReindex, and the reindex handler enqueues the job directly, so a successful reindex from any of those paths left readiness reporting index-failed until the process restarted or the realm was republished. The field was also per-replica, so a peer that had not run the boot could not report it. Readiness now derives the stage from the rows every replica reads: once the index lane is clear, a realm that still has no index whose newest from-scratch job was rejected is reported as index-failed, with that job's failure in the body. A reindex from any path that completes gives the realm an index and clears the state the moment it lands. The in-memory flag and the return value that fed it are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Host Test Results 1 files ±0 1 suites ±0 2h 40m 58s ⏱️ +49s Results for commit 770f945. ± Comparison against earlier commit 5953870. Realm Server Test Results 1 files 208 suites 1h 15m 25s ⏱️ Results for commit 770f945. |
Readiness took the absence of a realm_generations row as "the realm has no index". A from-scratch pass inserts that row at generation 0 before it visits anything and advances the generation only when it completes, so after a pass that was rejected the row exists at 0 and the check read a realm that serves nothing as one with an index; the terminal stage never fired. The generation is the signal: 0, or no row, means no pass has ever promoted rows. The two shared-state reads now live in one helper. The fixture also has seven files, not six — realm.json is visited too — so the canary test expected one visit too few. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lukemelia
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Two changes so that a boot index which cannot render anything fails within minutes, naming the cause, instead of grinding for hours and then reporting a realm as ready over an empty index.
The from-scratch pass gives up after consecutive idle render timeouts
A render that times out while its page is idle — main thread responsive, no script running, nothing fetching, no module import or document load in flight — was waiting on nothing, and nothing about the next file changes that. Several in a row describe the stack rather than the files: an origin the page needs (the host bundle, the realm-server, the icons server) is unreachable from the browser, and every remaining file would cost the full render timeout to fail the same way.
IndexRunner's visit loop recognises such a timeout from the diagnostics the prerender server captured as the timer fired. The signals that decide idleness are the ones captured from outside the page — the responsiveness probe, the CPU sample, the CDP request list — and each must be present and idle; the in-page counters disqualify when present and non-empty, and say nothing when absent. A timeout whose diagnostics are missing or show work in progress is a slow or stuck render and does not count.Consecutive idle timeouts are not proof of a broken stack on their own: the visit order groups a definition's instances together, so one card type that hangs on an untracked promise produces the same run. After three in a from-scratch pass the loop therefore renders a canary —
https://cardstack.com/base/card-api, a module every realm depends on and none owns, on the pass's own affinity and loader epoch. If the canary renders, the timeouts are the cards' own: the streak resets and the pass continues, recording them as the files' errors. If the canary cannot render either, the loop throws past the per-file error isolation with a message naming the files, the canary's failure, and the likely cause, and the job is rejected carrying it. Incremental passes are unchanged: their rows are the only record of a user's write, so they keep recording per-file errors rather than discarding the pass.INDEX_IDLE_RENDER_TIMEOUT_ABORT_AFTERoverrides the threshold;0disables the guard.A failed boot index is a terminal readiness stage
A brand-new realm awaits its first from-scratch index at startup, and when that job fails the failure is logged and swallowed: startup completes, the index lane is clear, and
_readiness-checkanswers 200 over an empty index. Once the lane is clear, readiness now also asks whether the realm has never had an index built — itsrealm_generationsrow is still at generation 0, or absent, since a pass inserts the row at 0 before visiting anything and advances it only on completion — and, if so, whether the newest from-scratch job for it was rejected. When both hold it answers 503 withX-Boxel-Not-Ready: index-failed, the job's failure in the body, and noRetry-After, since polling cannot cure it. Both facts come from the rows every replica reads, so every replica answers alike, and a reindex from any path — the realm's own_reindex/_full-reindex, a publish, the system-wide reindex — clears the state the moment it lands.Consumers stop on the terminal stage instead of polling to their own deadline:
waitForReady(the publish flow) throws with the failure from the body.ci:wait-for-realmsfails the step with the failure, alongside the response headers and Traefik diagnostics it already prints.docs/realm-server-health-signals.mddescribes the stage.Test plan
realm-endpoints/readiness-check-test.tsgains a module that builds a realm with an in-process worker and a prerenderer stub whose every index visit returns a render-timeout error with idle diagnostics, then starts it. With the canary module also failing, it checks that the pass stopped after three or four visits of six files with one canary render, that the job was rejected naming the idle timeouts and the canary, and that readiness answers 503index-failedwith noRetry-Afterand the failure in the body. With the canary rendering, it checks that all six files were visited with a canary per streak, the job completed, and readiness answers 200.runtime-commonandrealm-serveradds no errors against this worktree's baseline; eslint and prettier are clean on the changed files; shellcheck is clean on the wait task.🤖 Generated with Claude Code