Serve the status page from Redis, and keep the alarm honest - #149
Merged
Conversation
/api/crawlstats answered in 118 seconds. It fans out eleven reads and returns
when the slowest does; timed against production:
categoryStats 30,005ms (timed out)
jobBacklogs 11,255ms
failingFeeds(20) 5,172ms
crawlStats 4,975ms
the other seven under 600ms each
No rewrite fixes `categoryStats`. It is a group-by over 476,715 rows, and on
the same connection a bare `count(*)` of that table is 6.9s while `select
category, count(*) … group by category` does not finish inside the client's 30s
deadline. Dropping its conditional aggregates -- the fix that worked for
`crawlStats` in PR #96 -- changes nothing, because the cost is visiting every
row for a column no index covers.
The per-process cache that was already here could not save it either, for a
reason worth naming: `categoryStats` does not run slowly, it *fails*, and a
cache that only stores successes stores nothing. Every request paid the full
timeout, for ever. Redis plus serve-stale-on-failure inverts that -- one
success, any time, serves every later reader -- and it survives the deploys that
emptied the old cache. It also adds no writes to Turso, whose write path is the
binding constraint on everything else here.
The part that needed care is that a status page must never report a stalled
crawler as healthy. The rule that keeps it honest is to cache facts and derive
anything measured against now: `idleMinutes` is `now - lastSuccessAt` computed
inside the query, so caching the object freezes it, and a dead crawler would go
on reporting the same cheerful number. `liveStats` caches the timestamp and
redoes the subtraction, so the number climbs while the crawler is down.
`queueHistory` does the same with its hour labels, caching the sparse rows and
filling the window on the way out.
A cache that can hang is not a cache, so the lookups are bounded too, and every
failure path -- no REDIS_URL, a refused connection, a socket that accepts
commands and never answers -- falls through to the read it replaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/api/crawlstatsanswered in 118 seconds. It fans out eleven reads and returns when the slowest does. Timed against production:categoryStatsjobBacklogsfailingFeeds(20)crawlStatsPromise.allof all 11Why no rewrite fixes it
categoryStatsis agroup by categoryover 476,715 rows. On the same connection a barecount(*)offeedsis 6.9s, andselect category, count(*) … group by categorydoes not finish inside the client's 30s deadline. Removing its conditional aggregates — the fix that worked forcrawlStatsin PR #96 — changes nothing, because the cost is visiting every row for a column no index covers.Why the existing per-process cache didn't save it
It's already stale-while-revalidate, and it still didn't work, for a reason worth naming:
categoryStatsdoesn't run slowly, it fails — and a cache that only stores successes stores nothing. So every request paid the full timeout, for ever. It also died on every deploy and was per-instance.Redis plus serve-stale-on-failure inverts that: one success, any time, serves every later reader. It also adds no writes to Turso, whose write path is the binding constraint on everything else here — which is why this isn't a rollup table.
Keeping the alarm honest
A status page must never report a stalled crawler as healthy. The rule: cache facts, derive anything measured against now.
idleMinutesisnow - lastSuccessAtcomputed inside the query, so caching the object freezes it — a dead crawler would go on reporting the same cheerful number until the entry expired.liveStatscaches the timestamp and redoes the subtraction, so the number keeps climbing while the crawler is down.queueHistorydoes the same with its hour labels: it caches the sparse rows and fills the window on the way out, so a cached chart can't carry yesterday's axis.Liveness gets a 10s TTL and a 2-minute staleness ceiling; the breakdowns get minutes and a day.
Failure paths
A cache that can hang is not a cache, so the lookups are bounded too. Every failure mode falls through to the read it replaced: no
REDIS_URL(local and test), a refused connection, and — the one that would otherwise be worse than no cache — a socket that accepts commands and never answers. There's a test for each.12 new tests in
packages/db/test/cache.test.js. Full workspace suite green (11 packages, 0 failures); web build clean.🤖 Generated with Claude Code