Compute the slow breakdown where nothing is waiting for it - #150
Closed
ralyodio wants to merge 2 commits into
Closed
Compute the slow breakdown where nothing is waiting for it#150ralyodio wants to merge 2 commits into
ralyodio wants to merge 2 commits into
Conversation
/api/crawlstats answered in 118 seconds. It fans out eleven reads and returns
when the slowest does; timed against production:
categoryStats 30,005ms (timed out)
jobBacklogs 11,255ms
failingFeeds(20) 5,172ms
crawlStats 4,975ms
the other seven under 600ms each
No rewrite fixes `categoryStats`. It is a group-by over 476,715 rows, and on
the same connection a bare `count(*)` of that table is 6.9s while `select
category, count(*) … group by category` does not finish inside the client's 30s
deadline. Dropping its conditional aggregates -- the fix that worked for
`crawlStats` in PR #96 -- changes nothing, because the cost is visiting every
row for a column no index covers.
The per-process cache that was already here could not save it either, for a
reason worth naming: `categoryStats` does not run slowly, it *fails*, and a
cache that only stores successes stores nothing. Every request paid the full
timeout, for ever. Redis plus serve-stale-on-failure inverts that -- one
success, any time, serves every later reader -- and it survives the deploys that
emptied the old cache. It also adds no writes to Turso, whose write path is the
binding constraint on everything else here.
The part that needed care is that a status page must never report a stalled
crawler as healthy. The rule that keeps it honest is to cache facts and derive
anything measured against now: `idleMinutes` is `now - lastSuccessAt` computed
inside the query, so caching the object freezes it, and a dead crawler would go
on reporting the same cheerful number. `liveStats` caches the timestamp and
redoes the subtraction, so the number climbs while the crawler is down.
`queueHistory` does the same with its hour labels, caching the sparse rows and
filling the window on the way out.
A cache that can hang is not a cache, so the lookups are bounded too, and every
failure path -- no REDIS_URL, a refused connection, a socket that accepts
commands and never answers -- falls through to the read it replaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/api/crawlstats came down from 118s to 20.2s, and then sat there exactly, on
every request. The remaining 20 seconds were `categoryStats` hitting the
timeout, failing, caching nothing, and being asked again by the next reader.
Decomposed against production, the reason it cannot be made cheap:
feeds per category 6,340ms index-only
growth by day+category 1,046ms index-only
status x category 23,942ms
sum(item_count) per category 35,440ms
crawled-in-last-day 40,020ms
The slow three read columns no index covers, so they fetch every one of 476,715
rows. An index covering them would have to carry `status` and `last_success_at`,
both rewritten on every crawl -- buying a fast chart with a slower crawler, when
writes are the one thing this system has none of to spare.
The other half is that the 30-second ceiling was ours. It is
`TURSO_REQUEST_TIMEOUT_MS`, not a limit of the database, and given a longer
deadline the whole statement completes in 58.9 seconds.
So the query is unchanged and simply stops being on the request path. The poller
recomputes it every five minutes on a connection with a patient deadline and
primes the same Redis key the web service reads, so a reader finds it already
there. It is a read, and reads do not queue behind the single writer, so this
costs the crawler nothing.
`primeCache` goes through the same envelope `remember` writes, so the warmer and
the reader cannot drift on the format, and it is a no-op without REDIS_URL --
which is the local and test case, where the poller simply does not warm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by a clean rebase onto main after #149 was squash-merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #149. That took
/api/crawlstatsfrom 118s to 20.2s — and then it sat at exactly 20.2s on every request, because the remaining time wascategoryStatshitting the timeout, failing, caching nothing, and being asked again by the next reader.Why it can't be made cheap
Decomposed against production:
sum(item_count)per categoryThe slow three read columns no index covers, so they fetch every one of 476,715 rows. An index covering them would have to carry
statusandlast_success_at, both rewritten on every crawl — buying a fast chart with a slower crawler, when writes are the one thing this system has none of to spare.The 30s ceiling was ours
It's
TURSO_REQUEST_TIMEOUT_MS, not a limit of the database. Given a longer deadline the whole statement completes in 58.9 seconds.The fix
The query is unchanged and simply stops being on the request path. The poller recomputes it every five minutes on a connection with a patient deadline (
connect({ timeoutMs }), new and per-connection) and primes the same Redis key the web service reads. A reader finds it already there.Safe to run beside the crawler: it's a read, and reads don't queue behind the single writer everything else contends for. One long read every five minutes costs the crawler nothing.
primeCachegoes through the same enveloperememberwrites, so the warmer and the reader can't drift on the format. WithoutREDIS_URLit's a no-op — the local and test case, where the poller simply doesn't warm.The warm is logged as
stats-warm-skippedrather than…-erroron failure, deliberately:toEntryputs anything ending in "error" on the operational-alarm panel, and a missed warm isn't an alarm — the cache just keeps serving what it had.Full workspace suite green (11 packages, 0 failures); web build clean; poller parses.
🤖 Generated with Claude Code