Compute the slow breakdown where nothing is waiting for it

[ralyodio]

Follow-on to #149. That took /api/crawlstats from 118s to 20.2s — and then it sat at exactly 20.2s on every request, because the remaining time was categoryStats hitting the timeout, failing, caching nothing, and being asked again by the next reader.

Why it can't be made cheap

Decomposed against production:

piece	time
feeds per category	6,340ms	index-only
growth by day+category	1,046ms	index-only
status × category	23,942ms
sum(item_count) per category	35,440ms
crawled-in-last-day per category	40,020ms

The slow three read columns no index covers, so they fetch every one of 476,715 rows. An index covering them would have to carry status and last_success_at, both rewritten on every crawl — buying a fast chart with a slower crawler, when writes are the one thing this system has none of to spare.

The 30s ceiling was ours

It's TURSO_REQUEST_TIMEOUT_MS, not a limit of the database. Given a longer deadline the whole statement completes in 58.9 seconds.

The fix

The query is unchanged and simply stops being on the request path. The poller recomputes it every five minutes on a connection with a patient deadline (connect({ timeoutMs }), new and per-connection) and primes the same Redis key the web service reads. A reader finds it already there.

Safe to run beside the crawler: it's a read, and reads don't queue behind the single writer everything else contends for. One long read every five minutes costs the crawler nothing.

primeCache goes through the same envelope remember writes, so the warmer and the reader can't drift on the format. Without REDIS_URL it's a no-op — the local and test case, where the poller simply doesn't warm.

The warm is logged as stats-warm-skipped rather than …-error on failure, deliberately: toEntry puts anything ending in "error" on the operational-alarm panel, and a missed warm isn't an alarm — the cache just keeps serving what it had.

Full workspace suite green (11 packages, 0 failures); web build clean; poller parses.

🤖 Generated with Claude Code