/crawlstats: stop paying thirty seconds for a planner's guess

[ralyodio]

The headline

This database has never been ANALYZEd. There is no sqlite_stat1 table, so SQLite falls back to its built-in guess that an equality test is more selective than a range test.

Two of jobBacklogs's five reads ask for a date and also filter on a status. The guess picks the status index for both — and status is not selective here at all:

read	before	after	why
submitted	17,722ms	654ms	seeked 330k pending rows to find 5,954
enriched	16,067ms	119ms	seeked 109k active rows; the partial index is keyed by the very column being counted

Identical results either way — 27x and 135x purely from index choice. Both now use indexed by.

The other half

/api/crawlstats bypassed every cache the page uses. It called q.jobBacklogs directly, while /crawlstats has always called the cached reader. That is why the endpoint took 53s to serve numbers the page rendered in 3.7s.

It now uses the same cached reader. The liveness numbers stay uncached — crawlStats and logActivity are still read fresh on every request — so the endpoint still cannot report a dead crawler as alive, which is the one thing a status endpoint must never do.

Measured

• jobBacklogs: 27,875ms → 3,158ms cold, 263ms warm
• /api/crawlstats: 53s → roughly the page's own cost
• /crawlstats page: was 8.1s cold / 3.7s warm

Tests

indexed by is not a hint SQLite may ignore — naming an index that doesn't exist fails at prepare time, so a renaming migration would take the jobs board down. The new tests assert the plans and that both indexes exist, plus that the numbers survive. A plan regression is otherwise invisible: same rows, same answers, thirty times the wall clock.

Full suite: 1,139 tests, 0 failures.

Deliberately not fixed

• categoryStats totals wants status, last_success_at and item_count for every non-dead row and cannot finish inside the 30s deadline. The covering index that would fix it spans three columns rewritten on every crawl, and writes are the binding constraint. It is served stale-while-revalidate, and the stale value is what the page shows — worth knowing that the background refresh is currently always failing.
• failingFeeds (2.4s) sorts every error row by error_count, which is also written on every successful crawl.

Both want a rollup table (like crawl_hourly) rather than an index.

Follow-up worth considering: running ANALYZE once would fix this class of problem globally rather than query by query. I did not run it — it is a long write against a 444k-row table on a live crawler, and it changes plans everywhere at once with no way to test first.