A rate limit is a schedule instruction, not a broken feed - #142
Merged
Conversation
…140) Every crawl of a feed that publishes a contact address but names nobody has been failing at the write since author enrichment shipped. In the hour this was found, 985 of 1,385 crawls errored; the queue stopped draining and /crawlstats went with it. `feedContacts` built each contact as `{ url, network }` and dropped the channel element it came from. Both `feed_links.source` and `author_links.source` are `not null`, so the statement bound `undefined` -- which the remote libSQL client will not serialize at all. It throws `Unsupported type of value` before any SQL runs, with no column named and no row to point at, and the crawl recorded it as `could not be crawled`: a publisher who looks down. The population is large and it is not a platform quirk. Any feed with a `<webMaster>` or `managingEditor` address whose name fails the person test takes this path -- WordPress and Substack alike, and Substack additionally names nobody else, so every newsletter on it qualified. Nothing caught it because the local SQLite driver the tests use binds `undefined` as null without complaint. The difference only exists on the wire, so `link-binds.test.js` asserts what the remote client accepts rather than what a local write happens to survive. Two fixes, because either alone leaves a hole: contacts now carry `source` (provenance worth keeping in its own right -- a mailbox from `itunes:owner` is a stronger claim than one from `webMaster`), and the four link bind sites default it, so no caller can put an unbindable value in a not-null column again. Also: a newsletter whose host fills in the iTunes block is no longer a podcast. Substack emits `<itunes:owner>` on every publication it serves and nothing else that looks like a show -- no `itunes:type`, no `podcast:` namespace, image enclosures rather than audio -- and that one tag filed the whole platform under /podcasts. This is the correction the video branch already makes: the tag has to be corroborated by what the feed actually ships. A declared show still stands on its own, so a podcast that has not released an episode yet keeps its category. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Crawling faster got us 429s from Substack, and the crawler recorded each one as the publisher's fault: `markCrawlFailure` sets status='error', increments error_count, walks the backoff ladder, and at ten consecutive failures marks the feed dead. Every feed on one backend is throttled in the same minute, so this retires a whole platform for our own crawl rate -- and Substack is a large share of the directory. A 429 now reschedules and touches nothing else. `status`, `error_count`, `last_error` and `last_success_at` are left exactly as they were, and so is `last_fetched_at`: it means "when we last read this publisher", and a throttle is precisely the case where we did not. Stamping it would make a feed we have been bounced from all day look freshly crawled on every staleness report. 503 with a Retry-After is treated the same way. It is the same statement from a server that is briefly unwilling rather than permanently unable. The server picks the interval, since it is the only party that knows when its limit resets. `Retry-After` is parsed in both forms RFC 9110 allows -- a delay in seconds and an HTTP date -- floored at a minute so `Retry-After: 0` cannot spin, capped at a day so a misread date cannot mothball the feed, and defaulted to 30 minutes when the server names nothing. That is deliberately far shorter than the error ladder it replaces: the feed is healthy and we want it back soon. It is the *rate* that has to come down, and lengthening one feed's interval is the wrong instrument for that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Making the crawler faster today got us
http-429from Substack. The crawler recorded every one as the publisher's fault:markCrawlFailuresetsstatus='error', incrementserror_count, walks the backoff ladder — and at ten consecutive failures marks the feed dead. A rate limit is hit by every feed on one backend in the same minute, so this retires a whole platform for our own crawl rate. Substack is a large share of the directory.Observed in production: 61 of 72 crawl errors in one window were 429s, against feeds that are entirely healthy.
What changed
A 429 (and a 503 that carries
Retry-After— the same statement from a server briefly unwilling rather than permanently unable) now reschedules and touches nothing else.status,error_count,last_errorandlast_success_atare left exactly as they were.last_fetched_atis deliberately not stamped either. It means "when we last read this publisher", and a throttle is precisely the case where we did not — stamping it would make a feed we have been bounced from all day look freshly crawled on every staleness report.The server picks the interval, since it is the only party that knows when its limit resets.
Retry-Afteris parsed in both forms RFC 9110 allows (delay-seconds and HTTP-date), floored at a minute soRetry-After: 0cannot spin, capped at a day so a misread date cannot mothball the feed, and defaulted to 30 minutes when the server names nothing.That default is much shorter than the error ladder it replaces, on purpose: the feed is healthy and we want it back soon. It is the rate that has to come down, and lengthening one feed's interval is the wrong instrument for that.
Tests
throttle.test.jsasserts the negative — that the throttle path touches no health column — which is the actual guarantee, and that an ordinary 404 still counts against the feed so the guard cannot swallow real breakage.retry-after.test.jscovers both header forms, the past-date case, and the clamp.Full suite: 1,115 tests, 0 failures.
Not in this PR
The deeper issue is that politeness is capped per hostname, and every Substack newsletter is a distinct hostname on one backend — so we can burst a single platform without the cap noticing. Capping by registrable domain is the real fix and wants its own change.