Carry a contact's provenance, which the crawler was failing without

[ralyodio]

What was happening

The crawler was not slow. It was failing: 985 of 1,385 crawls in the hour this was found errored, every one of them with could not be crawled — Unsupported type of value. Each failure still costs two write slots (the failed transaction, then markCrawlFailure) and stores nothing, so the queue stopped draining and /crawlstats went with it — the API was taking 30–60s to answer.

Root cause

feedContacts built each contact as { url, network } and dropped the channel element it was read from. Both feed_links.source and author_links.source are not null, so the statement bound undefined.

The remote libSQL client will not serialize undefined at all — it throws TypeError: Unsupported type of value in valueToProto, before any SQL runs, with no column named and no row to point at. That surfaced as an uncrawlable feed, which reads like a publisher who is down.

Why no test caught it: the local SQLite driver every test in this repo uses binds undefined as null without complaining. The difference only exists on the wire.

Why so many feeds: this is not a platform quirk. Any feed publishing a <webMaster> or managingEditor address whose name fails the person test takes this path — WordPress and Substack alike. Verified on littlealmanack.com and politicsnc.com as well as four Substack newsletters.

The fixes

1. Contacts carry source. Worth keeping in its own right: a mailbox from itunes:owner is a stronger claim than one from webMaster, and the column exists to record exactly that.
2. All four link bind sites default it, so no caller can put an unbindable value into a not-null column again.
3. A newsletter whose host fills in the iTunes block is no longer a podcast. Substack emits <itunes:owner> on every publication it serves and nothing else that looks like a show — no itunes:type, no podcast: namespace, image enclosures rather than audio — and that one tag filed the whole platform under /podcasts. This is the correction the video branch already makes: the tag has to be corroborated by what the feed actually ships. A declared show still stands on its own, so a podcast with no episodes yet keeps its category. Categories are re-derived every crawl, so this self-heals.

Verification

• packages/db/test/link-binds.test.js asserts what the remote client accepts, not what a local write survives. Confirmed non-vacuous: it fails on the pre-fix code.
• Live feeds re-parsed after the fix: nemtodamulher.substack.com → kind=blog, contact source web-master; The Daily and Darknet Diaries still → kind=podcast.
• Full suite green: 1,104 tests, 0 failures.