Find the people behind the feeds when they never wrote themselves down - #133
Merged
Conversation
/authors/pierre-felgines is a name and nothing else: no email, no site, no accounts. The pass that produced it was not broken. It fetched felginep.github.io, found no rel="me", no h-card, no JSON-LD and exactly one outbound link — to the Jekyll theme its author happened to use — correctly concluded there was nobody to credit, and stamped the feed as checked. Meanwhile the author's GitHub account was named in the hostname the feed is served from, and one request to it returns "Pierre Felgines". That is the gap this closes, and it is not a corner. Of 369,056 feeds only 3,275 have ever been enriched, and 77% of the identity we hold came free out of feed documents rather than from visiting anything. Reading what publishers marked up works beautifully on the part of the small web that marks itself up; this reads the two things nobody had to publish. **The host.** A blog on <user>.github.io, <user>.substack.com, medium.com/@user and seven more names its owner's account in the address. Deriving it is string arithmetic on URLs already in hand, so a feed on an unrecognised platform costs nothing extra at all. **The profile behind it.** GitHub returns a real name, an avatar, a homepage and sometimes an email its owner chose to publish; GitLab a name and public email; Codeberg the same through Gitea; and a fediverse account returns its profile fields with the instance's own rel="me" verification already performed — the same handshake enrichFeedAuthors spends up to three fetches proving, arriving done. Three rules keep this from inventing people, which is the expensive direction: - An organisation is not a person. GitHub and Gitea serve both from one endpoint, so jekyll.github.io resolves to an account whose name is a product. The account is still stored as a link — a link is not a claim about who somebody is — but no author row is created. - A derived account is evidence, not proof, and sits below the 0.6 publishing floor until a profile answers. A 404 leaves no trace rather than half a person. - `verified` keeps meaning what the schema says. It is set when the profile links *back* at the site being enriched, which is the IndieWeb handshake in the other direction, and never merely because an account exists. safeFetch grew a headers option rather than being bypassed: a profile URL is built from a hostname read out of somebody else's feed, so it is exactly as untrusted as a page URL and must keep the private-address guard and the timeout. GITHUB_TOKEN is wired through the poller and is not optional in practice — the anonymous GitHub API allows 60 requests an hour per IP, which one batch spends, against 5,000 with a token. Measured against the live APIs while building, not assumed: GitLab's unauthenticated user search answers a reduced object (name and public_email only, the contact fields needing a token), and Mastodon bios are rendered HTML, where stripping every tag to a space turns "<b>InfoSec</b>." into "InfoSec ." — the test caught that before it reached anybody's page.
Every source the enrichment had was markup carrying identity as a side effect:
a link with a rel attribute, a microformat class, a byline in a feed document.
humans.txt is the one convention written to answer the question directly, and
the pass was walking past it.
Parsed rather than scraped, because the format inverts everything else: the key
is the role and the value is the person ("Chef: Jane Doe"), and the contact
lines that follow belong to the name above them. A parser that collected links
globally would hand Jane's Mastodon to Bob, which is worse than finding nothing
— it is a wrong contact address published as a right one.
Two sections of the file are not this blog's authors. /* THANKS */ credits other
people's work, so reading it as authorship attributes a blog to whoever its
author admires; /* SITE */ describes the build. Both are skipped, and the role
filters still apply, so "Developer: the web team" names nobody.
Fetched only when the ordinary pages named nobody. Most sites do not publish
one, so asking every site costs a request per feed across the directory to help
a minority; asking after the pages come back empty spends it exactly where it
decides between an author and no author. It is also read as text, not HTML —
plenty of servers answer every path with their 404 page, and parsing that would
turn a stylesheet reference into somebody's website.
/now and /uses join the page list for the same reason humans.txt earns a
request: they are conventions of exactly the population this directory indexes,
written in the first person, and a blog that has one often has no /about.
One fix found while testing: a "Site:" line is the most useful in the file and
classifyLink returns null for it, because it rightly refuses to file arbitrary
links as somebody's website. Here the key has already said that is what it is,
so the fallback is scoped to the keys that said so — Standards: and Language:
still produce nothing.
…t be reset The free sources read what a publisher put where we could find it. Some people put nothing anywhere, and for them a search engine is the only remaining route. It is also the one that can spend money and the one that can invent a person, so almost all of this is what it refuses to do. **It cannot run over the directory.** The credits come from CrawlProof's 25,000 a month, already shared with CrawlProof's own outreach runner, against 369,056 feeds — one query each would be fifteen times the monthly allowance. So it takes a key, a non-zero budget and an explicit switch, and is off without all three. **The budget survives a restart, because it is written down.** A counter in the process would be reset by every deploy, which is not a budget but a rate limit with a hole in it. `author_searches` is the ledger and the authority, and it is counted from the provider's own cycle — this account resets on the 13th, not the 1st, so a calendar month would let the allowance be spent twice across a reset. The ledger is also the audit trail: which people we spent money looking for, and whether it found anything, is how the gate gets tuned rather than guessed at. **It cannot invent a person.** Every query is scoped to the author's own domain as well as to one network, because the name is the ambiguous part — "Jane Doe" site:linkedin.com/in returns every Jane Doe, and the blog she writes is what distinguishes her from them. Results are filtered through classifyLink, so a company page, a job posting and an article about somebody are all discarded where a profile is kept. Nothing found this way is ever marked `verified`: that column means the IndieWeb handshake, and a search engine's opinion that two strings co-occur is not it. Every link is stamped `web-search` so a consumer can exclude the class outright. The gate is mean on purpose — confident it is a person, more than one word of name, publishes here, and currently unreachable — and it is stated twice, in the SQL that selects and in `worthSearching` that re-checks, so a drift between them costs nothing rather than money. On LinkedIn, since it is the thing that was actually asked for: a profile URL found this way is stored, because it is a public address the search engine has already indexed. The profile behind it is not fetched — auth-walled, 999 to anything automated, and its terms forbid scraping. We can say where somebody's LinkedIn is; we cannot say what is on it. Two things the tests caught rather than the reader: `authors` has no `site` column (it is `site_url`), and `addAuthorLinks` already existed and batches its inserts into one round trip, so the second one I wrote was deleted rather than kept.
Two things in the same area, both found while looking for a queue that turned out not to be needed. **A failure was recorded as a miss.** Every fetch in the enrichment fails softly — a dead host, a timeout and a 503 all come back as "no page" rather than as an exception — so a site that was simply down looked identical to a site that names nobody, and both were stamped as checked. That cost the publisher their enrichment for the whole ninety-day recheck cycle on the strength of one bad afternoon, and on a pass that has so far reached 3,275 of 369,056 feeds it quietly loses everyone on a flaky host. The pass now tracks whether it got an answer out of the publisher at all. Reached and found nobody: stamped, left alone until the recheck. Never reached: stamped to come back in a few days, because nothing was learned and the next attempt may well work. Still stamped either way, which is what keeps a permanently broken feed off the head of the queue. Done by back-dating the stamp rather than adding an attempts column, and that is a trade rather than a shortcut: writes here serialize and the crawl is already write-bound, so the fix that costs one UPDATE beats the tidier one that costs a migration and a second column on every read. **The pass had no row on /crawlstats.** It is the only job on the board that was invisible, which is a large part of why it looked like it had never started. It now shows its backlog, how many publishers it looked at in the last hour, and how far through the directory it is — read off the partial index 0024 already built, and counted as the stamped set rather than its complement, because 3,275 index entries is a cheap question and 369,056 rows is not. The queue rebuild this started as is deliberately not here. dueForAuthors already scopes to active feeds, so the pass has never been walking the 280,360 pending ones, and at the rate now observed the 84,398 active feeds are done in under a fortnight. A claim/lease table would have been machinery for a problem the numbers say does not exist. An existing test caught the new row before it shipped: every job on the board must report when it last ran, and one that emits an event nobody records is a row that reads as permanently stalled.
`0033_author_searches.sql` took the next sequential number, and there is no next sequential number any more: #131 froze the scheme at 0032 and added a test that says so, because 0006, 0018 and 0032 were each claimed by two or more branches before anybody noticed. Git never flags it — the files have different names — so the guard is the only thing that would. Renamed to the moment the file was actually written, 2026-08-19 08:18:24 UTC, per `packages/db/migrations/README.md`. Nothing referenced it by name, and it has never been applied anywhere, so this is a rename rather than a second migration. The failure was also hiding the rest of the suite: `pnpm -r` stops at the first package that fails, so packages/db going red meant ingest, auth, notify, translate, discover, web and poller never ran in CI at all. On Node 22, which is what CI uses: 1,067 pass, 0 fail, 0 cancelled, and `pnpm build` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/authors/pierre-felginesis a name and nothing else — no email, no site, no accounts. The pass that produced it was not broken: it fetchedfelginep.github.io, found norel="me", no h-card, no JSON-LD and one outbound link (to the Jekyll theme its author used), correctly concluded there was nobody to credit, and moved on.His GitHub account was named in the hostname the feed is served from.
Where the gap actually was
Reading what publishers mark up works beautifully on the part of the small web that marks itself up. This adds the sources for everyone else.
What it reads now
The host.
<user>.github.io,<user>.substack.com,medium.com/@userand seven more name their owner's account in the address. Deriving it is string arithmetic on URLs already in hand — a feed on an unrecognised platform costs nothing extra.The profile behind it. GitHub returns a real name, avatar, homepage and sometimes a public email; GitLab a name and public email; Codeberg the same via Gitea; and a fediverse account returns its profile fields with the instance's own
rel="me"verification already performed — the handshake we spend up to three fetches proving, arriving done.humans.txt, the one convention written to answer this exact question, fetched only when the ordinary pages named nobody. Plus
/nowand/uses.Bought search, off unless switched on, for the people who left no trail at all.
Measured on the real web, not asserted
Ran the new pass against live production feeds (read-only against prod, writing to a throwaway local DB):
github.ioetc., the case this exists for): 10/10 named a person, 10/10 gained links — Martí Bosch with fediverse + github + bluesky + linkedin + twitter, Martin Rotter with github + email + linkedin, and so on. These previously returned nothing.The three rules that keep it from inventing people
jekyll.github.ioresolves to an account whose name is a product. The account is still stored as a link — a link is not a claim about who somebody is — but no author row is created.verifiedkeeps meaning the handshake. Set when a profile links back at the site being enriched, never merely because an account exists, and never on a search result.The search is fenced, because it spends money
25,000 credits/month shared with CrawlProof's outreach runner, against 369,056 feeds — one query each would be 15× the monthly allowance. So: a key, a non-zero budget and an explicit switch, all three required. The budget is a ledger in the database, not a counter, because the poller restarts on every deploy and a budget that resets with the process is a rate limit with a hole in it. It counts from the provider's own cycle (this account resets on the 13th, not the 1st). A 402 stops the batch rather than being asked again.
On LinkedIn, since it is what was actually asked for: a profile URL found by search is stored, because it is a public address already indexed. The profile is not fetched — auth-walled, 999 to anything automated, terms forbid scraping. We can say where someone's LinkedIn is; we cannot say what is on it.
Two bugs found along the way
/crawlstats— the only job on the board that was invisible, which is much of why it looked like it had never started.What is deliberately not here
The claim/lease queue this began as.
dueForAuthorsalready scopes to active feeds, so it has never been walking the 280,360 pending ones, and at the observed rate the 84,398 actives finish in under a fortnight. It would have been machinery for a problem the numbers say does not exist.Verification
pnpm testexit 0.next buildpasses.verified, and each person in a humans.txt keeps their own accounts.authorshas nositecolumn,addAuthorLinksalready existed (batched, so mine was deleted), and stripping every HTML tag to a space turned<b>InfoSec</b>.intoInfoSec .in bios.Needs setting before it does much
GITHUB_TOKENon the poller — the anonymous GitHub API allows 60 requests/hour per IP, which one batch spends; 5,000 with a token. The search pass additionally needsAUTHOR_SEARCH=1,VALUESERP_API_KEYandAUTHOR_SEARCH_BUDGET.🤖 Generated with Claude Code