Conversation
… about it The project's reach was folklore -- a bug count somebody remembered, a sense that papers cite it. This builds it as data instead: 2,130 records, each one carrying the source it came from and the reason it counts, collected by a pipeline that can be re-run and disagreed with. What the section says 2,038 bugs across 41 database systems, 541 of them found by people outside the lab. 22 projects whose own evidence shows them running SQLancer and 9 more that have proposed it. 209 external papers, 69 built on the tool or its techniques, 34 calling it state of the art. Nine recorded talks where somebody stood up and described it. The bug count is published as a floor, and the page says so. Closed-source systems report nothing, some open-source ones (Materialize) report elsewhere, and a bug report need not name the tool that found it -- six projects that demonstrably run SQLancer contribute no counted bug at all, and the page names all six rather than implying the gap away. How a claim gets in Every record cites a source and an attribution rule, and the rules are ranked: a project saying "found by SQLancer" outranks a technique name, which outranks the fact that a known reporter was running a campaign. The weakest rule alone admits nothing. Where prose is ambiguous an LLM classifies, under three constraints that shaped most of the design: it answers about text it was given and cannot fetch more; it never supplies a quotation, only cites the id of a sentence already extracted, which makes fabricated evidence structurally impossible rather than caught afterwards; and it may answer "uncertain", which routes the candidate to a person instead of publishing a guess. Answers are cached by content hash and committed, so the dataset rebuilds offline and a rerun costs nothing. Nothing is published on a name match alone. An acronym like TLP or CERT needs corroborating context; a reproducer that generates `t0(c0 ...)` is a fingerprint, but 556 of the records rest on it together with a known reporter, never on the shape by itself. Papers are read, not counted `_data/papers/` holds one file per paper: the sentences that mention SQLancer, the citation markers that resolve to it -- which is how a paper saying "existing works [9]-[11]" is found at all -- what its artifact contains, and the classification each sentence supports. This is what separates a paper that builds on SQLancer from one that runs it as a baseline, a distinction name-matching got backwards. The PDFs themselves stay out of the repository, permanently. Talks are read four ways Slides can be quoted, a frame from the recording can only be shown, captions are verbatim with respect to the caption track and nothing more, and a viewer's account is a note. The four are kept apart on the page, because an automatic caption is not a quotation and a picture of a slide is not text. Also here: a page per database system, per paper and per year of bugs; charts; a review queue for what no rule settled, down to nine items; JSON Schema over every data file; and 281 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N
…pports Two failures on the first run of workflows that had never run anywhere, and both were real rather than incidental to CI. pypdf was missing. It is not an optional extra: intake reads a supplied PDF's title and first page with it, fulltext extracts paper text with it, and the artifact scanner finds links with it. Every one of those wraps the import in a try/except and returns "nothing found" when it fails, which is right at runtime and dangerous in a dependency list -- the weekly job installs from requirements.txt, so it would have run green while quietly extracting no paper text at all. The test that builds a PDF fixture was the only caller that said so out loud. It is now listed, and that test skips rather than fails without it, because a checkout without pypdf is a supported state. The hand-written `pip install jsonschema` is what let the two drift, so CI now installs from requirements.txt like the other workflow already did. Ruby 3.2 cannot build this site. The github-pages gem pins liquid 4.0.3, which calls Object#tainted?, removed in Ruby 3.2 -- so the build died in a 2023 blog post that has nothing to do with any of this. Its jekyll 3.9.0 also predates the Psych 4 that ships with Ruby 3.1. Ruby 3.0 is the newest that satisfies both, and is effectively what GitHub Pages builds with; the runner moves to 22.04 because setup-ruby ships no 3.0 for 24.04. Nothing about the site changes -- the pin was simply newer than the gems it had to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… about it
The project's reach was folklore -- a bug count somebody remembered, a sense that papers cite it. This builds it as data instead: 2,130 records, each one carrying the source it came from and the reason it counts, collected by a pipeline that can be re-run and disagreed with.
What the section says
2,038 bugs across 41 database systems, 541 of them found by people outside
the lab. 22 projects whose own evidence shows them running SQLancer and 9
more that have proposed it. 209 external papers, 69 built on the tool or its
techniques, 34 calling it state of the art. Nine recorded talks where
somebody stood up and described it.
The bug count is published as a floor, and the page says so. Closed-source
systems report nothing, some open-source ones (Materialize) report elsewhere,
and a bug report need not name the tool that found it -- six projects that
demonstrably run SQLancer contribute no counted bug at all, and the page
names all six rather than implying the gap away.
How a claim gets in
Every record cites a source and an attribution rule, and the rules are
ranked: a project saying "found by SQLancer" outranks a technique name,
which outranks the fact that a known reporter was running a campaign. The
weakest rule alone admits nothing.
Where prose is ambiguous an LLM classifies, under three constraints that
shaped most of the design: it answers about text it was given and cannot
fetch more; it never supplies a quotation, only cites the id of a sentence
already extracted, which makes fabricated evidence structurally impossible
rather than caught afterwards; and it may answer "uncertain", which routes
the candidate to a person instead of publishing a guess. Answers are cached
by content hash and committed, so the dataset rebuilds offline and a rerun
costs nothing.
Nothing is published on a name match alone. An acronym like TLP or CERT
needs corroborating context; a reproducer that generates
t0(c0 ...)is afingerprint, but 556 of the records rest on it together with a known
reporter, never on the shape by itself.
Papers are read, not counted
_data/papers/holds one file per paper: the sentences that mentionSQLancer, the citation markers that resolve to it -- which is how a paper
saying "existing works [9]-[11]" is found at all -- what its artifact
contains, and the classification each sentence supports. This is what
separates a paper that builds on SQLancer from one that runs it as a
baseline, a distinction name-matching got backwards.
The PDFs themselves stay out of the repository, permanently.
Talks are read four ways
Slides can be quoted, a frame from the recording can only be shown, captions
are verbatim with respect to the caption track and nothing more, and a
viewer's account is a note. The four are kept apart on the page, because an
automatic caption is not a quotation and a picture of a slide is not text.
Also here: a page per database system, per paper and per year of bugs; charts; a review queue for what no rule settled, down to nine items; JSON Schema over every data file; and 281 tests.
Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N