Skip to content

Add an /impact/ section: what SQLancer found, who uses it, who writes… - #10

Merged
mrigger merged 2 commits into
mainfrom
impact
Sep 13, 2026
Merged

Add an /impact/ section: what SQLancer found, who uses it, who writes…#10
mrigger merged 2 commits into
mainfrom
impact

Conversation

@mrigger

@mrigger mrigger commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

… about it

The project's reach was folklore -- a bug count somebody remembered, a sense that papers cite it. This builds it as data instead: 2,130 records, each one carrying the source it came from and the reason it counts, collected by a pipeline that can be re-run and disagreed with.

What the section says

2,038 bugs across 41 database systems, 541 of them found by people outside
the lab. 22 projects whose own evidence shows them running SQLancer and 9
more that have proposed it. 209 external papers, 69 built on the tool or its
techniques, 34 calling it state of the art. Nine recorded talks where
somebody stood up and described it.

The bug count is published as a floor, and the page says so. Closed-source
systems report nothing, some open-source ones (Materialize) report elsewhere,
and a bug report need not name the tool that found it -- six projects that
demonstrably run SQLancer contribute no counted bug at all, and the page
names all six rather than implying the gap away.

How a claim gets in

Every record cites a source and an attribution rule, and the rules are
ranked: a project saying "found by SQLancer" outranks a technique name,
which outranks the fact that a known reporter was running a campaign. The
weakest rule alone admits nothing.

Where prose is ambiguous an LLM classifies, under three constraints that
shaped most of the design: it answers about text it was given and cannot
fetch more; it never supplies a quotation, only cites the id of a sentence
already extracted, which makes fabricated evidence structurally impossible
rather than caught afterwards; and it may answer "uncertain", which routes
the candidate to a person instead of publishing a guess. Answers are cached
by content hash and committed, so the dataset rebuilds offline and a rerun
costs nothing.

Nothing is published on a name match alone. An acronym like TLP or CERT
needs corroborating context; a reproducer that generates t0(c0 ...) is a
fingerprint, but 556 of the records rest on it together with a known
reporter, never on the shape by itself.

Papers are read, not counted

_data/papers/ holds one file per paper: the sentences that mention
SQLancer, the citation markers that resolve to it -- which is how a paper
saying "existing works [9]-[11]" is found at all -- what its artifact
contains, and the classification each sentence supports. This is what
separates a paper that builds on SQLancer from one that runs it as a
baseline, a distinction name-matching got backwards.

The PDFs themselves stay out of the repository, permanently.

Talks are read four ways

Slides can be quoted, a frame from the recording can only be shown, captions
are verbatim with respect to the caption track and nothing more, and a
viewer's account is a note. The four are kept apart on the page, because an
automatic caption is not a quotation and a picture of a slide is not text.

Also here: a page per database system, per paper and per year of bugs; charts; a review queue for what no rule settled, down to nine items; JSON Schema over every data file; and 281 tests.

Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N

mrigger and others added 2 commits September 13, 2026 23:43
… about it

The project's reach was folklore -- a bug count somebody remembered, a sense
that papers cite it. This builds it as data instead: 2,130 records, each one
carrying the source it came from and the reason it counts, collected by a
pipeline that can be re-run and disagreed with.

What the section says

  2,038 bugs across 41 database systems, 541 of them found by people outside
  the lab. 22 projects whose own evidence shows them running SQLancer and 9
  more that have proposed it. 209 external papers, 69 built on the tool or its
  techniques, 34 calling it state of the art. Nine recorded talks where
  somebody stood up and described it.

  The bug count is published as a floor, and the page says so. Closed-source
  systems report nothing, some open-source ones (Materialize) report elsewhere,
  and a bug report need not name the tool that found it -- six projects that
  demonstrably run SQLancer contribute no counted bug at all, and the page
  names all six rather than implying the gap away.

How a claim gets in

  Every record cites a source and an attribution rule, and the rules are
  ranked: a project saying "found by SQLancer" outranks a technique name,
  which outranks the fact that a known reporter was running a campaign. The
  weakest rule alone admits nothing.

  Where prose is ambiguous an LLM classifies, under three constraints that
  shaped most of the design: it answers about text it was given and cannot
  fetch more; it never supplies a quotation, only cites the id of a sentence
  already extracted, which makes fabricated evidence structurally impossible
  rather than caught afterwards; and it may answer "uncertain", which routes
  the candidate to a person instead of publishing a guess. Answers are cached
  by content hash and committed, so the dataset rebuilds offline and a rerun
  costs nothing.

  Nothing is published on a name match alone. An acronym like TLP or CERT
  needs corroborating context; a reproducer that generates `t0(c0 ...)` is a
  fingerprint, but 556 of the records rest on it together with a known
  reporter, never on the shape by itself.

Papers are read, not counted

  `_data/papers/` holds one file per paper: the sentences that mention
  SQLancer, the citation markers that resolve to it -- which is how a paper
  saying "existing works [9]-[11]" is found at all -- what its artifact
  contains, and the classification each sentence supports. This is what
  separates a paper that builds on SQLancer from one that runs it as a
  baseline, a distinction name-matching got backwards.

  The PDFs themselves stay out of the repository, permanently.

Talks are read four ways

  Slides can be quoted, a frame from the recording can only be shown, captions
  are verbatim with respect to the caption track and nothing more, and a
  viewer's account is a note. The four are kept apart on the page, because an
  automatic caption is not a quotation and a picture of a slide is not text.

Also here: a page per database system, per paper and per year of bugs; charts;
a review queue for what no rule settled, down to nine items; JSON Schema over
every data file; and 281 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N
…pports

Two failures on the first run of workflows that had never run anywhere, and
both were real rather than incidental to CI.

pypdf was missing. It is not an optional extra: intake reads a supplied PDF's
title and first page with it, fulltext extracts paper text with it, and the
artifact scanner finds links with it. Every one of those wraps the import in a
try/except and returns "nothing found" when it fails, which is right at runtime
and dangerous in a dependency list -- the weekly job installs from
requirements.txt, so it would have run green while quietly extracting no paper
text at all. The test that builds a PDF fixture was the only caller that said
so out loud. It is now listed, and that test skips rather than fails without
it, because a checkout without pypdf is a supported state.

The hand-written `pip install jsonschema` is what let the two drift, so CI now
installs from requirements.txt like the other workflow already did.

Ruby 3.2 cannot build this site. The github-pages gem pins liquid 4.0.3, which
calls Object#tainted?, removed in Ruby 3.2 -- so the build died in a 2023 blog
post that has nothing to do with any of this. Its jekyll 3.9.0 also predates
the Psych 4 that ships with Ruby 3.1. Ruby 3.0 is the newest that satisfies
both, and is effectively what GitHub Pages builds with; the runner moves to
22.04 because setup-ruby ships no 3.0 for 24.04. Nothing about the site
changes -- the pin was simply newer than the gems it had to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvUdZCeqsafccQ8rt7jt3N
@mrigger
mrigger merged commit e6a07de into main Sep 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant