What this is
signetry-eval ships a 60-case OWASP detection corpus. Roughly a quarter of it is
SAFE decoys — code that looks like the vulnerable pattern but isn't, with an empty
expected list. Any finding on a SAFE case is a false positive.
Decoys are the half of a detection corpus that keeps it honest. A scanner that flags
everything gets recall 1.0; only decoys catch it.
The verified gap
Counting SAFE decoys per language across
signetry_eval/detection/corpus/:
| language |
cases |
SAFE decoys |
| python |
24 |
4 |
| go |
10 |
4 |
| php |
8 |
3 |
| javascript |
6 |
2 |
| java |
7 |
1 |
| csharp |
3 |
0 |
| ruby |
2 |
0 |
C# and Ruby have detection cases and no false-positive probe at all. Precision on those
two languages is currently unmeasured — not good, not bad, unmeasured. Java has seven cases
and one decoy.
The deliverable
One Case added to the corpus, with expected=[]. Pick a rule that already exists (one of
the languages above, ideally C#, Ruby or Java) and write the safe counterpart to it.
The shape, from crafted.py — this is a real committed decoy:
Case(
id="CRAFT-16-SAFE-parameterised-sql",
family=Family.CRAFTED,
language="python",
title="SAFE: parameterised query, no interpolation",
provenance="Crafted SAFE decoy: correct parameterisation (false-positive probe).",
files={"safe_repo.py": '''\
import sqlite3, flask
app = flask.Flask(__name__)
@app.route("/lookup")
def lookup():
uid = flask.request.args.get("id")
cur = sqlite3.connect("d").cursor()
cur.execute("SELECT * FROM t WHERE id = ?", (uid,))
return str(cur.fetchall())
'''},
expected=[], # SAFE
),
Fields are defined in
corpus/schema.py.
expected=[] is what makes it SAFE.
Why this is a genuinely safe first contribution
A positive case asserts a finding, so adding one for a rule that doesn't exist yet turns
the suite red — the corpus asserts recall 1.0 at zero false positives, so a case must not
outrun the detector. A decoy asserts the absence of a finding. It can't fail for a missing
rule. Worst case it exposes a real false positive, which is exactly the bug worth having.
What makes a decoy good
It has to be tempting. The decoy should share the vulnerable case's imports, its shape, and
its taint source — user input flowing to the same sink — and differ only in the one thing that
makes it safe. Ideas by language:
- C# — parameterised
SqlCommand with AddWithValue next to a string-concatenated one;
RandomNumberGenerator rather than Random for a token; JsonSerializer with a safe
TypeInfoResolver vs. BinaryFormatter.
- Ruby — ActiveRecord
where("id = ?", params[:id]) vs. string interpolation; system
with an argument array rather than a shell string; ERB with escaping on.
- Java —
PreparedStatement with a bound parameter; SecureRandom; a path check via
Path.normalize().startsWith(base) before opening a file.
The trap to avoid: a decoy so obviously different that no scanner would ever flag it proves
nothing. If the vulnerable case reaches the sink through a helper function, so should the
decoy.
Acceptance criteria
git clone https://github.com/Signetry/eval && cd eval
uv sync
uv run pytest -q
Comment with the language you're taking. One decoy per PR is ideal — small, reviewable, and
each one moves a precision number that currently doesn't exist.
What this is
signetry-evalships a 60-case OWASP detection corpus. Roughly a quarter of it isSAFE decoys — code that looks like the vulnerable pattern but isn't, with an empty
expectedlist. Any finding on a SAFE case is a false positive.Decoys are the half of a detection corpus that keeps it honest. A scanner that flags
everything gets recall 1.0; only decoys catch it.
The verified gap
Counting SAFE decoys per language across
signetry_eval/detection/corpus/:C# and Ruby have detection cases and no false-positive probe at all. Precision on those
two languages is currently unmeasured — not good, not bad, unmeasured. Java has seven cases
and one decoy.
The deliverable
One
Caseadded to the corpus, withexpected=[]. Pick a rule that already exists (one ofthe languages above, ideally C#, Ruby or Java) and write the safe counterpart to it.
The shape, from
crafted.py— this is a real committed decoy:Fields are defined in
corpus/schema.py.expected=[]is what makes it SAFE.Why this is a genuinely safe first contribution
A positive case asserts a finding, so adding one for a rule that doesn't exist yet turns
the suite red — the corpus asserts recall 1.0 at zero false positives, so a case must not
outrun the detector. A decoy asserts the absence of a finding. It can't fail for a missing
rule. Worst case it exposes a real false positive, which is exactly the bug worth having.
What makes a decoy good
It has to be tempting. The decoy should share the vulnerable case's imports, its shape, and
its taint source — user input flowing to the same sink — and differ only in the one thing that
makes it safe. Ideas by language:
SqlCommandwithAddWithValuenext to a string-concatenated one;RandomNumberGeneratorrather thanRandomfor a token;JsonSerializerwith a safeTypeInfoResolvervs.BinaryFormatter.where("id = ?", params[:id])vs. string interpolation;systemwith an argument array rather than a shell string;
ERBwith escaping on.PreparedStatementwith a bound parameter;SecureRandom; a path check viaPath.normalize().startsWith(base)before opening a file.The trap to avoid: a decoy so obviously different that no scanner would ever flag it proves
nothing. If the vulnerable case reaches the sink through a helper function, so should the
decoy.
Acceptance criteria
Casewithexpected=[],idending-SAFE-<slug>, following the existing naming.provenancesays it's a false-positive probe and what makes the code safe.uv run pytest -qgreen — meaning the detector produces no finding on it.Open it as a bug and link this issue; the decoy is the evidence.
Comment with the language you're taking. One decoy per PR is ideal — small, reviewable, and
each one moves a precision number that currently doesn't exist.