Skip to content

Add a SAFE decoy for C#, Ruby or Java (precision on those languages is unmeasured) #38

Description

@bkd-dotcom

What this is

signetry-eval ships a 60-case OWASP detection corpus. Roughly a quarter of it is
SAFE decoys — code that looks like the vulnerable pattern but isn't, with an empty
expected list. Any finding on a SAFE case is a false positive.

Decoys are the half of a detection corpus that keeps it honest. A scanner that flags
everything gets recall 1.0; only decoys catch it.

The verified gap

Counting SAFE decoys per language across
signetry_eval/detection/corpus/:

language cases SAFE decoys
python 24 4
go 10 4
php 8 3
javascript 6 2
java 7 1
csharp 3 0
ruby 2 0

C# and Ruby have detection cases and no false-positive probe at all. Precision on those
two languages is currently unmeasured — not good, not bad, unmeasured. Java has seven cases
and one decoy.

The deliverable

One Case added to the corpus, with expected=[]. Pick a rule that already exists (one of
the languages above, ideally C#, Ruby or Java) and write the safe counterpart to it.

The shape, from crafted.py — this is a real committed decoy:

Case(
    id="CRAFT-16-SAFE-parameterised-sql",
    family=Family.CRAFTED,
    language="python",
    title="SAFE: parameterised query, no interpolation",
    provenance="Crafted SAFE decoy: correct parameterisation (false-positive probe).",
    files={"safe_repo.py": '''\
import sqlite3, flask
app = flask.Flask(__name__)

@app.route("/lookup")
def lookup():
    uid = flask.request.args.get("id")
    cur = sqlite3.connect("d").cursor()
    cur.execute("SELECT * FROM t WHERE id = ?", (uid,))
    return str(cur.fetchall())
'''},
    expected=[],  # SAFE
),

Fields are defined in
corpus/schema.py.
expected=[] is what makes it SAFE.

Why this is a genuinely safe first contribution

A positive case asserts a finding, so adding one for a rule that doesn't exist yet turns
the suite red — the corpus asserts recall 1.0 at zero false positives, so a case must not
outrun the detector. A decoy asserts the absence of a finding. It can't fail for a missing
rule. Worst case it exposes a real false positive, which is exactly the bug worth having.

What makes a decoy good

It has to be tempting. The decoy should share the vulnerable case's imports, its shape, and
its taint source — user input flowing to the same sink — and differ only in the one thing that
makes it safe. Ideas by language:

  • C# — parameterised SqlCommand with AddWithValue next to a string-concatenated one;
    RandomNumberGenerator rather than Random for a token; JsonSerializer with a safe
    TypeInfoResolver vs. BinaryFormatter.
  • Ruby — ActiveRecord where("id = ?", params[:id]) vs. string interpolation; system
    with an argument array rather than a shell string; ERB with escaping on.
  • JavaPreparedStatement with a bound parameter; SecureRandom; a path check via
    Path.normalize().startsWith(base) before opening a file.

The trap to avoid: a decoy so obviously different that no scanner would ever flag it proves
nothing. If the vulnerable case reaches the sink through a helper function, so should the
decoy.

Acceptance criteria

  • One Case with expected=[], id ending -SAFE-<slug>, following the existing naming.
  • provenance says it's a false-positive probe and what makes the code safe.
  • It shares imports/framework/taint source with the corresponding vulnerable case.
  • uv run pytest -q green — meaning the detector produces no finding on it.
  • If it does produce a finding: don't delete the case. That's a real false positive.
    Open it as a bug and link this issue; the decoy is the evidence.
git clone https://github.com/Signetry/eval && cd eval
uv sync
uv run pytest -q

Comment with the language you're taking. One decoy per PR is ideal — small, reviewable, and
each one moves a precision number that currently doesn't exist.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions