Skip to content

Classify which SBOM report claims are supported by local evidence #117

Description

@stacknil

Context

The SBOM diff tool intentionally combines local change evidence, heuristic risk buckets, and report-level evidence confidence. An external review would help verify that the JSON and Markdown reports communicate those boundaries without implying facts that require vulnerability, reputation, provenance, or runtime evidence.

Use the checked-in CycloneDX examples and the current implementation/docs:

  • tools/sbom-diff-and-risk/examples/cdx_before.json
  • tools/sbom-diff-and-risk/examples/cdx_after.json
  • tools/sbom-diff-and-risk/examples/sample-report.json
  • tools/sbom-diff-and-risk/examples/sample-report.md
  • tools/sbom-diff-and-risk/src/sbom_diff_risk/risk.py
  • tools/sbom-diff-and-risk/src/sbom_diff_risk/evidence_confidence.py
  • tools/sbom-diff-and-risk/docs/dependency-risk-heuristics.md
  • tools/sbom-diff-and-risk/docs/case-study-bounded-policy-review.md

This is distinct from #103: that issue traces a requirements diff through expected local risk buckets; this one audits the claim/evidence boundary of the CycloneDX report surfaces without prescribing the expected classification.

Diagnostic question

Which reviewer-visible claims are supported directly by checked-in local evidence, which are heuristic inferences, and which would require external evidence? Do the JSON and Markdown surfaces express that distinction consistently, or is any wording/field behavior ambiguous or overly strong?

Evidence required

Produce a compact matrix with:

  • report field or rendered phrase;
  • originating code, policy, or documentation path;
  • local evidence available in the checked-in examples;
  • inference being made;
  • external evidence that would be required for a stronger conclusion;
  • classification as expected behavior, documentation ambiguity, or probable overclaim/defect.

Also record the default-branch commit, environment, exact local commands, and whether regenerated JSON and Markdown agree semantically. Please post the matrix before proposing a change.

Change threshold

A PR is warranted only after the evidence identifies a specific mismatch between the implemented output and the documented evidence boundary. Prefer a focused test, wording correction, or contract clarification over a new scoring model.

Behavior that must remain unchanged

  • Canonical package identity and opaque identifier handling.
  • Deterministic output and existing report/schema compatibility.
  • The distinction between observed dependency changes, heuristic prioritization, and external security verdicts.

Boundaries

  • Use checked-in examples only; no network, registry, CVE, reputation, or malware lookups.
  • Do not make claims about real package safety, compromise, or exploitability.
  • No new risk taxonomy, universal identifier normalization, or broad documentation rewrite is in scope.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededquestionFurther information is requested

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions