feat(vuln-id): normalize Vulnerability_Id into a Vulnerability entity + ordered references (entity-only cutover)

[Maffooch]

Shortcut: sc-13887

Normalize Vulnerability_Id into a Vulnerability entity + ordered through-table

Replaces the per-finding Vulnerability_Id rows with a normalized Vulnerability registry
entity plus an ordered FindingVulnerabilityReference through-table. This is an entity-only
cutover: after the migration the entity/reference store is the single source of truth for all
reads and writes. There is no dual-write, no read feature flag, and no reconciliation command —
those were part of an earlier reversible design that has been dropped in favor of a clean conversion.

The legacy Vulnerability_Id model and its dojo_vulnerability_id table are retained (unused) for
a transition period — nothing reads or writes them after the cutover, but they are kept as a
frozen pre-cutover snapshot and removed together in a future release, once the entity store has
been validated in production. The destructive DROP TABLE is the one irreversible step here, so it
is deliberately not taken in the same release that introduces the entity.

Models

• Vulnerability — global registry of identifier strings (CVE-…, GHSA-…), stored EXACTLY
  as supplied (never case-normalized — the string is the natural key, the display form, and the
  hash input, so normalizing would churn hash_codes; case-insensitivity is an index concern via an
  Upper() index). Carries nullable EPSS/KEV columns for future enrichment (NULL = "not
  enriched"). Rows are never deleted by finding write paths — orphan entities are the (future)
  enrichment registry.
• FindingVulnerabilityReference — ordered Finding → Vulnerability link. order == 0 is the
  primary id (== Finding.cve), replacing the implicit cve == vulnerability_ids[0] coupling. Two
  UniqueConstraints: (finding, vulnerability) and (finding, order) — the latter is a
  structural guarantee of a single primary id per finding (never two order-0 rows), compatible
  with the delete-then-insert write path.

Vocabulary added in the model docstrings (Dependency-Track/OSV/Snyk convention): a Finding is
the occurrence in your environment; a Vulnerability is the definition it references. Porting
trap documented in place: legacy code filtering the reverse relation with strings via
vulnerability_id__in=<strings> must now go through vulnerability__vulnerability_id__in (a bare
vulnerability_id__in on a reference is now "FK pk in").

Write seam (dojo/vulnerability/manager.py)

Every writer persists ids through one module. Single-finding writes and the importer batch path both
delete-then-insert a finding's references inside a transaction, assigning order 0..n-1 from the
cleaned id list. finding.cve sync stays with the callers exactly as before. Entity rows are
get_or_created race-safely (bulk_create(ignore_conflicts=True) + refetch) — shared global id
strings collide routinely across concurrent imports and every caller converges on the surviving PK.

Hash invariant (unchanged, by construction)

Finding.get_vulnerability_ids() returns "".join(sorted(...)), so the new order column is
irrelevant to the hash, and the entity stores strings byte-identically to the legacy rows. New
imports still hash from the in-memory unsaved_vulnerability_ids (upstream of either store), so
import-time dedupe is unaffected. vulnerability_id_type is never part of the hash. No hash_code
recalculation is required by this change.

Migrations

• 0286 creates both tables EMPTY, so all indexes (including the two GIN indexes) build on zero
  rows — effectively instant, no AddIndexConcurrently. (A >50M-row install can strip the GIN
  indexes here and add them CONCURRENTLY after 0287; documented in the migration, not the default.)
• 0287 (atomic = False) runs the convergent backfill in
  dojo/vulnerability/backfill.py: entity inserts are insert-only with ON CONFLICT DO NOTHING; the
  reference build windows by finding_id and, per window, DELETEs that window's references then
  re-inserts them from the current legacy rows, inside one transaction per window. This is what makes
  a pre-run + deploy-time catch-up safe: ids that changed between the pre-run and the deploy are
  reconciled (stale references removed, not left to drift), and there is no unique_finding_order
  collision at occupied order slots. Each window commits independently, so a crash resumes cleanly
  and any re-run converges to a no-op. The legacy source tables are never rewritten, locked, or
  deleted.
• No drop migration in this PR. The legacy Vulnerability_Id model + dojo_vulnerability_id
  table are retained. The physical DROP TABLE is deferred to a future release and must be guarded
  there with a one-query safety invariant: abort the drop if dojo_vulnerability_id still has
  rows while the reference table is empty (i.e. the backfill never ran / never completed) — the only
  moment such a check can still prevent data loss.

PostgreSQL only

DefectDojo is a PostgreSQL-only application (the models rely on GIN / full-text indexes). The backfill
SQL therefore runs unconditionally and fails loudly on any other backend — the previous silent
vendor-skip is gone, so a backend that can't run the backfill fails at 0287 rather than silently
leaving the entity store empty.

Upgrade note (draft)

• Big installs may pre-run manage.py migrate_vulnerability_ids on the current release; 0287 is then
  a convergent catch-up during the deploy that also reconciles anything that changed in between.
• No manual reconciliation/verify step is needed post-cutover: parity is structural (the reference set
  mirrors the legacy per-finding row set), and the legacy table is retained as a fallback snapshot
  until the future drop (which carries the last-line invariant guard).

cve-only findings

A finding with a cve but zero legacy Vulnerability_Id rows (pre-entity drift, e.g. a directly
edited cve) gets no reference — the reference set mirrors the legacy per-finding row set exactly,
so get_vulnerability_ids() (and thus the hash) is byte-identical to what the legacy read produced.
The cve string still lands in the registry as an orphan Vulnerability entity; the cve field is
untouched.

Test evidence (unittests/test_vulnerability_id.py)

Entity read/write correctness, order/uniqueness constraints, dedup-identity (canonical, order-
independent hash input) across scanner-corpus and odd-string shapes, cve-only-has-no-reference, full
entity-only lifecycle, and copy-preserves-ids. Perf counts re-baselined on both sides (not
suppressed). Consumers on the Pro side move in the paired PR.

Not in scope (deferred)

• Finding.cve retirement — the field stays; order == 0 == cve keeps them in lockstep. Removal
  is its own follow-up (it is a public v2-API field and touches ~100s of call sites + 107 parsers).
• Enrichment writing to the entity's EPSS/KEV columns — columns exist, populated later; per-finding
  Finding.epss_score remains the current source until then.
• Explorer grouping by the entity FK instead of the id string — deferred perf refinement.
• Watson search index / classic server-rendered search — the classic search view already returns 410
  upstream, so the vuln-id lane there is inert.

---

⛔ Merge order: the Pro consumer PR DefectDojo-Inc/dojo-pro#2002 reads the entity
unconditionally (via a runtime import of the entity models). Merge this OSS PR first.

🤖 Generated with Claude Code