MAINT: P311-19 hash pinned classifier datasets with hashlib.file_digest

[ARJUN K (arjun-k047)]

Description

Addresses P311-19 from #3166.

compute_dataset_hashes() on LocalViolenceClassifierScorer and LocalRefusalClassifierScorer
read each pinned training CSV fully into memory before hashing it. It now streams each file through
hashlib.file_digest, so hashing no longer holds the whole file in memory. The returned
dict[str, str] is unchanged.

Hashing a 64 MiB file: peak Python memory 64.0 MiB before, 0.4 MiB after. The shipped CSVs are
small, so the benefit matters mainly for larger files.

Sites investigated and left unchanged:

• _load_training_rows() in both scorers hashes the bytes it then parses. Hashing separately would
  read the file twice and decouple the verified bytes from the trained-on bytes.
• LocalFileSurfaceSource._read_entry is a bounded, cancellable read loop that retains content
  (excluded by the item's own guidance).
• DataTypeSerializer.get_sha256 and MemoryInterface media digests run on async paths through
  storage I/O that may be remote, and reuse the content.
• FileDocumentStorage._compute_version hashes bytes the caller already holds.

Tests and Documentation

• Added test_compute_dataset_hashes_reads_current_files to both scorer test modules: it writes
  ~700 KB files (larger than file_digest's internal buffer) under a patched dataset path and checks
  the result against hashlib.sha256 of the same bytes. Existing pin tests still pass. These tests
  also pass on main, since this is a refactor; they guard that the returned hashes are unchanged.
• uv run -m pytest tests/unit/score/test_local_violence_classifier_scorer.py tests/unit/score/test_local_refusal_classifier_scorer.py:
  165 passed (main: 163 passed).
• uv run pre-commit run --files <changed files>: all hooks passed.
• No documentation changes: behaviour and return values are unchanged. JupyText not applicable.

AI assistance: I used Claude Code (Claude Opus 5.5) to find the hashing call sites, draft the
change and tests, and run the checks. I reviewed every changed line, the reasoning for each site
left unchanged, and the test results, and I can explain the change.

[ARJUN K (arjun-k047)]

@microsoft-github-policy-service agree