fix: bound generic text reads

[dsfaccini]

This pull request was posted by codex-tui using gpt-5.6-sol on behalf of David.

Summary

• read only the paper-classification prefix
• stream generic word counting with existing whitespace semantics
• cap actual generic prompt and tokenizer reads at _FILE_CHAR_CAP
• reuse the capped path for evidence binding and cover each path with guarded-reader tests

Fixes #3260.

Validation

• uv run --frozen pytest tests/test_detect.py tests/test_file_slice.py tests/test_pdf_token_estimate.py -p no:xdist — pass
• changed-file pre-commit hooks — pass
• GRAPHIFY_MAX_WORKERS=2 uv run graphify update . — pass

Scope

Full-coverage document slicing and converted PDF/Office content remain unchanged; streaming those paths requires a separate design that preserves complete-document coverage.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Bounds file reads to avoid loading whole files into memory: count_words now streams the file in 64 KB chunks with str.split-equivalent whitespace semantics, and _looks_like_paper reads only its first 3000-char prefix rather than reading everything then slicing. Caps text extraction at _FILE_CHAR_CAP in _file_to_text for both PDF and plain-text paths, and routes _estimate_file_tokens through _file_to_text so estimation honors the same cap.

Worth a look

• _file_to_text now silently truncates returned file contents — graphify/llm.py:538 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• _estimate_file_tokens now catches wrong exception type from _file_to_text — graphify/llm.py:2099 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Token estimation now parses PDFs — graphify/llm.py:2099 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 2370 functions depend on the 707 functions this change touches.

Health — this change adds coupling hotspots:

• new: extract() — 520 callers, 43 callees
• new: _rebuild_code() — 113 callers, 50 callees
• new: detect() — 108 callers, 15 callees
• new: deduplicate_entities() — 63 callers, 21 callees
• new: build_merge() — 62 callers, 13 callees
• new: save_manifest() — 40 callers, 11 callees
• new: extract_files_direct() — 17 callers, 20 callees
• new: extract_xaml() — 19 callers, 17 callees
• …and 44 more — each is listed as a finding

Verification — 2370 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 1375 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify count\_words.

The verifier did not have enough to check count\_words, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_looks\_like\_paper.

The verifier did not have enough to check \_looks\_like\_paper, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_estimate\_file\_tokens.

The verifier did not have enough to check \_estimate\_file\_tokens, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `unit` is annotated `'Path | FileSlice'` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_file\_to\_text.

The verifier did not have enough to check \_file\_to\_text, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set

· 52 more finding(s) on lines outside this diff (see the check run).