PaperFlow verbatim-quote validation rejects faithful quotes due to PDF extraction artifacts (whitespace, ligatures, hyphenation, page numbers, chunk boundaries)

[russellsteb-tech]

Summary

_validate_research_draft in quantmind/flows/_paper_summary.py requires a research finding's quote to be an exact substring of the cited chunk's text (finding.quote not in chunk.text). With PDF-sourced papers, pymupdf's extracted text contains layout artifacts inside sentences, so a model that quotes the words faithfully — but normalizes layout — fails the check, and PaperFlow.build() raises ValueError: research finding quote is not present in its chunk.

We hit this repeatedly running PaperFlow on the repo's own example paper (arXiv 1706.03762v7) with a non-OpenAI model (kimi-k3 via the litellm path), at temperature 0 and 1. Every rejected quote we inspected was word-for-word present in the paper; none was fabricated.

Artifact classes observed (all real cases from 1706.03762v7)

1. Line breaks / irregular indentation inside sentences — extracted text: "permission to\nreproduce the tables and figures in this paper solely for use in journalistic or\n scholarly works."; the model quotes the sentence with single spaces.
2. Ligature glyphs — pymupdf emits ff/fi (e.g. effective); models write effective. NFKC normalization folds these.
3. End-of-line hyphenation — "English-\nto-German" (hyphen must be kept when joining) vs. cases where the hyphen should be dropped. Neither plain whitespace collapsing nor unconditional dehyphenation handles both.
4. Page numbers interleaved mid-sentence at page breaks — "...faster than recurrent layers when the sequence\n6\n\nlength n is smaller than..." (the 6 is the page number). A faithful quote naturally omits it.
5. Chunk-boundary spans and near-miss chunk attribution — with chunk_size=512, chunk_overlap=64, a faithful quote can cross the cited chunk's boundary, and models sometimes cite a neighboring chunk index within their group. The quote is verbatim in the material the agent was shown, but not within the single cited chunk.

Suggested direction

Keep the anti-fabrication guarantee (every quoted fragment must appear verbatim in the text the agent saw) but compare on a normalized form:

• NFKC-normalize + collapse whitespace runs on both sides before the substring test;
• try both hyphenation joins for end-of-line hyphens (-\n → "" and -\n → -);
• strip standalone page-number lines;
• as a fallback, match against the overlap-deduplicated concatenation of the agent's chunk group (treating chunk_index/page_number as provenance hints rather than exact pointers), so boundary-spanning quotes pass.

We implemented exactly this as a local wrapper around _validate_research_draft and confirmed: (a) all previously rejected faithful quotes pass, (b) a deliberately fabricated quote is still rejected, and (c) PaperFlow then completes end-to-end on 1706.03762v7. Happy to turn it into a PR if the direction sounds right.

Environment

• quant-mind @ 10e9dbd0 (v0.2.0), editable install, Python 3.12.13 (aarch64 linux)
• Model: litellm/moonshot/kimi-k3 (also reproduced with a local qwen3:14b via litellm/ollama_chat/...)
• Same behavior at temperature=0.0 and 1.0