Keep flash summaries within the model's context and on their own section

[justi]

Problem

I index long PDFs in flash mode with a local model (Ollama, 8k context). As the document grows:

1. Prompts overrun the context. The document description gets the whole tree, a leaf summary gets all of its pages, and an expand prompt gets all of its pages. Ollama does not fail on an oversized prompt - it silently cuts it (keeps the first few tokens and the tail), so the model summarizes a fragment.
2. Model JSON with LaTeX breaks the summary. Models write LaTeX inside JSON strings without doubling the backslashes. An invalid escape (\alpha) makes the reply unparseable and the raw reply, fence and all, becomes the summary. A valid one (\times, \text) is decoded by json into a TAB or a carriage return.
3. A leaf summary covers its neighbours. A leaf is summarized from whole pages, so a section that starts mid-page also gets the end of the previous section and the start of the next one.

Changes

New behaviour is opt-in; without the two new options every prompt is byte-for-byte the same as on main (checked over the whole flash pipeline on two PDFs). The JSON fixes always apply.

client = PageIndexLocalClient(summary_max_input_tokens=8192,   # the indexing model's context
                              summary_scope="section")         # flash mode only
# or: PageIndexLocalClient(index={"summary_max_input_tokens": 8192, "summary_scope": "section"})

• JSON replies (parse_summary, parse_title): a field whose JSON does not parse is read from the raw reply; escapes are decoded with LaTeX in mind (inside $...$ a backslash before a letter is kept, outside it \t, \r, \b, \f before a lowercase letter are kept); a parsed field that came back with control characters is read raw instead.
• summary_max_input_tokens - the context size of the indexing model:
  • the document description is cut from the deepest tree level up until it fits,
  • a leaf too long for one call is summarized in parts (split at line boundaries) and the part summaries are combined,
  • an expand prompt that would overrun it is skipped with a warning (the node stays collapsed),
  • the size check counts digits one by one: per-digit tokenizers (Gemma, Llama) need more tokens for an index full of page numbers than tiktoken reports.
• summary_scope="section" (flash only): a leaf is summarized from the layout blocks between its heading and the next located heading.
  • headings of bookmark trees are located on the start page, or on a heading block one page off,
  • numbered headings are kept in the blocks,
  • an intro node starts at its parent's heading,
  • a leaf falls back to its pages when a section without a located heading may start inside it,
  • mode="standard" refuses it, like the other flash-only knobs.

Results

Flash mode, gemma4 E4B via Ollama on an RTX 4090 (8192 context). main vs this branch (with #549) and summary_max_input_tokens=8192, summary_scope="section". The book numbers are means over 3 runs per side.

	Llama 3 paper (92 p.)		DL theory book (471 p.)
	main	branch	main	branch
prompts cut by Ollama	4	0	7	0
summaries that are raw JSON	7	0	157	0
control characters from LaTeX	0	0	18	0
leaf summary words from its own section	43%	67%	45%	61%
leaf summary words found only in neighbouring sections	26.4%	3.6%	16.1%	2.7%
indexing time	74 s	70 s	266 s	242 s

A leaf's own section is the text from its heading block to the next heading block. The two content rows are the share of a summary's distinctive words found only in that text, or only in the rest of the leaf's pages.

Known limits:

• a JSON reply that does not parse and has a paragraph break between two dollar amounts ($5 ... $10) keeps a literal \n there (as on main);
• a real TAB before a lowercase letter in such a reply is read as a LaTeX command (none in the 294 replies I recorded);
• section text needs a located heading; an unlocated one falls back to pages, as on main.

Tests

New tests in tests/test_summary_replies.py, tests/test_summary_budget.py and tests/test_summary_scope.py; the full suite passes on every commit.

🤖 Generated with Claude Code