fix(kb): scale chunk_overlap with the heading-reduced budget in MarkdownChunker

[L4XB]

Motivation / 动机

Fixes #9998. MarkdownChunker._sections_to_chunks deducts the inherited heading prefix from the chunk_size it hands to the recursive fallback chunker (effective_chunk_size = max(chunk_size // 4, chunk_size - prefix_len)), but passed the caller's chunk_overlap through unchanged. A valid external configuration such as chunk_size=256, chunk_overlap=100 therefore became overlap=100 >= effective_chunk_size=64 for a section under a long heading, and RecursiveCharacterChunker raised chunk_overlap must be less than chunk_size, failing a Markdown upload at the chunking stage. The same document chunks fine with include_heading_context=False.

Modifications / 改动点

• New MarkdownChunker._scale_overlap(chunk_overlap, chunk_size, effective_chunk_size): when the body budget was reduced, the overlap is scaled proportionally (100/256 → 25/64) and always kept strictly below the effective chunk size; when the budget is not reduced the overlap is untouched.
• _sections_to_chunks passes the scaled overlap to the fallback chunker.
• New tests/test_markdown_chunker.py: the reproducer from the issue (200-char parent heading, ## Child, 600 chars of body, chunk_size=256, chunk_overlap=100) now chunks without error and keeps all body text, also when the sizes are passed as chunk() overrides; include_heading_context=False is unchanged; parametrized cases for _scale_overlap. The reproducer tests fail on master and pass with the fix.

Verification / 验证

uv run pytest tests/test_markdown_chunker.py   (8 passed)
ruff check / ruff format --check on the changed files

Checklist / 检查清单

• 我已经确认了我的更改不会引入新的错误 / I have confirmed my changes do not introduce new errors
• 我已经添加了必要的测试 / I have added the necessary tests
• 我已经更新了相关文档 (if needed) / I have updated the relevant documentation (if needed)

Summary by Sourcery

Keep Markdown chunk overlap valid when inherited heading context reduces the available body budget.

Bug Fixes:

• Prevent Markdown chunking failures when heading context reduces the effective chunk size by scaling overlap proportionally and keeping it below the fallback chunk size.

Tests:

• Add regression coverage for long heading prefixes, chunk-size overrides, heading-context-disabled behavior, and overlap scaling.

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="tests/test_markdown_chunker.py" line_range="17" />
<code_context>
+    chunks = await chunker.chunk(LONG_HEADING_DOC)
+
+    assert chunks
+    assert "".join(chunks).count("B") >= 600
+    # The overlap now scales with the reduced budget instead of exceeding it.
+    assert all("B" * 65 not in chunk for chunk in chunks)
</code_context>
<issue_to_address>
**issue (testing):** The body-preservation assertions only check the total number of repeated `B` characters across all chunks. Because chunk overlap duplicates characters, the assertions still pass when a portion of the body is lost, so the regression tests can silently pass while chunking drops content.

**Triggers:** When the fallback chunker omits a body segment but produces enough overlapping duplicate `B` characters to keep the aggregate count at least 600.

**Suggested fix:** Use a body containing unique markers or assert exact coverage of the expected body sequence rather than counting repeated characters across overlapping chunks.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: tests/test_markdown_chunker.py:17

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook