fix(chunkers): make SimpleTextSplitter fallback URL-safe

[MatthewZhuang]

Description

Fixes the dependency-free SimpleTextSplitter fallback used by the multi-modal file parsing pipeline.

The fallback called protect_urls() and restore_urls() without inheriting their implementation, causing AttributeError whenever the optional LangChain-backed chunkers were unavailable. This change:

• extracts the shared URL helpers into URLProtectionMixin;
• makes BaseChunker and SimpleTextSplitter reuse the same implementation;
• keeps URL placeholders atomic when a raw chunk boundary crosses them;
• prevents the final overlap from being emitted repeatedly as shrinking suffix chunks;
• adds focused regression coverage for fallback, URL, boundary, and overlap behavior.

This is the clean replacement for #2116, based directly on dev-v2.0.29 with only 3 related files changed.

Related Issue (Required): Fixes #2115

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Unit Test
• Test Script Or Test Steps

Local verification:

• ruff format --check on all changed Python files
• ruff check on all changed Python files
• git diff --check
• 9 focused SimpleTextSplitter test cases via a lightweight harness loading the actual changed modules
• 264 URL-boundary property smoke cases across chunk sizes, overlaps, and URL positions

The repository's full pytest environment was not available locally because the base interpreter lacked concurrent-log-handler; GitHub CI should run the complete matrix.

Checklist

• I have performed a self-review of my own code
• I have commented my code in hard-to-understand areas
• I have added tests that prove my fix is effective
• I have linked the issue to this PR
• I have mentioned the person who will review this PR

@WeiminLee please review this clean replacement for #2116.

Reviewer Checklist

• closes fix: SimpleTextSplitter fallback 在文件解析链路中报错：缺少 protect_urls #2115
• Made sure Checks passed
• Tests have been provided

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2200
Task: 9f2181dc08e1834a
Base: dev-v2.0.29
Head: fix/2115-simple-text-splitter-dev-v2.0.29

🔍 OpenCodeReview found 3 issue(s) in this PR.

---

1. src/memos/chunkers/simple_chunker.py (L42-L45)

When a chunk boundary falls inside a URL placeholder and placeholder_start - start <= chunk_overlap, _align_end_to_placeholder unconditionally returns placeholder_end, which can exceed start + chunk_size with no upper bound. For a long URL whose placeholder is, say, 40 characters wide, the emitted chunk can silently overshoot chunk_size by up to len(placeholder) - 1 characters. More critically, if a very long URL placeholder straddles a chunk boundary and placeholder_end itself is still in the middle of the text, the next chunk's next_start = max(start + 1, placeholder_end - chunk_overlap) may again land inside the same placeholder — triggering the same expansion on the next iteration and causing start to creep forward one character at a time through the placeholder, producing O(placeholder_length) near-duplicate micro-chunks until the placeholder is finally cleared.

Consider bounding the expansion or, at minimum, advancing next_start past placeholder_end when the end was adjusted to it:

💡 Suggested Change

Before:

            if placeholder_start < end < placeholder_end:
                if placeholder_start - start > chunk_overlap:
                    return placeholder_start
                return placeholder_end

After:

            if placeholder_start < end < placeholder_end:
                if placeholder_start - start > chunk_overlap:
                    return placeholder_start
                # Extend to include the full placeholder; the caller should
                # advance start past placeholder_end to avoid re-entering it.
                return placeholder_end

---

2. tests/chunkers/test_simple_chunker.py (L79)

The expected value ["X" * 100, "X" * 70] was verified against the current implementation by tracing the algorithm:

• Iteration 1: start=0, end=100. No separator found in "X"*150, so end stays 100. Chunk = "X"*100. next_start = max(1, 100-20) = 80.
• Iteration 2: start=80, end=min(80+100,150)=150 (last chunk, no separator search). Chunk = "X"*70. Break.

The assertion is correct and deterministic. No issue here — this is a valid regression guard.

💡 Suggested Change

Before:

    assert chunks == ["X" * 100, "X" * 70]

After:

    assert chunks == ["X" * 100, "X" * 70]  # verified against _simple_split_text: start=80 on second pass

---

3. tests/chunkers/test_simple_chunker.py (L70)

The upper bound < 10 is far too loose to be a meaningful regression guard. For a ~240-character input ('A'*95 + url(24 chars) + ' ' + 'B'*120) with chunk_size=100 and chunk_overlap=20, a correct implementation produces at most 3–4 chunks. A degenerate implementation that produces 9 near-empty chunks would still pass this assertion, defeating the stated purpose of verifying "sufficient progress".

Consider tightening the bound to reflect the expected chunk count for this specific input (e.g., <= 4).

💡 Suggested Change

Before:

    assert len(chunks) < 10, "splitter made insufficient progress around the URL boundary"

After:

    assert len(chunks) <= 4, "splitter made insufficient progress around the URL boundary"

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (9/9 executed). memos_python_core/changed-repo-python: 9/9. Duration: 5s [advisory, non-gating] AI-generated tests on branch test/auto-gen-9f2181dc08e1834a-20260803203805: 70/72 passed, 2 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/2115-simple-text-splitter-dev-v2.0.29