Skip to content

fix: keep docstring when PythonCodeSplitter would leave an empty body - #12356

Draft
Harsh23Kashyap wants to merge 1 commit into
deepset-ai:mainfrom
Harsh23Kashyap:fix/python-code-splitter-docstring-only-body
Draft

fix: keep docstring when PythonCodeSplitter would leave an empty body#12356
Harsh23Kashyap wants to merge 1 commit into
deepset-ai:mainfrom
Harsh23Kashyap:fix/python-code-splitter-docstring-only-body

Conversation

@Harsh23Kashyap

Copy link
Copy Markdown

Related Issues

Proposed Changes:

PythonCodeSplitter with strip_docstrings=True reduced a unit whose body was only the docstring to a header with no body (def foo():\n, class Foo:\n), which is a SyntaxError. The same bug applied to a class whose class-header slice contained only the class-level docstring and no other body statement before the first method: stripping the docstring from the header left class Foo:\n, also invalid.

The fix is in _strip_docstring: after confirming the first body statement is the docstring, also check that there is at least one other body statement within the slice. If not, return the original slice unchanged and None for the docstring, so the chunk stays valid Python and the docstring is not moved to meta["docstrings"] (consistent with how units that have nothing to strip are handled today). Functions, methods, and classes whose body has both a docstring and other statements are unaffected: the docstring is still stripped and moved to meta["docstrings"] as before.

How did you test it?

  • Added test_strip_docstrings_keeps_docstring_when_body_is_only_docstring in TestDocstringStripping, covering a function, a class, a nested class, and a method whose body is only the docstring. Each case asserts the emitted content (a) parses as valid Python via ast.parse, (b) still contains the docstring inline, and (c) did not move anything to meta["docstrings"].
  • Added test_strip_docstrings_keeps_class_header_docstring_when_no_other_body covering the class-header-with-only-docstring case (docstring + methods, but no other body statement before the first method).
  • The pre-existing test_strip_docstrings_moves_them_to_meta, test_strip_docstrings_preserves_module_docstring, and test_strip_class_header_docstring_moves_to_meta all still pass — the change only affects the empty-body edge case.
  • Full module: 71 passed (was 69). The wider preprocessor surface (test/components/preprocessors/) is 358 passed with no regressions. hatch run fmt clean. hatch run test:types reports 17 pre-existing errors in 9 unrelated files (azure, openai_counter, etc.) — confirmed pre-existing via git stash of this branch.
  • Verified the bug end-to-end with a standalone reproducer before the fix (4 cases produce SyntaxError; all 4 produce valid Python after the fix).

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the release note (reno note in releasenotes/notes/)
  • I have run hatch run fmt and hatch run test:types locally
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

`PythonCodeSplitter` with `strip_docstrings=True` reduced a unit whose
body was only the docstring to a header with no body (`def foo():\n`,
`class Foo:\n`), which is a `SyntaxError` and so the emitted chunk
could not be parsed back. The same bug applied to a class whose
class-header slice contained only the class-level docstring and no
other body statement before the first method (`class Foo: \\"\\"\\"doc\\"\\"\\"\n`
followed by methods): stripping the docstring from the header left
`class Foo:\n`, also invalid.

The fix is in `_strip_docstring`: after confirming the first body
statement is the docstring, also check that there is at least one other
body statement within the slice. If not, return the original slice
unchanged and `None` for the docstring, so the chunk stays valid
Python and the docstring is not moved to `meta["docstrings"]`
(consistent with how units that have nothing to strip are handled
today). Functions, methods, and classes whose body has both a
docstring and other statements are unaffected: the docstring is still
stripped and moved to `meta["docstrings"]` as before.

Two new tests cover the docstring-only-body case for a function, a
class, a nested class, a method, and the class-header-only case.

Fixes deepset-ai#12330
@Harsh23Kashyap
Harsh23Kashyap requested a review from a team as a code owner August 14, 2026 20:23
@Harsh23Kashyap
Harsh23Kashyap requested review from julian-risch and removed request for a team August 14, 2026 20:23
@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

@Harsh23Kashyap is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

Copy link
Copy Markdown
Contributor

Hi @Harsh23Kashyap, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12330 is already being addressed by open pull request(s) #12331, #12346. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @Harsh23Kashyap, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch August 14, 2026 21:28
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Aug 14, 2026
@HaystackBot
HaystackBot marked this pull request as draft August 14, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-pending PR is in draft until the contributor signs the CLA topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PythonCodeSplitter with strip_docstrings=True emits invalid Python for docstring-only function/method/class bodies

3 participants