fix: keep docstring when PythonCodeSplitter would leave an empty body - #12356
fix: keep docstring when PythonCodeSplitter would leave an empty body#12356Harsh23Kashyap wants to merge 1 commit into
Conversation
`PythonCodeSplitter` with `strip_docstrings=True` reduced a unit whose body was only the docstring to a header with no body (`def foo():\n`, `class Foo:\n`), which is a `SyntaxError` and so the emitted chunk could not be parsed back. The same bug applied to a class whose class-header slice contained only the class-level docstring and no other body statement before the first method (`class Foo: \\"\\"\\"doc\\"\\"\\"\n` followed by methods): stripping the docstring from the header left `class Foo:\n`, also invalid. The fix is in `_strip_docstring`: after confirming the first body statement is the docstring, also check that there is at least one other body statement within the slice. If not, return the original slice unchanged and `None` for the docstring, so the chunk stays valid Python and the docstring is not moved to `meta["docstrings"]` (consistent with how units that have nothing to strip are handled today). Functions, methods, and classes whose body has both a docstring and other statements are unaffected: the docstring is still stripped and moved to `meta["docstrings"]` as before. Two new tests cover the docstring-only-body case for a function, a class, a nested class, a method, and the class-header-only case. Fixes deepset-ai#12330
|
@Harsh23Kashyap is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @Harsh23Kashyap, thanks for your interest in contributing to Haystack! 🙏 This is an automated message to help us keep the review queue healthy. |
|
|
|
Hi @Harsh23Kashyap, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
Related Issues
Proposed Changes:
PythonCodeSplitterwithstrip_docstrings=Truereduced a unit whose body was only the docstring to a header with no body (def foo():\n,class Foo:\n), which is aSyntaxError. The same bug applied to a class whose class-header slice contained only the class-level docstring and no other body statement before the first method: stripping the docstring from the header leftclass Foo:\n, also invalid.The fix is in
_strip_docstring: after confirming the first body statement is the docstring, also check that there is at least one other body statement within the slice. If not, return the original slice unchanged andNonefor the docstring, so the chunk stays valid Python and the docstring is not moved tometa["docstrings"](consistent with how units that have nothing to strip are handled today). Functions, methods, and classes whose body has both a docstring and other statements are unaffected: the docstring is still stripped and moved tometa["docstrings"]as before.How did you test it?
test_strip_docstrings_keeps_docstring_when_body_is_only_docstringinTestDocstringStripping, covering a function, a class, a nested class, and a method whose body is only the docstring. Each case asserts the emitted content (a) parses as valid Python viaast.parse, (b) still contains the docstring inline, and (c) did not move anything tometa["docstrings"].test_strip_docstrings_keeps_class_header_docstring_when_no_other_bodycovering the class-header-with-only-docstring case (docstring + methods, but no other body statement before the first method).test_strip_docstrings_moves_them_to_meta,test_strip_docstrings_preserves_module_docstring, andtest_strip_class_header_docstring_moves_to_metaall still pass — the change only affects the empty-body edge case.71 passed(was 69). The wider preprocessor surface (test/components/preprocessors/) is358 passedwith no regressions.hatch run fmtclean.hatch run test:typesreports 17 pre-existing errors in 9 unrelated files (azure, openai_counter, etc.) — confirmed pre-existing viagit stashof this branch.SyntaxError; all 4 produce valid Python after the fix).Checklist
releasenotes/notes/)hatch run fmtandhatch run test:typeslocally