Fix: classify.is_f_string()'s pre-3.12 fallback (added in PR #330) flags ANY... - #376
Open
M001N wants to merge 2 commits into
Open
Fix: classify.is_f_string()'s pre-3.12 fallback (added in PR #330) flags ANY...#376M001N wants to merge 2 commits into
M001N wants to merge 2 commits into
Conversation
added 2 commits
August 16, 2026 17:00
…Python <3.12 (PyCQA#367) is_f_string()'s pre-3.12 fallback (from PyCQA#330) matched any token whose string started with f"""/f''' as an f-string, regardless of bracket or assignment context. That flag was consumed generically by _get_unmatched_start_end_indices() to decide whether a token's start row should be pinned to the previous token's end row -- a fix-up that only makes sense for PEP 701's split FSTRING_START/MIDDLE/END token sequence (Python 3.12+). Before 3.12, an f-string is always a single STRING token, so there is nothing to stitch together and the check should never fire. Because it fired anyway, a multi-line f-string nested inside a parenthesized expression (e.g. `return (f"""...""",)`) had its row bookkeeping corrupted, causing tokenize.untokenize() to raise "ValueError: start precedes previous end" on Python 3.10/3.11 -- the crash reported in PyCQA#367. (This is unrelated to the PyCQA#364 bracket guard in is_attribute_docstring/_is_inside_brackets, which is never reached here since there is no preceding "=".) Verified the crash and the fix directly against Python 3.11.9 (embeddable distribution) since this environment only ships Python 3.12/3.14: - Pre-fix: tests/formatter/test_do_format_code.py::issue_367 raised ValueError: start (2,8) precedes previous end (3,4) on 3.11.9. - Post-fix: the same test passes on 3.11.9, and the full suite shows no new failures (the lone "preserve_line_ending" failure and the VIRTUAL_ENV-dependent end-to-end errors are pre-existing environment limitations, reproduced identically before this change). The regression test added in a prior commit (using the exact issue PyCQA#367 reproducer) now actually exercises the buggy pre-3.12 code path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrote is_f_string() so the pre-3.12 branch unconditionally returns False (with an explanatory comment) instead of doing the unguarded f"""/f''' string-prefix match. On Python <3.12 there is never a legitimate case where this row-stitching should fire, since f-strings aren't split into multiple tokens by that version's tokenizer.
Problem
PyCQA/docformatter issue reference: #367
Root Cause
classify.is_f_string()'s pre-3.12 fallback (added in PR #330) flags ANY token whose string starts with f"""/f''' as an f-string, with no bracket or assignment-context check. This predicate is consumed generically in format.py's _get_unmatched_start_end_indices(), which uses it to decide whether a token's start row should be pinned to the previous token's end row instead of incrementing normally -- a stitching fix-up that is only meaningful for Python 3.12+'s PEP 701 tokenizer, which splits an f-string into FSTRING_START/MIDDLE/END tokens that need to be glued back onto one row. On Python <3.12 an f-string is always a single STRING token, so this stitching never applies, but the naive prefix check made it fire anyway for the multi-line f-string nested inside
return (f"""...""",), corrupting that token's row bookkeeping and cascading into the untokenize ValueError. This is unrelated to the #364_is_inside_bracketsguard in is_attribute_docstring, which is never reached here because there is no '=' before the string in the repro -- confirming the previous attempt's root-cause claim was wrong, as the reviewer found.Testing
PASS: on Python 3.11.9 the issue_367 test now passes (previously crashed with the reported ValueError, verified both before and after the fix); full 3.11 suite shows 515 passed vs. only the pre-existing, fix-unrelated 'preserve_line_ending' failure and 36 VIRTUAL_ENV-dependent CLI-subprocess errors (both reproduced identically on unmodified HEAD, confirming they are environment limitations, not regressions). Python 3.14 full suite: 516 passed with the same single pre-existing failure.
Related Issue
#367