Skip to content

Improve Python comments filtering#96

Open
Vladyslav-Kuksiuk wants to merge 2 commits into
improve-csharp-filteringfrom
improve-python-filtering
Open

Improve Python comments filtering#96
Vladyslav-Kuksiuk wants to merge 2 commits into
improve-csharp-filteringfrom
improve-python-filtering

Conversation

@Vladyslav-Kuksiuk

Copy link
Copy Markdown
Collaborator

This PR improves Python comments filtering.

Resolves this issue.

@Vladyslav-Kuksiuk Vladyslav-Kuksiuk self-assigned this Jul 6, 2026
@Vladyslav-Kuksiuk Vladyslav-Kuksiuk marked this pull request as ready for review July 6, 2026 14:53
@Oleg-Melnik

Oleg-Melnik commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Claude Review

LGTM — nothing blocking found.

Checked and confirmed clean:

  • Escapes: true is correct for Python, including the raw-string trap: even in r"""...""", the tokenizer treats backslash-quote as non-terminating (the backslash stays in the value), so escape-skipping in textBlockEnd matches CPython's lexing for both regular and raw triple-quoted strings.
  • Prefixed literals (f, r, b, rb, …) work by construction: prefix letters fall through as code bytes and the """/''' delimiter still matches at the quote position.
  • Scan order: text blocks are consumed before single quotes, so ''' wins over ''+', and inline # is checked after strings — # inside any string form is preserved, while ''' inside a # comment doesn't open a phantom block.
  • The escaped-delimiter test traces correctly: """a\"""" — open (3) + a + escaped \" + close (3) consumes exactly the literal, leaving the trailing comment to be stripped.
  • Right altitude: reuses the TextBlockMarker mechanism from Improve Go comments filtering #94 instead of adding a fourth bespoke lexer; YAML/shell keep hashLineSyntax untouched.

One informational limitation, not a blocker: since Python 3.12 (PEP 701), a multi-line f-string can contain real # comments inside its {...} expression holes, e.g.

message = f"""
Total: {
    compute_total(items)  # real comment
}
"""

The filter treats the whole triple-quoted region as opaque text, so such a comment is kept instead of stripped. The failure direction is safe — extra text survives, string content is never corrupted, and the block still closes at the right delimiter — and handling it would require f-string-aware interpolation lexing (as done for C#). Fine to leave as a documented limitation until someone actually hits it.

@Oleg-Melnik Oleg-Melnik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vladyslav-Kuksiuk LGTM with comment from Claude.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants