gh-154719: Preserve trailing whitespace in t-string interpolation expressions

[lpyu001]

1. Interpolation.expression did not match its documented contract

Doc/library/string.templatelib.rst states that, for interpolations created by t-string literals, expression is

the expression text found inside the curly brackets ({ & }), including
any whitespace, excluding the curly brackets themselves, and ending before
the first !, :, or = if any is present.

Whitespace after { was preserved, but whitespace before }, !, : or a debug = was removed:

>>> t"{ x }".interpolations[0].expression
' x'      # expected ' x '
>>> t"{  x  }".interpolations[0].expression
'  x'     # expected '  x  '
>>> t"""{
...   x
... }""".interpolations[0].expression
'\n  x'   # expected '\n  x\n'

The asymmetry is not documented anywhere: leading whitespace was kept, trailing whitespace was dropped.

2. ast.unparse() could emit code that no longer parses

When the interpolation ended with an explicit line continuation, stripping the trailing whitespace also removed the newline that terminated it, leaving a dangling backslash in Interpolation.str:

>>> src = 't"""{x \\\n}"""'
>>> ast.parse(src).body[0].value.values[0].str
'x \\'                     # the line continuation lost its newline
>>> ast.unparse(ast.parse(src))
"t'{x \\}'"
>>> ast.parse(ast.unparse(ast.parse(src)))
SyntaxError: unexpected character after line continuation character

Root cause

_strip_interpolation_expr() in Parser/action_helpers.c was applied to every interpolation, not just debug ones. It walked backwards over the lexer metadata removing every trailing whitespace character and =, which is only correct for
the debug (=) form, where the trailing = and the text after it belong to the debug text rather than to the expression.

Solution

• Regular interpolations now use the lexer metadata verbatim.
• Debug interpolations keep the text that precedes the debug = and discard only the = itself plus the whitespace and explicit line continuations that follow it. Per the documentation the expression ends before the =, so t"""{value =\<newline>}""" yields 'value ', and the \<newline> stays in the debug text.
• If the debug marker turns out to be absent, the metadata is returned unchanged instead of silently dropping source text.

[pablogsal]

I am not sure this comment is accurate: after the lexer fix, removing comments can never drop the debug = (the marker cannot be inside a comment and the metadata always extends past it), so IIUC this branch is only reachable via things like a line continuation right after the =. Can you adjust the comment or add a test that reaches this branch?

[lpyu001]

Thanks, I will address it.

[lpyu001]

A valid case that reached
this branch is an explicit line continuation after the debug marker:

t'''{value =\
}'''

The first patch returned "value =\\\n" as the expression. This conflicts with the [Interpolation.expression documentation(https://docs.python.org/3.16/library/string.templatelib.html#string.templatelib.Interpolation.expression),which says the expression ends before =. The expected expression is "value ".

I updated the code to strip explicit line continuations after the debug marker and added regression tests.

[pablogsal]

The lexer fix also changes the debug text for f-strings (_PyLexer_set_ftstring_expr is shared when in_debug is set), e.g. a debug expression combining a comment with an escaped quote was truncated before this. Can you add a test for the f-string case in test_fstring as well? Maybe worth mentioning f-strings in the NEWS entry too.

[lpyu001]

I have added the tests and updated the NEWS file