🐛 Use the CommonMark whitespace set, not the Python one

[Nexory]

Summary

str.strip() and str.split() without arguments use str.isspace(), which treats U+001C, U+001D, U+001E, U+001F and U+0085 as whitespace. CommonMark does not, and neither does String.prototype.trim, so upstream markdown-it keeps those characters. Ten call sites in this package relied on the Python behaviour, while common/utils.py already defines the correct set a few lines above them in MD_WHITESPACE.

Two consequences, both measured against markdown-it@14.1.0 on the same input:

A reference definition can resolve a usage that does not name it. normalizeReference collapsed the extra characters, so [a\x85b] and [a b] became the same label:

IN   [a b]

     [a\x85b]: http://example.com
PY   <p><a href="http://example.com">a b</a></p>
JS   <p>[a b]</p>

The characters are dropped from the output. They are content, not whitespace:

IN   x\x85            PY  <p>x</p>                    JS  <p>x\x85</p>
IN   # h\x85          PY  <h1>h</h1>                  JS  <h1>h\x85</h1>
IN   |a|\n|---|\n|c\x85|   PY  <td>c</td>             JS  <td>c\x85</td>
IN   ```py\x85rest    PY  class="language-py"         JS  class="language-py\x85rest"

The change

MD_TRIM_CHARS and mdTrim() in common/utils.py, used at the sites that were relying on str. In renderer.py the fence info string was also being cut with str.split(), which splits on the same wider set, so that becomes a split over MD_TRIM_CHARS too.

MD_TRIM_CHARS is what String.prototype.trim removes, minus U+FEFF. The set this module already defines, MD_WHITESPACE together with U+2000 to U+200A, turned out to be a strict subset of trim; the three missing characters are U+2028, U+2029 and U+FEFF. The first two behave identically in both implementations today, so they are included. U+FEFF is excluded on purpose: trim does remove it, and that produces exactly the label folding shown above, with the roles reversed.

IN   [a b]

     [a\ufeffb]: http://example.com
PY   <p>[a b]</p>
JS   <p><a href="http://example.com">a b</a></p>

I did not touch that direction, since making the port faithful there would mean importing the defect.

Deliberately not changed

• validateLink() in common/normalize_url.py also uses str.strip(), but there the wider Python set is stricter rather than looser. Measured: [a](\x85javascript:alert(1)) produces no link here and produces <a href="%C2%85javascript:alert(1)"> upstream. Aligning it would let more URLs through, so it stays as it is.
• rules_inline/backticks.py uses .strip() in the code span edge rule. The rendered output does differ between the two implementations for ` \x85 `, but I could not explain what upstream does there, and I would rather leave it than guess.
• markdown_it/utils.py:184 reads spec fixture files and is not on a parsing path.

Testing

tests/test_port/test_whitespace.py, 15 cases. Against the current master 12 of them fail on the assertion; the 3 control cases, which check that real whitespace is still trimmed and that a real space still separates the fence language from its attributes, pass before and after. That is the point of including them.

• Full suite: 981 passed before, 996 passed after, which is the 15 added cases and nothing else.
• Both numbers with pytest tests/, the command tox.ini runs.
• mypy strict: no issues in 66 source files.
• ruff check on the touched files: 2 findings before, the same 2 after. Both pre-existing.
• Measured in python:3.12-slim, digest sha256:2c941e860699f878900b0edc2403613c234d4b32eda3cc9fa7036991a2a63c4a.
• The 15 rendering cases above were produced by running both implementations and comparing the output byte for byte, not by transcribing a table. Before the change 11 of 15 differ, after it none do.

One note on the checklist

AGENTS.md asks for a CHANGELOG.md entry. The three most recently merged PRs, #389, #391 and #394, do not touch that file, and the entries there carry pull request links that only exist after merge, so I left it out rather than create a conflict at release time. Happy to add one if you would prefer it in the PR.

[chrisjsewell]

Thanks @Nexory, this was a well-argued fix and a careful write-up of what to leave alone. Your commit has been merged to master unchanged via #433, with your authorship preserved (this fork did not allow maintainer edits, so it could not be brought up to date in place). The only additions were hoisting the per-call regex to a module-level compiled pattern and a changelog entry crediting you. It will go out in the next release. Closing this in favour of #433.