Fix three toml-test 1.1.0 compliance gaps - #587
Open
thatch wants to merge 8 commits into
Open
Conversation
Bumps tests/toml-test to d168c2a (main), pulling in 23 new upstream
commits including duplicate-repeated-table and no-close-table array
coverage. Three of the new cases expose genuine tomlkit bugs and are
marked xfail(strict=True) with the toml-test commit/issue that added
them, so a fix will surface as an unexpected pass:
- valid/utf8-bom-{01,02}: leading UTF-8 BOM not stripped before parsing
(toml-test 542746b, toml-lang/toml-test#199)
- invalid/control/linetab-number-{01,02,03}: trailing \x0b after a
number is not rejected (toml-test 4f76d84, toml-lang/toml-test#195)
- invalid/{float,integer}/arabic-zero-*: Arabic-Indic digit zero (٠)
is accepted as a digit in several number positions (toml-test
d736b6f, toml-lang/toml-test#196)
… char
int()/float() silently strip leading/trailing whitespace-like characters
(e.g. \x0b, vertical tab) when converting a string, so a number token
like "1\x0b" parsed as valid instead of raising. _NUM_STOP does not
include \x0b, so the character was absorbed into the raw token rather
than ending it.
Guard _parse_number to reject any raw token containing a str.isspace()
character before attempting the int()/float() conversion. A
syntactically valid number token never contains whitespace, so this is
safe.
Fixes invalid/control/linetab-number-{01,02,03} (toml-test 4f76d84,
toml-lang/toml-test#195).
Python's int()/float() accept any Unicode decimal-digit character
(category Nd), such as Arabic-Indic zero (٠, U+0660), as equivalent
to ASCII digits. TOML numbers are ASCII-only, so tomlkit was
accepting invalid literals like `1٠` or `0.1٠`.
Fixes invalid/float/arabic-zero-{01,03,04} and
invalid/integer/arabic-zero-{01,02} (toml-test d736b6f,
toml-lang/toml-test#196).
A single leading BOM (U+FEFF) is allowed by TOML and must be ignored, but a BOM anywhere else in the document is invalid. Parser.__init__ now strips one leading BOM before handing the string to Source, and parse() re-attaches it to the first item's indent trivia so that as_string() still reproduces the original text. Fixes toml-test 542746b, toml-lang/toml-test#199.
All three known toml-test gaps it tracked (BOM, vertical-tab, and Arabic-Indic digit handling) are fixed, so the dict is permanently empty. Use plain pytest.param(..., id=case_id) instead.
A document consisting only of a leading BOM has no body item to attach the BOM to, so it's stored as its own Whitespace entry. Without fixed=True, Container's insertion logic treats it as discardable filler and inserts new top-level items before it, moving the BOM off the front of the file.
Ruff's RUF003 flagged the literal ٠ character in the comment as an ambiguous unicode character. Reference it by code point instead.
The pre-commit ruff-format hook wanted a valid_cases pytest.param call collapsed onto one line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the
tests/toml-testsubmodule to the latest toml-test corpus and fixes three parsing bugs it surfaced:Also fixes an additional edge case found while manually verifying the BOM fix: mutating a document that consists only of a leading BOM (no other content) moved the BOM to the end of the file instead of keeping it at the start.
Test plan
pytestpasses locally (1085 passed, 0 xfailed)xfailmarkers remaining