Repository navigation
Commit d8af5ab
committed
gh-107862: Add Hypothesis property tests for the codecs machinery
The existing codec tests check each codec against fixed strings, and
test_basics runs the incremental and stream interfaces on "abc123" only.
This adds test_codecs_property.py, which runs every codec in
all_unicode_encodings through the incremental codecs (any chunking,
getstate/setstate round trips), StreamReader/StreamWriter, the error
handlers (backslashreplace decoded back through unicode_escape,
xmlcharrefreplace through html.unescape, surrogateescape round trips),
charmap_decode/charmap_encode against a small oracle, name normalization
in codecs.lookup, and the transform codecs, all on generated input and
compared against the stateless codec functions.
Six divergences turned up, each recorded as an expectedFailure with the
minimal input as an explicit example and carved out of the broad tests
only where the comments say so:
- euc_jp, euc_jis_2004, euc_jisx0213 and gb18030 incremental decoders
drop input after an incomplete trailing sequence under surrogateescape.
- The ISO-2022 encoders emit U+001B as a bare ESC (iso2022_kr also SO and
SI) which their decoders reject or misread.
- StreamReader.read() never decodes with final=True, so an incomplete
trailing character is silently dropped.
- The punycode incremental codecs transform each chunk on its own.
- euc_kr encodes U+3164 to the Hangul filler bytes it cannot decode alone.
- iso2022_jp_3 decodes plane 2 0x7D3B with the 2004 table because the
2000 emulation is followed by "if" rather than "else if".
The by-design differences (BOM-less utf-16/32 incrementally, utf-7 and
lone surrogates, the many-to-one compatibility mappings in cp932, cp950,
euc_jp and shift_jis, surrogateescape on non-ASCII-compatible codecs) are
documented in the module rather than hidden in the strategies.
With Hypothesis the file runs in about five seconds; 3000 examples per
test pass as well. Without it the explicit examples replay through the
stubs.1 parent 662beae commit d8af5ab
2 files changed
Lines changed: 825 additions & 0 deletions
File tree
- Lib/test
- Misc/NEWS.d/next/Tests
0 commit comments