fix: decode CSV delimiter-sniff sample as a single stream - #4444
Open
2sumtech wants to merge 1 commit into
Open
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Bug fix for partition_csv() delimiter sniffing to decode sample bytes as a single stream, fixing UTF-16 crashes and mojibake. Includes focused tests, changelog entry, and version bump.
Re-trigger cubic
Delimiter sniffing in partition_csv() decoded its sample line-by-line: readlines() splits the raw bytes on 0x0A, which in multi-byte encodings like UTF-16 lands mid-code-unit, and only the first fragment carries the BOM. A UTF-16-LE file raised 'UnicodeDecodeError: truncated data' even when the correct encoding was passed; UTF-16-BE silently sniffed mojibake. Decode the joined sample with an incremental decoder (final=False) so a character split at the read boundary is buffered rather than raising.
2sumtech
force-pushed
the
fix/csv-delimiter-utf16
branch
from
August 24, 2026 21:44
4f24153 to
23d8a48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4443
Summary
partition_csv()crashed withUnicodeDecodeError: truncated dataon UTF-16-LE files even when the correctencodingwas passed, because_CsvPartitioningContext.delimiterdecoded its sniff sample line-by-line:readlines()splits raw bytes on0x0A, which lands mid-code-unit in UTF-16, and only the first fragment carries the BOM. UTF-16-BE didn't raise but silently fed the sniffer mojibake for every line after the first.Fix
Keep the whole-line read semantics from #2998, but join the line fragments back into one byte stream and decode once with
codecs.getincrementaldecoder(encoding)(...).decode(sample, final=False)— a character split at the read boundary is buffered instead of raising. Wrong-encoding inputs (e.g. latin-1 bytes decoded as utf-8) still raise exactly as before.Testing
test_partition_csv_with_utf_16_le_encoding(end-to-end) andand_it_auto_detects_the_delimiter_for_a_UTF_16_encoded_CSV_file— both fail with the exactUnicodeDecodeErrorwithout the fix and pass with it. Fulltest_csv.py: 36 passed (includes the #2643 long-line regression, semicolon, single-column, and BE-utf-16 cases);test_tsv.py15 passed;test_text.py50 passed.ruff check+ruff format --checkclean on touched files. CHANGELOG entry and version bump 0.26.3 → 0.26.4 included per repo convention.Disclosure: prepared with AI assistance (Claude Code); I reviewed the change and take responsibility for it.