Repository navigation
Exactly-once parallel CSV, run-time schemas in Rust and C++, safer bindings, coverage gates - #149
Open
GabrielMarquezMatte wants to merge 42 commits into
Open
GabrielMarquezMatte wants to merge 42 commits into
GabrielMarquezMatte wants to merge 42 commits into
Conversation
…sing its start Every CSV chunk was guessed to start outside quotes, so one that began inside a quoted field found the wrong record start, or none, and was read again serially in the merge. On a CSV with quoted fields that is most chunks: a 600,000-row file with every field quoted re-read 14 of 16 chunks at four threads and ran at 0.57x the sequential speed. The first quote in the chunk's first 4 KB that is not half of a "" pair now decides: one followed by a delimiter or a line end closes a field, so the chunk starts inside one. The confirmation after the parse is unchanged, so a wrong inference still costs only a re-read; that now takes a quoted field that begins with the delimiter or a line break, or that runs past 4 KB without a quote. Same file, four threads: 58.8 to 18.5 ms with every field quoted, 34.8 to 14.3 ms with text fields quoted, 30.0 to 17.8 ms with quoted newlines; unquoted CSV is unchanged.
…nc or AggregateAsync failure A partitioned read binds rows without numbering them, so an ExcelParseException from ForEachAsync or AggregateAsync carried Row 0 on every format. The merges already walk the partitions in sheet order and know how many rows each read, failing one included, so they now place the failure: CSV from the records each chunk delivered, XLSX and XLSB from each chunk's row count. The XLSX recovery tail reports its failure through the same outcome as a chunk, so it is numbered the same way. ParseAsync and the parallel CSV enumerable share the renumbering helper.
…Async chunk fails Workers drained the whole chunk plan even after a chunk failed, so a body that threw on the first record still ran for nearly every other record in the file (19,811 of 20,000 in the new test). The merge only ever throws the first failure in sheet order, so that work was wasted and its side effects happened after the error. A shared gate records the lowest failed chunk index; chunks past it are skipped, and one in progress stops at its next record. Chunks before the failure still run to the end, so everything a sequential read would deliver first is delivered. If the failing chunk was misguessed and its re-read succeeds, the merge reads the skipped chunks itself.
… quoting A chunk's quote state was read off the first quote after its start that was not half of a "" pair, so a run like ,""",x" (a field opening with a literal quote) or a field starting with the delimiter or a line break was guessed wrong, and ForEachAsync delivered that chunk twice. The guess now works on whole runs of quotes, reaching back over a run the start cuts. Under RFC 4180 an odd run that opens a field follows a delimiter or a line start and one that closes it precedes a delimiter, a line end or the end of the source, so a run fitting only one decides. A run fitting both is no longer guessed: the chunk is left to the merge, which reads it from its confirmed start. The window grows from 4 KB to 64 KB and may look past the chunk's end. ForEachAsync is now exactly once on a CSV that follows RFC 4180 quoting with no quoted field past 64 KB, which a seeded random-CSV test checks over 25 files and five chunk sizes. Text after a closing quote, a stray quote in an unquoted field or a longer field still guess outside and keep the at-least-once fallback. None of the four benchmark files defers or misguesses a chunk, and four-thread timings match the previous commit within noise.
…ot define The XLSX and XLSB readers skipped a sheet whose r:id had no entry in the workbook's relationships part. When no sheet resolved, the empty list already threw, but in a workbook where only some sheets resolved the rest were dropped and the workbook opened as if they never existed. Both now throw InvalidDataException naming the sheet and the missing id, the exception the reader uses for any other corrupt package. The XLSB lookup goes through the dictionary's span lookup, so it no longer copies the id into a string first.
The rules lived only in the native README, so a C consumer reading the header had no way to know them, and the Rust crate's Send/Sync impls cited a guarantee the header never made. The header now says that a workbook handle takes calls from several threads at once, that cursors, typed readers, Arrow streams and writer handles are used by one thread at a time, that a closed handle returns XL_INVALID_HANDLE, and that the last error is per thread.
Both targets were registered in the harness and seeded, but only the PR smoke check ran them. They take raw bytes, so they run without a dictionary, and the -dict flag is now passed only when a target has one.
codecov.yml sets an 88% project target on each language's flag, and the
codecov/project/{csharp,python,rust} checks are now required on master.
Codecov posts those checks only for a commit it received an upload for,
and every upload is path-filtered, so a docs-only PR would wait on them
forever. When the .NET tests are skipped, the Linux test job now sends an
empty upload, which passes the checks on carried-forward coverage. Both
uploads in ci.yml fail the job on error, so a lost upload shows as a red
job instead of a check that never arrives.
The section on parallel CSV still said only quote-free sources were delivered exactly once and that GuessStart treated every chunk as starting outside quotes. Since 767db66 the worker infers the chunk's quote state from the runs of quotes around its start, and a chunk whose quotes fit both readings waits for its confirmed start. The section now describes that, and limits the caller obligation to the sources that break RFC 4180 quoting or carry a quoted field longer than 64 KB.
A reader coming from a document library had no way to tell, short of the API reference, that formula text, merged cells, comments, hyperlinks, charts, images and conditional formatting are out of scope. The README now says so, along with what is read (values, types, cached formula results, sheet names, visibility, tables) and the styling the writers support.
ExcelDataReader is the most-used reader on NuGet and the comparison a reader evaluating the library makes first, but no benchmark included it. ReadBenchmark now reads the XLSX and XLSB workbooks through it, and XlsReadBenchmark the XLS one, accumulating through the same IDataReader path as OfficeIMO.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #149 +/- ##
==========================================
- Coverage 91.30% 91.22% -0.09%
==========================================
Files 276 278 +2
Lines 21232 22633 +1401
Branches 3233 3652 +419
==========================================
+ Hits 19385 20646 +1261
- Misses 1352 1423 +71
- Partials 495 564 +69
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
codecov/patch defaulted to the base commit's coverage as its target, so a PR had to be covered better than the whole project (91.3% today) to pass, and #149 failed at 88.9%. It now uses the fixed 88% the project targets use.
AlreadyAtRecordStart returned false for offset 0, so GuessStart on a chunk starting at the source start scanned past the first terminator and skipped the first record. The merge always hands chunk 0 its confirmed start, so no read reached this, but the guess itself was wrong; the start of the source is a record start.
InferParity had no test for a run that the window cuts after a start inside it, for a run at index 0 with nothing behind it, or for a start inside a "" pair with no later quote; all three are unknown. The XLSX recovery tail was never long enough to reach its cancellation check, so a 5,000-row sheet now recovers early and reads the rest through it.
…type
_buffer_of discarded the byte length it measured, and NativeColumn.length
took table.row_count on trust, so ten I64 rows over a single array('q')
had the native writer read past an 8-byte copy, and an array('i') declared
I64 was written as whatever two int32s packed into one int64. Each value
buffer is now checked for a 1-D layout, the element size and signed or
float format its ColumnType needs, and exactly row_count elements (string
offsets: row_count + 1 int32s), and a validity bitmap for at least
row_count bits, before any pointer to it exists. The offsets' contents
were already validated natively.
write_workbook, encrypt_package and SheetWriter.close ignored what destination.write() returned, so a destination that took four bytes of a 2,286-byte XLSX left a four-byte file and the call still succeeded. All three now go through _write_all, which calls write() again for the rest. A write() returning None is taken to have written everything, as files that predate the int return convention do; one that returns 0, or more than was left, raises OSError instead of looping. SheetWriter.close still releases the native handle when that raises.
Sheet.rows handed the decoder buffer.raw, a copy of the whole buffer: 64 KiB per row at first, and more for every row after one that grew it. A slice to written.value copies only the row's own bytes.
The column_* accessors are safe functions, and __private exported XlColumn with public fields, so code under #![forbid(unsafe_code)] could point a column at any address and read it. The accessors and ColumnBinding::assign now take a ColumnRef, whose field is private to this crate and which only TableView builds over a table the native library returned; XlColumn is no longer exported. The derive's output is unchanged and still compiles under forbid(unsafe_code).
ColumnStat.Observe marked every date-styled cell as a date, so a column of 2024-01-02 12:30 inferred DateColumn and a parse with that schema dropped the time. A date cell whose value has a time of day, read through TryGetDateTime with the workbook's date system, now marks a timestamp; midnight and an out-of-range serial still mark a date, and a mix promotes the column to TimestampColumn.
GetName returned Column{i} for a column without a header, but
GetOrdinal only knew real headers, so GetOrdinal(GetName(0)) threw on a
headerless sheet. BuildOrdinals now adds the synthesized names too; a
real header of the same name keeps its ordinal.
assign_field only assigned a string when end > start, so a valid empty
cell left std::string name = "fallback" holding "fallback". A valid empty
value now assigns T{}; a null still leaves the default, as documented.
assign_field cast an int64 column value straight into the field, so 300 read into a uint8_t became 44, and the writers cast every integer to int64_t, so a uint64_t above INT64_MAX was written as a negative number. parse and TypedReader::next now check each integer field's column once, when the table is handed over, and return XL_INVALID_ARGUMENT naming the column and row; TableView::operator[] keeps returning T. write_sheet and WriterHandle::write refuse a value above INT64_MAX.
Bumps [syn](https://github.com/dtolnay/syn) from 3.0.3 to 3.0.9. - [Release notes](https://github.com/dtolnay/syn/releases) - [Commits](dtolnay/syn@3.0.3...3.0.9) --- updated-dependencies: - dependency-name: syn dependency-version: 3.0.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [rust_xlsxwriter](https://github.com/jmcnamara/rust_xlsxwriter) from 0.99.1 to 0.100.0. - [Changelog](https://github.com/jmcnamara/rust_xlsxwriter/blob/main/CHANGELOG.md) - [Commits](jmcnamara/rust_xlsxwriter@v0.99.1...v0.100.0) --- updated-dependencies: - dependency-name: rust_xlsxwriter dependency-version: 0.100.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.4.4 to 1.8.0. - [Release notes](https://github.com/rust-lang/cc-rs/releases) - [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md) - [Commits](rust-lang/cc-rs@cc-v1.4.4...cc-v1.8.0) --- updated-dependencies: - dependency-name: cc dependency-version: 1.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [trybuild](https://github.com/dtolnay/trybuild) from 1.0.120 to 1.0.122. - [Release notes](https://github.com/dtolnay/trybuild/releases) - [Commits](dtolnay/trybuild@1.0.120...1.0.122) --- updated-dependencies: - dependency-name: trybuild dependency-version: 1.0.122 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps Meziantou.Analyzer from 3.0.303 to 3.0.315 --- updated-dependencies: - dependency-name: Meziantou.Analyzer dependency-version: 3.0.315 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: analyzers ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps OfficeIMO.Excel from 3.4.5 to 3.5.1 --- updated-dependencies: - dependency-name: OfficeIMO.Excel dependency-version: 3.5.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: benchmarks ... Signed-off-by: dependabot[bot] <support@github.com>
…t/develop/syn-3.0.9 build(deps): bump syn from 3.0.3 to 3.0.9 in /rust
…t/develop/rust_xlsxwriter-0.100.0 build(deps): bump rust_xlsxwriter from 0.99.1 to 0.100.0 in /rust
…t/develop/cc-1.8.0 build(deps): bump cc from 1.4.4 to 1.8.0 in /rust
…t/develop/trybuild-1.0.122 build(deps): bump trybuild from 1.0.120 to 1.0.122 in /rust
infer_schema returned columns nothing could consume: every read needed a compile-time ExcelMapper type. Sheet now takes a slice of InferredColumn in parse_with_schema, parse_batches_with_schema, to_arrow_with_schema and to_arrow_stream_with_schema. A column with a name is matched by it, one without by its index. The non-Arrow reads return a Table whose columns are the Column values write_columns takes, so a table writes back as it was read.
infer_schema returned columns nothing could consume: every read needed a compile-time ExcelMapper type. parse, parse_batches, to_arrow and to_arrow_stream gain overloads taking a span of InferredColumn. A column with a name is matched by it, one without by its index. parse returns an xl::Table and parse_batches an xl::TableReader of them; a Table's columns are the ColumnRefs write_columns takes, so it writes back as it was read. TypedReader's iterator moves into detail::BatchIterator, shared by both readers; TypedReader<T>::iterator remains as an alias.
…ts/ExcelReader.Benchmarks/develop/benchmarks-dec10d3203 Bump the benchmarks group with 1 update
…elop/analyzers-e265310f8e Bump the analyzers group with 1 update
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parallel reads (
SheetParallel.ForEachAsync/AggregateAsync):Readers:
Run-time schemas. Until now
infer_schemareturned columns that nothing could read: every read needed a compile-timeExcelMappertype.A named column is matched by name, an unnamed one by index. A
Table's columns are whatwrite_columnstakes, so a table writes back the way it was read.Binding fixes:
column_*accessors andColumnBinding::assigntake an opaqueColumnRef, andXlColumnis no longer exported. Before this, safe code under#![forbid(unsafe_code)]could point a column at any address and read it. Derive output is unchanged.parseandTypedReader::nextreturnXL_INVALID_ARGUMENT, naming the column and row, for an integer the field cannot hold (300 intouint8_tused to become 44). The writers refuseuint64_taboveINT64_MAX. A valid empty string assignsT{}instead of leaving the field's default.write_workbook,encrypt_packageandSheetWriter.closefinish partialwrite()s, and awrite()that makes no progress raisesOSError.Sheet.rowsdecodes only the bytes of each row, not the whole 64 KiB+ buffer.Tooling and docs:
excelreader.hstates the thread-safety contract (workbook handle: any thread; cursors, typed readers, Arrow streams, writer handles: one thread at a time).deflateanddeflate-segments.cpp/include/xlsymlink. CI sends an empty upload when the path filter skips the .NET tests, so docs-only PRs still get the checks.ReadBenchmark/XlsReadBenchmarkcompare against ExcelDataReader.Evidence
Before: a CSV with every field quoted re-read 14 of 16 chunks at 4 threads (0.57x sequential). A body throwing on the first record still ran for 19,811 of 20,000 records.
After: 58.8 → 18.5 ms with every field quoted, 34.8 → 14.3 ms with text fields quoted, unquoted unchanged. The seeded random-CSV test sees no duplicate delivery over 25 files and 5 chunk sizes. The failing body stops the run.
Before: a workbook whose second sheet's
r:idis unresolved opens with one sheet.After:
XlsxSheetWithUnresolvedRelationshipThrows/XlsbSheetWithUnresolvedRelationshipThrowsfail on 7.1.0 and pass here.Before:
2024-01-02 12:30infersDateColumn; Pythonsheet.parse(sheet.infer_schema())returned day 19724, with no time.After:
SchemaInferenceTestscover 1900/1904, mixed columns, out-of-range serials and writer-producedDateTime.test_write_workbook_rejects_a_column_buffer_that_does_not_match,test_write_workbook_finishes_a_partial_write,test_a_writer_over_a_stream_that_stalls_raises_and_still_closes,test_rows_after_one_that_grew_the_buffer_decode_only_their_own_bytes.compile_faildoctest onColumnRef, andderive_ui/pass/forbid_unsafe.rs.cpp/tests/write.cpp.an_inferred_schema_parses_without_a_row_type,a_parsed_table_writes_back_unchanged,schema_batches_equal_the_whole_sheet(Rust);test_runtime_schema_matches_the_typed_parse,test_runtime_schema_batches_equal_whole_sheet(C++).Merge Danger
Door: two-way
Every change reverts cleanly. The required Codecov checks are branch-protection settings, already applied, and can be removed with
gh api. The newcppflag reports but is not required.Blast Radius: consumers
InferSchema/xl_infer_schemanow returnTimestampColumnfor date columns with a time of day. Code that matched onDateColumnfor those columns sees a new type.array('i')declared I64, short arrays, short bitmaps. Awrite()returning 0 now raises.parse/TypedReader::nextnow fail on out-of-range integers that used to wrap.__private::XlColumnno longer compiles. That module is#[doc(hidden)]and outside semver.ForEachAsynccallbacks on RFC 4180 CSV no longer see repeated records. Callbacks that compensated for repeats keep working.