Skip to content

Exactly-once parallel CSV, run-time schemas in Rust and C++, safer bindings, coverage gates - #149

Open
GabrielMarquezMatte wants to merge 42 commits into
masterfrom
develop
Open

GabrielMarquezMatte wants to merge 42 commits into
masterfrom
develop

Conversation

@GabrielMarquezMatte

@GabrielMarquezMatte GabrielMarquezMatte commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Summary

Parallel reads (SheetParallel.ForEachAsync / AggregateAsync):

 CSV chunk start
-  guess outside quotes, re-read serially when wrong
+  infer quote state from the quote runs around the start (64 KB window)
+  run fits open and close → no guess, wait for the confirmed start
 on chunk failure
-  workers drain the whole plan, exception reports Row 0
+  later chunks skip, the running one stops at its next record
+  exception reports the row a sequential read would

Readers:

 XLSX / XLSB sheet whose r:id is missing from the workbook rels
-  sheet silently dropped, workbook opens without it
+  InvalidDataException naming the sheet and the id
 InferSchema on a date cell with a time of day
-  DateColumn, the time is dropped on parse
+  TimestampColumn; midnight stays DateColumn, a mix promotes to timestamp
 SheetDataReader without a header
-  GetName(0) == "Column0", GetOrdinal("Column0") throws
+  GetOrdinal resolves the synthesized name; a real header of the same name wins

Run-time schemas. Until now infer_schema returned columns that nothing could read: every read needed a compile-time ExcelMapper type.

 Rust  Sheet
+  parse_with_schema(&[InferredColumn])          -> Table
+  parse_batches_with_schema(..)                 -> TableBatches
+  to_arrow_with_schema / to_arrow_stream_with_schema
 C++   xl::Sheet
+  parse(span<InferredColumn>)                   -> xl::Table
+  parse_batches(span<InferredColumn>)           -> xl::TableReader
+  to_arrow / to_arrow_stream overloads

A named column is matched by name, an unnamed one by index. A Table's columns are what write_columns takes, so a table writes back the way it was read.

Binding fixes:

  • Rust: the column_* accessors and ColumnBinding::assign take an opaque ColumnRef, and XlColumn is no longer exported. Before this, safe code under #![forbid(unsafe_code)] could point a column at any address and read it. Derive output is unchanged.
  • C++: parse and TypedReader::next return XL_INVALID_ARGUMENT, naming the column and row, for an integer the field cannot hold (300 into uint8_t used to become 44). The writers refuse uint64_t above INT64_MAX. A valid empty string assigns T{} instead of leaving the field's default.
  • Python: before any native pointer exists, the writer checks each column buffer's layout, element size, format and length, plus the validity bitmap size. write_workbook, encrypt_package and SheetWriter.close finish partial write()s, and a write() that makes no progress raises OSError. Sheet.rows decodes only the bytes of each row, not the whole 64 KiB+ buffer.

Tooling and docs:

  • excelreader.h states the thread-safety contract (workbook handle: any thread; cursors, typed readers, Arrow streams, writer handles: one thread at a time).
  • libFuzzer now also runs deflate and deflate-segments.
  • Codecov: 88% project target per flag (C#, Python, Rust, C++), and patch also held to 88% instead of the base commit's coverage. C++ coverage comes from a Linux Debug build with gcov + gcovr, mapped through the cpp/include/xl symlink. CI sends an empty upload when the path filter skips the .NET tests, so docs-only PRs still get the checks.
  • ReadBenchmark / XlsReadBenchmark compare against ExcelDataReader.
  • README states what is read and written. ARCHITECTURE.md describes the exactly-once CSV path. The Rust and C++ READMEs show reading through an inferred schema.

Evidence

  • Parallel CSV
    Before: a CSV with every field quoted re-read 14 of 16 chunks at 4 threads (0.57x sequential). A body throwing on the first record still ran for 19,811 of 20,000 records.
    After: 58.8 → 18.5 ms with every field quoted, 34.8 → 14.3 ms with text fields quoted, unquoted unchanged. The seeded random-CSV test sees no duplicate delivery over 25 files and 5 chunk sizes. The failing body stops the run.
  • Dangling relationships
    Before: a workbook whose second sheet's r:id is unresolved opens with one sheet.
    After: XlsxSheetWithUnresolvedRelationshipThrows / XlsbSheetWithUnresolvedRelationshipThrows fail on 7.1.0 and pass here.
  • Schema inference
    Before: 2024-01-02 12:30 infers DateColumn; Python sheet.parse(sheet.infer_schema()) returned day 19724, with no time.
    After: SchemaInferenceTests cover 1900/1904, mixed columns, out-of-range serials and writer-produced DateTime.
  • Bindings: every fix landed with a regression that failed before it.
    • Python: test_write_workbook_rejects_a_column_buffer_that_does_not_match, test_write_workbook_finishes_a_partial_write, test_a_writer_over_a_stream_that_stalls_raises_and_still_closes, test_rows_after_one_that_grew_the_buffer_decode_only_their_own_bytes.
    • Rust: a compile_fail doctest on ColumnRef, and derive_ui/pass/forbid_unsafe.rs.
    • C++: overflow and empty-string cases in cpp/tests/write.cpp.
  • Run-time schemas: an_inferred_schema_parses_without_a_row_type, a_parsed_table_writes_back_unchanged, schema_batches_equal_the_whole_sheet (Rust); test_runtime_schema_matches_the_typed_parse, test_runtime_schema_batches_equal_whole_sheet (C++).
  • Suites: .NET 3778/3778, CTest 10/10, Python 217 passed / 1 xfailed. C# line coverage 94.1%.

Merge Danger

Door: two-way

Every change reverts cleanly. The required Codecov checks are branch-protection settings, already applied, and can be removed with gh api. The new cpp flag reports but is not required.

Blast Radius: consumers

  • A workbook with a sheet entry pointing at a missing relationship now throws instead of opening without that sheet. Excel treats such a file as corrupt.
  • InferSchema / xl_infer_schema now return TimestampColumn for date columns with a time of day. Code that matched on DateColumn for those columns sees a new type.
  • The Python writer now rejects buffers it used to accept and write wrong: array('i') declared I64, short arrays, short bitmaps. A write() returning 0 now raises.
  • C++ parse / TypedReader::next now fail on out-of-range integers that used to wrap.
  • Rust code that reached into __private::XlColumn no longer compiles. That module is #[doc(hidden)] and outside semver.
  • ForEachAsync callbacks on RFC 4180 CSV no longer see repeated records. Callbacks that compensated for repeats keep working.
  • A PR to master whose CI produces no Codecov upload and no empty upload would wait on the coverage checks.

…sing its start

Every CSV chunk was guessed to start outside quotes, so one that began
inside a quoted field found the wrong record start, or none, and was
read again serially in the merge. On a CSV with quoted fields that is
most chunks: a 600,000-row file with every field quoted re-read 14 of
16 chunks at four threads and ran at 0.57x the sequential speed.

The first quote in the chunk's first 4 KB that is not half of a ""
pair now decides: one followed by a delimiter or a line end closes a
field, so the chunk starts inside one. The confirmation after the parse
is unchanged, so a wrong inference still costs only a re-read; that now
takes a quoted field that begins with the delimiter or a line break, or
that runs past 4 KB without a quote.

Same file, four threads: 58.8 to 18.5 ms with every field quoted, 34.8
to 14.3 ms with text fields quoted, 30.0 to 17.8 ms with quoted
newlines; unquoted CSV is unchanged.
…nc or AggregateAsync failure

A partitioned read binds rows without numbering them, so an
ExcelParseException from ForEachAsync or AggregateAsync carried Row 0
on every format. The merges already walk the partitions in sheet order
and know how many rows each read, failing one included, so they now
place the failure: CSV from the records each chunk delivered, XLSX and
XLSB from each chunk's row count. The XLSX recovery tail reports its
failure through the same outcome as a chunk, so it is numbered the same
way.

ParseAsync and the parallel CSV enumerable share the renumbering helper.
…Async chunk fails

Workers drained the whole chunk plan even after a chunk failed, so a
body that threw on the first record still ran for nearly every other
record in the file (19,811 of 20,000 in the new test). The merge only
ever throws the first failure in sheet order, so that work was wasted
and its side effects happened after the error.

A shared gate records the lowest failed chunk index; chunks past it
are skipped, and one in progress stops at its next record. Chunks
before the failure still run to the end, so everything a sequential
read would deliver first is delivered. If the failing chunk was
misguessed and its re-read succeeds, the merge reads the skipped
chunks itself.
… quoting

A chunk's quote state was read off the first quote after its start that
was not half of a "" pair, so a run like ,""",x" (a field opening with a
literal quote) or a field starting with the delimiter or a line break
was guessed wrong, and ForEachAsync delivered that chunk twice.

The guess now works on whole runs of quotes, reaching back over a run
the start cuts. Under RFC 4180 an odd run that opens a field follows a
delimiter or a line start and one that closes it precedes a delimiter,
a line end or the end of the source, so a run fitting only one decides.
A run fitting both is no longer guessed: the chunk is left to the merge,
which reads it from its confirmed start. The window grows from 4 KB to
64 KB and may look past the chunk's end.

ForEachAsync is now exactly once on a CSV that follows RFC 4180 quoting
with no quoted field past 64 KB, which a seeded random-CSV test checks
over 25 files and five chunk sizes. Text after a closing quote, a stray
quote in an unquoted field or a longer field still guess outside and
keep the at-least-once fallback. None of the four benchmark files
defers or misguesses a chunk, and four-thread timings match the
previous commit within noise.
…ot define

The XLSX and XLSB readers skipped a sheet whose r:id had no entry in the
workbook's relationships part. When no sheet resolved, the empty list
already threw, but in a workbook where only some sheets resolved the rest
were dropped and the workbook opened as if they never existed.

Both now throw InvalidDataException naming the sheet and the missing id,
the exception the reader uses for any other corrupt package. The XLSB
lookup goes through the dictionary's span lookup, so it no longer copies
the id into a string first.
The rules lived only in the native README, so a C consumer reading the
header had no way to know them, and the Rust crate's Send/Sync impls cited
a guarantee the header never made. The header now says that a workbook
handle takes calls from several threads at once, that cursors, typed
readers, Arrow streams and writer handles are used by one thread at a time,
that a closed handle returns XL_INVALID_HANDLE, and that the last error is
per thread.
Both targets were registered in the harness and seeded, but only the PR
smoke check ran them. They take raw bytes, so they run without a
dictionary, and the -dict flag is now passed only when a target has one.
codecov.yml sets an 88% project target on each language's flag, and the
codecov/project/{csharp,python,rust} checks are now required on master.

Codecov posts those checks only for a commit it received an upload for,
and every upload is path-filtered, so a docs-only PR would wait on them
forever. When the .NET tests are skipped, the Linux test job now sends an
empty upload, which passes the checks on carried-forward coverage. Both
uploads in ci.yml fail the job on error, so a lost upload shows as a red
job instead of a check that never arrives.
The section on parallel CSV still said only quote-free sources were
delivered exactly once and that GuessStart treated every chunk as
starting outside quotes. Since 767db66 the worker infers the chunk's
quote state from the runs of quotes around its start, and a chunk whose
quotes fit both readings waits for its confirmed start. The section now
describes that, and limits the caller obligation to the sources that
break RFC 4180 quoting or carry a quoted field longer than 64 KB.
A reader coming from a document library had no way to tell, short of the
API reference, that formula text, merged cells, comments, hyperlinks,
charts, images and conditional formatting are out of scope. The README
now says so, along with what is read (values, types, cached formula
results, sheet names, visibility, tables) and the styling the writers
support.
ExcelDataReader is the most-used reader on NuGet and the comparison a
reader evaluating the library makes first, but no benchmark included it.
ReadBenchmark now reads the XLSX and XLSB workbooks through it, and
XlsReadBenchmark the XLS one, accumulating through the same IDataReader
path as OfficeIMO.
@codecov-commenter

codecov-commenter commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.64880% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.22%. Comparing base (0d952b7) to head (a3f53f6).

Files with missing lines Patch % Lines
rust/excelreader/src/workbook.rs 83.06% 32 Missing ⚠️
src/ExcelReader.Native/include/excelreader.hpp 89.05% 4 Missing and 11 partials ⚠️
...er.Core/Parser/ParallelCsv/ParallelCsvProcessor.cs 85.71% 2 Missing and 3 partials ⚠️
python/src/excelreader/writer.py 94.44% 2 Missing ⚠️
...elReader.Core/Parser/ParallelCsv/CsvChunkWorker.cs 95.65% 0 Missing and 1 partial ⚠️
...er.Core/Parser/SheetPipeline/XlsxPipelineSource.cs 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #149      +/-   ##
==========================================
- Coverage   91.30%   91.22%   -0.09%     
==========================================
  Files         276      278       +2     
  Lines       21232    22633    +1401     
  Branches     3233     3652     +419     
==========================================
+ Hits        19385    20646    +1261     
- Misses       1352     1423      +71     
- Partials      495      564      +69     
Flag Coverage Δ
cpp 90.24% <90.50%> (?)
csharp 91.35% <94.30%> (+0.03%) ⬆️
python 89.46% <94.87%> (+0.11%) ⬆️
rust 91.56% <85.52%> (-0.66%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

GabrielMarquezMatte and others added 17 commits October 20, 2026 19:05
codecov/patch defaulted to the base commit's coverage as its target, so a
PR had to be covered better than the whole project (91.3% today) to pass,
and #149 failed at 88.9%. It now uses the fixed 88% the project targets
use.
AlreadyAtRecordStart returned false for offset 0, so GuessStart on a
chunk starting at the source start scanned past the first terminator
and skipped the first record. The merge always hands chunk 0 its
confirmed start, so no read reached this, but the guess itself was
wrong; the start of the source is a record start.
InferParity had no test for a run that the window cuts after a start
inside it, for a run at index 0 with nothing behind it, or for a start
inside a "" pair with no later quote; all three are unknown. The XLSX
recovery tail was never long enough to reach its cancellation check, so
a 5,000-row sheet now recovers early and reads the rest through it.
…type

_buffer_of discarded the byte length it measured, and NativeColumn.length
took table.row_count on trust, so ten I64 rows over a single array('q')
had the native writer read past an 8-byte copy, and an array('i') declared
I64 was written as whatever two int32s packed into one int64. Each value
buffer is now checked for a 1-D layout, the element size and signed or
float format its ColumnType needs, and exactly row_count elements (string
offsets: row_count + 1 int32s), and a validity bitmap for at least
row_count bits, before any pointer to it exists. The offsets' contents
were already validated natively.
write_workbook, encrypt_package and SheetWriter.close ignored what
destination.write() returned, so a destination that took four bytes of a
2,286-byte XLSX left a four-byte file and the call still succeeded. All
three now go through _write_all, which calls write() again for the rest.
A write() returning None is taken to have written everything, as files
that predate the int return convention do; one that returns 0, or more
than was left, raises OSError instead of looping. SheetWriter.close still
releases the native handle when that raises.
Sheet.rows handed the decoder buffer.raw, a copy of the whole buffer:
64 KiB per row at first, and more for every row after one that grew it.
A slice to written.value copies only the row's own bytes.
The column_* accessors are safe functions, and __private exported
XlColumn with public fields, so code under #![forbid(unsafe_code)] could
point a column at any address and read it. The accessors and
ColumnBinding::assign now take a ColumnRef, whose field is private to this
crate and which only TableView builds over a table the native library
returned; XlColumn is no longer exported. The derive's output is
unchanged and still compiles under forbid(unsafe_code).
ColumnStat.Observe marked every date-styled cell as a date, so a column
of 2024-01-02 12:30 inferred DateColumn and a parse with that schema
dropped the time. A date cell whose value has a time of day, read through
TryGetDateTime with the workbook's date system, now marks a timestamp;
midnight and an out-of-range serial still mark a date, and a mix promotes
the column to TimestampColumn.
GetName returned Column{i} for a column without a header, but
GetOrdinal only knew real headers, so GetOrdinal(GetName(0)) threw on a
headerless sheet. BuildOrdinals now adds the synthesized names too; a
real header of the same name keeps its ordinal.
assign_field only assigned a string when end > start, so a valid empty
cell left std::string name = "fallback" holding "fallback". A valid empty
value now assigns T{}; a null still leaves the default, as documented.
assign_field cast an int64 column value straight into the field, so 300
read into a uint8_t became 44, and the writers cast every integer to
int64_t, so a uint64_t above INT64_MAX was written as a negative number.
parse and TypedReader::next now check each integer field's column once,
when the table is handed over, and return XL_INVALID_ARGUMENT naming the
column and row; TableView::operator[] keeps returning T. write_sheet and
WriterHandle::write refuse a value above INT64_MAX.
Bumps [syn](https://github.com/dtolnay/syn) from 3.0.3 to 3.0.9.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](dtolnay/syn@3.0.3...3.0.9)

---
updated-dependencies:
- dependency-name: syn
  dependency-version: 3.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [rust_xlsxwriter](https://github.com/jmcnamara/rust_xlsxwriter) from 0.99.1 to 0.100.0.
- [Changelog](https://github.com/jmcnamara/rust_xlsxwriter/blob/main/CHANGELOG.md)
- [Commits](jmcnamara/rust_xlsxwriter@v0.99.1...v0.100.0)

---
updated-dependencies:
- dependency-name: rust_xlsxwriter
  dependency-version: 0.100.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.4.4 to 1.8.0.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Changelog](https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md)
- [Commits](rust-lang/cc-rs@cc-v1.4.4...cc-v1.8.0)

---
updated-dependencies:
- dependency-name: cc
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [trybuild](https://github.com/dtolnay/trybuild) from 1.0.120 to 1.0.122.
- [Release notes](https://github.com/dtolnay/trybuild/releases)
- [Commits](dtolnay/trybuild@1.0.120...1.0.122)

---
updated-dependencies:
- dependency-name: trybuild
  dependency-version: 1.0.122
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps Meziantou.Analyzer from 3.0.303 to 3.0.315

---
updated-dependencies:
- dependency-name: Meziantou.Analyzer
  dependency-version: 3.0.315
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: analyzers
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps OfficeIMO.Excel from 3.4.5 to 3.5.1

---
updated-dependencies:
- dependency-name: OfficeIMO.Excel
  dependency-version: 3.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: benchmarks
...

Signed-off-by: dependabot[bot] <support@github.com>
…t/develop/syn-3.0.9

build(deps): bump syn from 3.0.3 to 3.0.9 in /rust
…t/develop/rust_xlsxwriter-0.100.0

build(deps): bump rust_xlsxwriter from 0.99.1 to 0.100.0 in /rust
…t/develop/cc-1.8.0

build(deps): bump cc from 1.4.4 to 1.8.0 in /rust
…t/develop/trybuild-1.0.122

build(deps): bump trybuild from 1.0.120 to 1.0.122 in /rust
infer_schema returned columns nothing could consume: every read needed a
compile-time ExcelMapper type. Sheet now takes a slice of InferredColumn in
parse_with_schema, parse_batches_with_schema, to_arrow_with_schema and
to_arrow_stream_with_schema. A column with a name is matched by it, one
without by its index.

The non-Arrow reads return a Table whose columns are the Column values
write_columns takes, so a table writes back as it was read.
infer_schema returned columns nothing could consume: every read needed a
compile-time ExcelMapper type. parse, parse_batches, to_arrow and
to_arrow_stream gain overloads taking a span of InferredColumn. A column
with a name is matched by it, one without by its index.

parse returns an xl::Table and parse_batches an xl::TableReader of them;
a Table's columns are the ColumnRefs write_columns takes, so it writes back
as it was read. TypedReader's iterator moves into detail::BatchIterator,
shared by both readers; TypedReader<T>::iterator remains as an alias.
…ts/ExcelReader.Benchmarks/develop/benchmarks-dec10d3203

Bump the benchmarks group with 1 update
…elop/analyzers-e265310f8e

Bump the analyzers group with 1 update
@GabrielMarquezMatte GabrielMarquezMatte changed the title Exactly-once parallel CSV, correct failure rows, dangling sheet relationships rejected, coverage gate Exactly-once parallel CSV, run-time schemas in Rust and C++, safer bindings, coverage gates Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants