perf(arrow/csv): avoid materializing rows before writing - #1190
Conversation
00bb6bc to
b651bf0
Compare
zeroshade
left a comment
There was a problem hiding this comment.
Found one blocking compatibility regression: CSV output is corrupted when a custom type converter reuses its result slice across column callbacks; see the inline comment for the reproducer and fix direction.
This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. If you think the finding is misapplied, please reply on the PR and a maintainer will weigh in.
More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
| } | ||
| for i, row := range rows { | ||
| recs[i][j] = row | ||
| columns[j] = rows |
There was a problem hiding this comment.
Blocking: Retaining each converter result until all columns are transformed introduces an undocumented slice-lifetime requirement for WithCustomTypeConverter. A converter can currently reuse one scratch []string across callback invocations because the old implementation copied each result immediately. With two Int32 columns [1,2] and [10,20], such a converter produces 10,10 / 20,20 on this branch instead of 1,10 / 2,20; the same probe passes on the merge base. Please preserve the previous behavior—e.g. copy handled custom-converter results before invoking it for the next column—and add a regression test.
zeroshade
left a comment
There was a problem hiding this comment.
Re-reviewed the converter-lifetime fix and the added decimal formatting optimization. Custom converter results are now copied before subsequent callbacks, and the regression covers scratch-slice reuse. CSV tests, race tests, repeated decimal suites, random all-scale decimal property checks, vet, formatting, and diff checks pass.
Rationale for this change
Writer.Writecurrently converts columns to strings, then copies every string header into a row-major[][]stringbefore callingWriteAll. This allocates one[]stringfor every row. Large record batches spend a lot of memory on this temporary matrix.Decimal formatting also used
math/big.Float, which adds extra work and allocations for normal decimal values.What changes are included in this PR?
[]stringwhile writing each CSV record.math/big.Floatpath for values outside the supported precision.For the existing 1,000-row, 16-column benchmark, medians from 6 runs on an Apple M1 Pro were:
Compared with the earlier version of this PR, the decimal formatting change reduces
B/opby 78.5% and allocations by 51.7%:Are these changes tested?
go test ./arrow/... -count=1go test ./arrow/csv -run "^$" -bench "^BenchmarkWrite$" -benchmem -count=6Are there any user-facing changes?
No. The generated CSV output is unchanged.