Skip to content

perf(arrow/csv): avoid materializing rows before writing - #1190

Merged
zeroshade merged 4 commits into
apache:mainfrom
fallintoplace:perf/csv-stream-rows
Aug 26, 2026
Merged

perf(arrow/csv): avoid materializing rows before writing#1190
zeroshade merged 4 commits into
apache:mainfrom
fallintoplace:perf/csv-stream-rows

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Writer.Write currently converts columns to strings, then copies every string header into a row-major [][]string before calling WriteAll. This allocates one []string for every row. Large record batches spend a lot of memory on this temporary matrix.

Decimal formatting also used math/big.Float, which adds extra work and allocations for normal decimal values.

What changes are included in this PR?

  • Keep the converted strings in their existing column slices.
  • Reuse one row-sized []string while writing each CSV record.
  • Format decimal values directly from their exact integer representation when the value fits the supported precision.
  • Keep the existing math/big.Float path for values outside the supported precision.
  • Preserve the existing flush and error behavior.

For the existing 1,000-row, 16-column benchmark, medians from 6 runs on an Apple M1 Pro were:

main this PR change
time/op 4.78 ms 0.84 ms -82.4%
B/op 2,292,279 432,927 -81.1%
allocs/op 41,514 19,568 -52.9%

Compared with the earlier version of this PR, the decimal formatting change reduces B/op by 78.5% and allocations by 51.7%:

before decimal formatting after decimal formatting change
B/op 2,012,183 432,927 -78.5%
allocs/op 40,515 19,568 -51.7%

Are these changes tested?

  • go test ./arrow/... -count=1
  • go test ./arrow/csv -run "^$" -bench "^BenchmarkWrite$" -benchmem -count=6

Are there any user-facing changes?

No. The generated CSV output is unchanged.

@fallintoplace
fallintoplace marked this pull request as ready for review August 24, 2026 10:35

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one blocking compatibility regression: CSV output is corrupted when a custom type converter reuses its result slice across column callbacks; see the inline comment for the reproducer and fix direction.


This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. If you think the finding is misapplied, please reply on the PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/csv/writer.go
}
for i, row := range rows {
recs[i][j] = row
columns[j] = rows

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: Retaining each converter result until all columns are transformed introduces an undocumented slice-lifetime requirement for WithCustomTypeConverter. A converter can currently reuse one scratch []string across callback invocations because the old implementation copied each result immediately. With two Int32 columns [1,2] and [10,20], such a converter produces 10,10 / 20,20 on this branch instead of 1,10 / 2,20; the same probe passes on the merge base. Please preserve the previous behavior—e.g. copy handled custom-converter results before invoking it for the next column—and add a regression test.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the converter-lifetime fix and the added decimal formatting optimization. Custom converter results are now copied before subsequent callbacks, and the regression covers scratch-slice reuse. CSV tests, race tests, repeated decimal suites, random all-scale decimal property checks, vet, formatting, and diff checks pass.

@zeroshade
zeroshade merged commit e117ecf into apache:main Aug 26, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants