perf(parquet): reuse DataPageV2 assembly buffer for eager writes - #1192
perf(parquet): reuse DataPageV2 assembly buffer for eager writes#1192fallintoplace wants to merge 1 commit into
Conversation
0415d4a to
a240648
Compare
zeroshade
left a comment
There was a problem hiding this comment.
The core ownership change looks correct. Eager pages are consumed synchronously, dictionary-buffered pages retain independent backing storage, and compression, encryption, page-index, and release paths do not keep aliases to the reused buffer.
I found two issues to address:
- The optimization adds approximately one page of retained capacity per live V2 column writer, which materially increases peak memory for wide buffered row groups.
- The modified round-trip test replaces existing DataPageV1 coverage instead of adding DataPageV2 coverage.
Targeted DataPageV2 tests and race tests pass locally. The new benchmark also reproduces the intended allocation reduction. GitHub currently reports no CI checks for this head.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
| combined = owned.Bytes() | ||
| } else { | ||
| w.uncompressedData.Reset() | ||
| w.uncompressedData.Grow(combinedSize) |
There was a problem hiding this comment.
bytes.Buffer.Reset() retains capacity, so this introduces one page-sized high-water allocation for every live DataPageV2 column writer. That is not visible in the per-operation allocation benchmark.
In a buffered row group with 200 live columns and DataPageSize=1 MiB, post-GC heap usage increased by approximately 200 MiB versus the merge base—one retained MiB per column. This brings V2 in line with the existing V1 behavior, but it is still a material new peak-memory trade-off for wide schemas.
Could we avoid retaining scratch independently in every column writer, such as through pooled/shared eager-page scratch? At minimum, please clear uncompressedData in columnWriter.Close() and document the retained-memory trade-off alongside the allocation reduction.
| props = parquet.NewWriterProperties(parquet.WithDictionaryDefault(false), parquet.WithDataPageSize(pageSize)) | ||
| props = parquet.NewWriterProperties( | ||
| parquet.WithDictionaryDefault(false), | ||
| parquet.WithDataPageVersion(parquet.DataPageV2), |
There was a problem hiding this comment.
This converts the existing buffered, multi-page, dictionary-disabled regression test from the default DataPageV1 path to DataPageV2, removing its previous V1 coverage.
Please make the test table-driven over both DataPageV1 and DataPageV2 rather than replacing one with the other. For the V2 case, coverage with non-empty definition/repetition levels or compression would also better exercise concatenation into the reused buffer across page boundaries.
Rationale for this change
buildDataPageV2assembles the definition/repetition levels and encoded values into a newbytes.Bufferfor every page. When dictionary encoding is disabled or has fallen back, the page is written synchronously and the assembly storage can be reused. The current allocation pattern adds page-sized churn.What changes are included in this PR?
columnWriter.uncompressedDatafor eagerly written DataPageV2 pages.Medians from 6 runs on an Apple M1 Pro were:
Are these changes tested?
go test ./parquet/... -count=1go test ./parquet/file -run "^TestBufferedMultiPageDisabledDictionary$" -count=1go test ./parquet/file -run "^$" -bench "^BenchmarkWriteDataPageV2Eager$" -benchmem -count=6The multi-page DataPageV2 test covers the eager path, while the existing dictionary coverage keeps the retained-page path covered.
Are there any user-facing changes?
No. The Parquet output and public API are unchanged.