Skip to content

perf(parquet): reuse DataPageV2 assembly buffer for eager writes - #1192

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-datapage-v2-buffer
Open

perf(parquet): reuse DataPageV2 assembly buffer for eager writes#1192
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-datapage-v2-buffer

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

buildDataPageV2 assembles the definition/repetition levels and encoded values into a new bytes.Buffer for every page. When dictionary encoding is disabled or has fallen back, the page is written synchronously and the assembly storage can be reused. The current allocation pattern adds page-sized churn.

What changes are included in this PR?

  • Reuse columnWriter.uncompressedData for eagerly written DataPageV2 pages.
  • Keep an owned buffer for pages retained while dictionary encoding is active.
  • Add a benchmark for 64 eager 64 KiB pages with uncompressed and Snappy codecs.
  • Exercise the eager DataPageV2 path in the existing multi-page round-trip test.

Medians from 6 runs on an Apple M1 Pro were:

codec metric main this PR change
uncompressed time/op 1.08 ms 650 us -39.6%
uncompressed B/op 4,547,602 383,763 -91.6%
uncompressed allocs/op 461 321 -30.4%
Snappy time/op 9.05 ms 8.88 ms -1.9%
Snappy B/op 3,027,460 489,403 -83.8%
Snappy allocs/op 459 325 -29.2%

Are these changes tested?

  • go test ./parquet/... -count=1
  • go test ./parquet/file -run "^TestBufferedMultiPageDisabledDictionary$" -count=1
  • go test ./parquet/file -run "^$" -bench "^BenchmarkWriteDataPageV2Eager$" -benchmem -count=6

The multi-page DataPageV2 test covers the eager path, while the existing dictionary coverage keeps the retained-page path covered.

Are there any user-facing changes?

No. The Parquet output and public API are unchanged.

@fallintoplace
fallintoplace force-pushed the perf/parquet-datapage-v2-buffer branch from 0415d4a to a240648 Compare August 17, 2026 21:06
@fallintoplace
fallintoplace marked this pull request as ready for review August 17, 2026 21:07

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core ownership change looks correct. Eager pages are consumed synchronously, dictionary-buffered pages retain independent backing storage, and compression, encryption, page-index, and release paths do not keep aliases to the reused buffer.

I found two issues to address:

  1. The optimization adds approximately one page of retained capacity per live V2 column writer, which materially increases peak memory for wide buffered row groups.
  2. The modified round-trip test replaces existing DataPageV1 coverage instead of adding DataPageV2 coverage.

Targeted DataPageV2 tests and race tests pass locally. The new benchmark also reproduces the intended allocation reduction. GitHub currently reports no CI checks for this head.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

combined = owned.Bytes()
} else {
w.uncompressedData.Reset()
w.uncompressedData.Grow(combinedSize)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes.Buffer.Reset() retains capacity, so this introduces one page-sized high-water allocation for every live DataPageV2 column writer. That is not visible in the per-operation allocation benchmark.

In a buffered row group with 200 live columns and DataPageSize=1 MiB, post-GC heap usage increased by approximately 200 MiB versus the merge base—one retained MiB per column. This brings V2 in line with the existing V1 behavior, but it is still a material new peak-memory trade-off for wide schemas.

Could we avoid retaining scratch independently in every column writer, such as through pooled/shared eager-page scratch? At minimum, please clear uncompressedData in columnWriter.Close() and document the retained-memory trade-off alongside the allocation reduction.

props = parquet.NewWriterProperties(parquet.WithDictionaryDefault(false), parquet.WithDataPageSize(pageSize))
props = parquet.NewWriterProperties(
parquet.WithDictionaryDefault(false),
parquet.WithDataPageVersion(parquet.DataPageV2),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This converts the existing buffered, multi-page, dictionary-disabled regression test from the default DataPageV1 path to DataPageV2, removing its previous V1 coverage.

Please make the test table-driven over both DataPageV1 and DataPageV2 rather than replacing one with the other. For the V2 case, coverage with non-empty definition/repetition levels or compression would also better exercise concatenation into the reused buffer across page boundaries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants