Skip to content

perf(write): write input changelog and data files in parallel - #350

Merged
lxy-9602 merged 4 commits into
apache:mainfrom
lszskye:changelog_input_mode_double_write
Sep 17, 2026
Merged

lxy-9602 merged 4 commits into
apache:mainfrom
lszskye:changelog_input_mode_double_write

Conversation

@lszskye

@lszskye lszskye commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #174

Improve INPUT changelog write performance by writing data and changelog
files concurrently.

This change:

  • writes changelog files asynchronously while data files are written on
    the caller thread;
  • waits for both write pipelines before publishing their results;
  • aborts both writers when either pipeline fails;
  • adds changelog-producer.input.parallel-write, enabled by default;
  • preserves the previous sequential behavior when the option is disabled;
  • extracts common sort-merge writing logic;
  • reuses a shared helper when creating data and changelog rolling writers.

Tests

  • Extended CoreOptionsTest to verify the default and configured values
    of the new option.
  • Updated MergeTreeWriterTest.TestInputChangelog to support
    nondeterministic file numbering during parallel writes.
  • Added MergeTreeWriterTest.TestInputChangelogWithParallelWriteDisabled
    to verify the sequential fallback and file creation order.

API and Format

This change adds the public option
changelog-producer.input.parallel-write.

It does not change the storage format or protocol.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

@lszskye
lszskye marked this pull request as draft September 15, 2026 10:34
@lszskye
lszskye marked this pull request as ready for review September 16, 2026 06:07
Comment thread include/paimon/defs.h

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 200e485 into apache:main Sep 17, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants