Skip to content

perf: avoid unnecessary large allocations#22558

Open
ariel-miculas wants to merge 4 commits into
apache:mainfrom
ariel-miculas:avoid-split-off
Open

perf: avoid unnecessary large allocations#22558
ariel-miculas wants to merge 4 commits into
apache:mainfrom
ariel-miculas:avoid-split-off

Conversation

@ariel-miculas
Copy link
Copy Markdown
Contributor

@ariel-miculas ariel-miculas commented May 27, 2026

Which issue does this PR close?

Related a bit to #22526

Needs rebasing once #22416 is merged

Rationale for this change

split_off does this:

Returns a newly allocated vector containing the elements in the range [at, len). After the call, the original vector will be left containing the elements [0, at) with its previous capacity unchanged.

which is bad when taking a small slice from a large Vec, for two reasons:

  • it will allocate memory for the remaining elements, which are a lot more than n
  • it will return a Vec with a very large capacity compared to its length

split_vec_min_alloc still has some issues: #22548 but it uses drain + collect when n is small, which is better because it only allocates for the initial n elements and doesn't inflate the capacity

What changes are included in this PR?

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added logical-expr Logical plan and expressions common Related to common crate functions Changes to functions implementation physical-plan Changes to the physical-plan crate labels May 27, 2026
@github-actions github-actions Bot removed common Related to common crate functions Changes to functions implementation labels May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant