[rustyconover]

kll_sketch's constructor allocates storage for k items straight away. For the default k = 200 with double items, that's 1.6 KB per sketch before anything has been added.

That's costly when an application creates many sketches that stay empty or small, for example one per group in a grouped aggregation. Query engines also create short-lived partial states that are only ever merged into, or that see a single row each before being combined.

This PR allocates items_ on the first update() or non-empty merge() instead:

• allocate_items() centralizes allocation for updates and merges, including merges where the source has items only in higher levels.
• copy_items() is shared by the copy and converting constructors and leaves item storage unallocated when the source has none.
• item_ptr(index) centralizes access to item ranges. It returns nullptr when item storage is unallocated; sort_level_zero(), check_sorting() and get_sorted_view() use it to handle empty ranges without pointer arithmetic on null storage.
• The destructor and const_iterator already handle items_ == nullptr.
• Empty sorted views do not allocate item storage.

Results and the serialized format are unchanged.

Tests

Regression coverage uses test_allocator to check:

• Empty sketches, their copies, and empty merges allocate only the levels; the first update allocates item storage, and destruction releases it.
• Copy and move merges into sketches without item storage preserve counts and extrema through estimation mode.
• Empty sorted views are safe and allocate nothing after construction, copying, type conversion, moving, assignment over an existing cached view, and both serialization round trips. First updates after those empty views still allocate and sort correctly.
• Numeric sketches and sketches receiving only NaN updates also return empty sorted views safely.

The custom-type empty-view tests expose null-pointer arithmetic under UBSan if sort_level_zero() uses raw items_ + levels_[0] instead of item_ptr().

Validation:

• All 21 GitHub checks pass for commit 1fcf432f, including the compiler matrix, sanitizers, libc++ hardening, coverage and license audit.
• All 17 CTest suites pass on Linux aarch64 with GCC 14.
• KLL, integration, and deserialization-hardening suites pass with GCC AddressSanitizer, UndefinedBehaviorSanitizer, and libstdc++ assertions enabled.
• The full KLL suite passes on macOS with Apple Clang 17, libc++, and UBSan.

Context

We found this in the DuckDB DataSketches extension. For GROUP BY queries with many groups, DuckDB creates a separate aggregate state for each input row and merges them later, and the eager allocation made a KLL aggregate over 2M groups use tens of GB. DuckDB made the same change to its own t-digest for the same reason. The corresponding t-digest change is in #561. REQ also allocates its first compactor eagerly and is covered by the project-wide allocation proposal.

🤖 Generated with Claude Code

[proost]

Idea is good, But null checking isn't enough.

For example, sort_level_zero methods should check null for items_.

And we should guard all null cases in one place not each call sites.

[rustyconover]

@proost thanks for the review, agreed on both points. I've pushed 1fcf432, which moves all handling of an unallocated items_ into the helpers that own the item storage, so callers no longer guard anything:

• allocate_items() is the only place items_ gets allocated (update and merge).
• copy_items() is the only place items are copied. The copy and converting constructors both use it, which replaces their duplicated null checks.
• item_ptr(index) is the only place an item index becomes a pointer for readers. Without item storage it returns nullptr, so every level range is empty. sort_level_zero(), check_sorting() and get_sorted_view() go through it and no longer need guards of their own.

That also lets me drop the is_empty() early return I had added to get_sorted_view(), so it's back to its original form. Serialization and the quantile and rank queries already check is_empty() before touching items. The iterator already treats a null items_ as begin() == end().

To confirm the guard is load-bearing: if I put the raw items_ + levels_[0] back in sort_level_zero(), the UBSan build fails the new empty-sketch tests with applying non-zero offset 800 to null pointer. With item_ptr(), kll_test passes in Release, UBSan (undefined,float-cast-overflow) and ASan builds.

The CI workflows on this PR are still waiting for maintainer approval, so they haven't run yet.

[leerho]

Thanks, Rusty. The DuckDB numbers make a strong case, and the reworked storage helpers from proost's review are a clean way to handle an unallocated items_.

Because the same choice affects several sketches (KLL, t-digest and REQ here, and the same sketches in Java), we'd like to agree on one convention across the project before merging any of them. I've posted a proposal on dev@ as a lazy consensus ending Nov 7, 01:00 UTC: https://lists.apache.org/thread/rv730qo61qqkd99dcy4wnglf3hz1s0fz

In short, it proposes allocating item storage on first use by default, with an explicit reserve() for users who want it up front, and keeping the basic exception guarantee on the first update()/merge() when allocation fails.

I'm marking this PR as a draft until that resolves. If it passes, we'd like to build on this PR, possibly with a reserve() added. Comments on the proposal are very welcome, on the list or here.

[stojkomilos]

@rustyconover Nice work. Could you please add geometric increasing of size for the first level/layer of the sketch, just like your other PR #561, and just like theta and tuple do this also already on master.

If you don't want to do that in this PR, that is reasonable. I will do it after you merge this one (I already have a drafted PR for this, here #569)

Perhaps @leerho can give his 2 cents.

[rustyconover]

@leerho following up now that the proposal's November 7, 01:00 UTC deadline has passed. I see a supportive reply and no objections in the archived thread. Could you confirm whether the proposal is accepted and when we can take #558 and #561 out of draft? You offered to make the C++ and Java changes starting from these PRs; are you planning to add reserve(), the allocation-failure tests and the associated documentation, or would you prefer that I update these branches?

@stojkomilos thanks for offering to handle geometric growth and putting together #569. I'd like to keep this PR focused on allocation on first use and leave geometric growth to your follow-up in #569 after this merges. Please go ahead with polishing that PR on that basis, subject to the project's allocation convention.

@proost could you take another look at 1fcf432f? It centralizes allocation, copying and access to item ranges in allocate_items(), copy_items() and item_ptr(), including the empty sort_level_zero() path you flagged. All 21 checks now pass, including ASan, UBSan and libc++ hardening. I've also updated the PR description to match the current implementation. Thanks!