fix: prevent DISTINCT ORDER BY LIMIT from returning too few rows - #24932
Open
geoffreyclaude wants to merge 1 commit into
Open
fix: prevent DISTINCT ORDER BY LIMIT from returning too few rows#24932geoffreyclaude wants to merge 1 commit into
geoffreyclaude wants to merge 1 commit into
Conversation
geoffreyclaude
force-pushed
the
fix/distinct-topk-limit
branch
from
September 3, 2026 23:48
4224764 to
568975a
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24932 +/- ##
==========================================
- Coverage 81.64% 81.58% -0.06%
==========================================
Files 1123 1123
Lines 410248 410965 +717
Branches 410248 410965 +717
==========================================
+ Hits 334940 335300 +360
- Misses 55617 55916 +299
- Partials 19691 19749 +58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
fix/distinct-topk-limit
branch
from
September 4, 2026 04:42
568975a to
c13259d
Compare
geoffreyclaude
force-pushed
the
fix/distinct-topk-limit
branch
from
September 4, 2026 08:28
c13259d to
e983c3f
Compare
geoffreyclaude
force-pushed
the
fix/distinct-topk-limit
branch
5 times, most recently
from
September 4, 2026 09:51
3471076 to
00baec9
Compare
geoffreyclaude
force-pushed
the
fix/distinct-topk-limit
branch
from
September 4, 2026 09:58
00baec9 to
b30d074
Compare
geoffreyclaude
marked this pull request as ready for review
September 4, 2026 10:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
With multiple partitions,
SELECT DISTINCT ... ORDER BY ... LIMITcan return fewer rows than requested even when enough distinct values exist.DISTINCTruns in partial and final stages. Sort pushdown was moving the final TopKfetch(the sort's row limit) below the final aggregation, while duplicate partial results still existed:Duplicates could fill the limit and discard another value before the final
DISTINCThad a chance to remove them.The corrected plan keeps the limit above the final
DISTINCT:I bisected this regression to
450c861a8c(#14821), first released in 47.0.0. That refactor made the ordering-satisfied fast path forwardfetchto the current plan's children. It correctly established that the current plan preserved the requested ordering, but preserving order does not mean that moving a limit is safe. The shortcut consequently bypassed the existing limit-pushdown and cardinality checks: the parent of that commit returns0,1, and2, while the commit itself returns only0and2.What changes are included in this PR?
When a plan already satisfies the requested ordering, sort pushdown first tries to keep a pending
fetchon that plan through the genericExecutionPlan::with_fetchcapability. It forwards the fetch to children only when the plan explicitly supports limit pushdown and produces exactly one output row for every input row.If an already ordered plan can neither retain nor safely forward the fetch, the optimizer keeps the limit above it with a
LocalLimitExecorGlobalLimitExec, adding a sort-preserving merge when a single output partition is required. This carries the limit without introducing a redundant sort and gives subsequent optimizer runs the same canonical plan shape.What is the testing strategy for this PR?
An SQLLogicTest in
limit.sltuses the exact two-partition query from #24927 and verifies that it returns0,1, and2. Focused physical optimizer tests verify the stable limit-carrier plan across another optimizer pass and verify that fetch is still forwarded through a safe equal-cardinality plan.Are there any user-facing changes?
Yes. Affected queries now return all requested distinct rows. There are no API or configuration changes.