Mysql string primary key partitioner - #38093
Open
peterdukelarsen wants to merge 7 commits into
Open
Conversation
Discovers boundaries that split a table string primary key space into per-worker ranges of roughly equal estimated row counts. Ranges the optimizer estimates too large are recursively subdivided at each distinct key prefix one character longer, probing through KeyProber, then accumulated into per-worker buckets, so discovery costs EXPLAIN index dives instead of an O(rows) index pass. Inaccurate estimates skew bucket sizes but never correctness: any ordered boundary list partitions the key space. All key ordering happens server-side under the column collation. The walk guards against non-advancing prefixes and caps children per split so a misbehaving server cannot hang it. KeyProber steps past exact keys shorter than the prefix length, so a lone short key among keys extending it cannot leave a range unsplittable. Also documents the caller contracts on like_prefix_pattern and explain_row_estimate.
Drop MAX_DEPTH, MAX_CHILDREN_PER_SPLIT, and the non-advancing prefix guard. On healthy data the walk terminates because child ranges shrink and fresh estimates track them. The pathological cases (phantom estimates, misbehaving servers) will be bounded by the per-table request budget once it lands, rather than by per-mechanism caps. Reformulate bucket sizing as a per-worker share divided by BUCKETS_PER_WORKER, dropping the double-to-8 bucket floor for small worker counts.
partition() becomes a composition of three stages: bucket_target_rows (pure sizing math), split_into_ranges (the only stage touching PartitionDb), and assign_boundaries (pure bucket accumulation). The pure stages are now directly unit-testable and the signatures document the data flow.
Rename the sizing knobs to say what they mean (TARGET_RANGES_PER_WORKER, min_rows_per_worker, target_max_rows_per_range), trim module and function docs to the essentials, and stop deduplicating repeated boundary ends. Duplicate ends only arise from non-advancing servers, and the snapshot layer validates boundary monotonicity server-side before using boundaries.
The trait is the partitioner-owned seam over the concrete KeyProber, a distinct name keeps it from shadowing the prober it wraps.
partition_table_by_pk_prefix becomes partition_table, the trait methods take the probe names they delegate to, and Range names its exclusive lower bound prefix.
peterdukelarsen
force-pushed
the
pl/mysql-pk-partitioner
branch
2 times, most recently
from
August 6, 2026 23:36
8753eb8 to
50b66ec
Compare
peterdukelarsen
marked this pull request as ready for review
August 7, 2026 00:03
peterdukelarsen
force-pushed
the
pl/mysql-pk-partitioner
branch
4 times, most recently
from
August 8, 2026 00:08
c42ed7c to
6cdaec2
Compare
Simplify children_prefixes into a clean prefix walk that accepts skipping exact keys shorter than the probe depth. Split breadth first with a coarse target of 1/max(workers, 8) of the table, restore key order via per-parent ordinal sort keys instead of client-side key comparison, and fold boundary assignment into compute_boundaries. Estimates are u64 end to end and a missing optimizer estimate is now a named MissingRowEstimate error raised inside estimate_range_rows.
peterdukelarsen
force-pushed
the
pl/mysql-pk-partitioner
branch
from
August 8, 2026 01:44
6cdaec2 to
fe76d04
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part of SS-97.
Working on a faster way to identify partition boundaries for string primary keys.
Description
Adds code for partitioning the primary keys of a table with a single-column string primary key.
The core idea for partitioning without needing to understand MySQL's string sort order rules is to walk through string prefixes, in order, estimating the sizes and refining as you go based on table size estimates.
We're doing a BFS here so in the future if we want to exit early (which is planned for a subsequent PR) we have a better shot of useful partitioning.
Recreated from #38047 with an in-repo head branch so this PR can join the GitHub stack.