SOLR-18335 : SPLITSHARD fails to migrate documents when using a numeric PointField as router.field - #4717
Conversation
Previously, shard splitting could not utilize numeric point fields as router fields. This change introduces the necessary logic to extract routing values from point fields, either via docValues or stored fields, thus expanding the types of fields available for shard splitting.
509d181 to
f5f7b5c
Compare
|
Hi @dsmiley! Sorry for the direct ping, but since you set yourself as a reviewer, I figured this PR might have just slipped through the cracks. Do you think you might have some time to review it before the next Solr 10.x release? I'm getting a little anxious about my cluster if I can't split these fast-growing shards 😄 Thanks a lot for your time and help! |
dsmiley
left a comment
dsmiley
left a comment
There was a problem hiding this comment.
I'll take it over from here. Getting this into 9.11.
| return Long.toString(value); | ||
| } | ||
|
|
||
| return switch (numberType) { |
There was a problem hiding this comment.
This can be less low-level.. and thus less worry about the specific types. I'll contribute an improvement.
| case LONG -> Long.toString(value); | ||
| case FLOAT -> Float.toString(Float.intBitsToFloat((int) value)); | ||
| case DOUBLE -> Double.toString(Double.longBitsToDouble(value)); | ||
| case DATE -> Long.toString(value); |
There was a problem hiding this comment.
I doubt we truly support dates for router.field.
| } | ||
| } | ||
|
|
||
| if (field.getType().isPointField()) { |
There was a problem hiding this comment.
From what I see, this is too specific. The reason we need to split off to do another algorithm is not specifically that this is a "point field". It's that there is not a terms index; the lower code depends on Terms. So if the router.field isn't "indexed" but it has docValues, then the code you wrote could be improved to handle that. I'll do that.
SOLR-18335: derive numeric route field values via docValues generically, not raw bit decoding getRouteFieldValue hand-decoded NumericDocValues bits with a NumberType switch, plus a stored-field fallback. Rework it to ask the field's FieldType for a ValueSource/FunctionValues instead, so each field type decodes its own docValues rather than SolrIndexSplitter guessing at the encoding. Generalize the branch in split(): the deciding factor isn't "is this a PointField" but "does this field have a term index to iterate" -- fields without one (PointFields, or any indexed=false docValues-only field) now fall back to reading docValues, regardless of type. Drop the stored-field fallback; docValues are now required for a route field with no term index, and a missing-docValues field now fails with a clear SolrException instead of a confusing Lucene IllegalStateException surfaced from the raw (unwrapped) reader. Test coverage reuses the schema's existing parameterized numeric field type (randomized between Trie/Point) rather than adding a dedicated Point-only field type, forcing docValues on when Points are randomly selected so the test is deterministic either way.
github-actions
Bot
added
documentation
2 participants
https://issues.apache.org/jira/browse/SOLR-18335
Description
When executing a SPLITSHARD operation on a collection configured with a custom router.field backed by a numeric PointField (e.g. plong), the operation completes with status: 0 but existing documents are not migrated to the new sub-shards.
Sub-shards are created and set to ACTIVE, but remain empty. The parent shard is marked INACTIVE with all original documents still inside, making them unsearchable.
Collections using the default id field with composite prefix syntax (company_id!doc_id) are not affected.
Solution
A dedicated code path splitPointField() is introduced for PointField route fields. Instead of term enumeration, it reads the routing value per document using:
The routing hash is then computed from this value using HashBasedRouter.sliceHash(), consistent with how documents were originally routed at index time.
An explicit error is thrown if a point-based router.field exposes neither docValues nor stored values, rather than silently producing empty sub-shards.
This implementation and the associated tests were fully generated with AI assistance, and then validated through the external reproducer.
Tests
In addition to the automated Solr tests, I also validated the fix against the minimal external reproducer shared in the bug report:
https://github.com/olivierboudet/solr-splitshard-test/tree/main
With the patch applied,
SPLITSHARDno longer completes with empty sub-shards when the collection uses a custom numericrouter.field. Existing documents are correctly redistributed into the new active sub-shards and remain searchable after the split.Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.