OAK-12353: Group Elasticsearch dynamic boost nested documents by boost score - #3079
Open
fabriziofortino wants to merge 9 commits into
Open
OAK-12353: Group Elasticsearch dynamic boost nested documents by boost score#3079fabriziofortino wants to merge 9 commits into
fabriziofortino wants to merge 9 commits into
Conversation
Dynamic boost properties are mapped as nested fields, with one nested document per value. When many values share the same boost score, this generates a lot of nested documents which is expensive in Elasticsearch. Group values sharing the same boost score into a single nested document with an array value, behind FT_OAK-12353 (disabled by default). Querying is unaffected since text fields accept arrays natively.
Cover both the default (grouped) and toggle-disabled (one nested doc per value) behaviour, verifying queries still match on any value grouped into a shared nested document.
Similar to the FT_OAK-12206 test in ElasticIndexWriterTest: fails once the deadline passes, as a reminder to remove FT_OAK-12353 and its guards once the dynamic-boost grouping default has been in production long enough.
… from array to object for downstream consumers
Grouping values by boost score makes the nested "value" field's token count vary with group size, which would otherwise skew BM25 length normalization and change ranking based on how many tags happen to share a boost score. Boost is already applied explicitly via field_value_factor, so length normalization on this field isn't meaningful; disabling norms keeps matching scores stable regardless of group size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dynamic boost properties (e.g.
predictedTagsDynamicBoost) are indexed in Elasticsearchas nested documents, with one nested document per value:
For properties with many values sharing the same boost score, this creates a large
number of nested documents, which is expensive to index and store.
This change groups values that share the same boost score into a single nested
document, with value holding an array instead of a scalar:
No mapping or query changes were needed: value is a plain analyzed text field,
which Elasticsearch accepts as an array natively, and the existing nested match /
field_value_factor query in ElasticRequestHandler works unchanged against the
grouped structure.
Feature toggle
Guarded by FT_OAK-12353, enabled by default. Set to false at runtime to revert
to the previous one-nested-document-per-value behavior.
A time-bombed test (ElasticDocumentTest#ft_oak_12353_toggleShouldBeRemoved) will start
failing after 2027-08-12 as a reminder to remove the toggle and its guards once the
grouped format has been running in production long enough.
Changes
by boost score in addDynamicBoostField/getProperties when enabled.
query coverage (grouped and ungrouped) in ElasticDynamicBoostTest.
Test plan
toggle-disabled fallback to the original per-value nested documents
instance, confirming queries still match on values grouped into a shared nested
document, both with grouping enabled (default) and disabled