Skip to content

OAK-12353: Group Elasticsearch dynamic boost nested documents by boost score - #3079

Open
fabriziofortino wants to merge 9 commits into
apache:trunkfrom
fabriziofortino:issue/OAK-12353
Open

OAK-12353: Group Elasticsearch dynamic boost nested documents by boost score#3079
fabriziofortino wants to merge 9 commits into
apache:trunkfrom
fabriziofortino:issue/OAK-12353

Conversation

@fabriziofortino

Copy link
Copy Markdown
Contributor

Summary

Dynamic boost properties (e.g. predictedTagsDynamicBoost) are indexed in Elasticsearch
as nested documents, with one nested document per value:

"predictedTagsDynamicBoost": [
  { "value": "Replacement Cost", "boost": 1 },
  { "value": "Theft", "boost": 1 },
  { "value": "Alberta", "boost": 1 },
  { "value": "GENERAL INSURANCE COMPANY", "boost": 0.988 }
]

For properties with many values sharing the same boost score, this creates a large
number of nested documents, which is expensive to index and store.

This change groups values that share the same boost score into a single nested
document, with value holding an array instead of a scalar:

"predictedTagsDynamicBoost": [
  { "value": ["Replacement Cost", "Theft", "Alberta"], "boost": 1 },
  { "value": "GENERAL INSURANCE COMPANY", "boost": 0.988 }
]

No mapping or query changes were needed: value is a plain analyzed text field,
which Elasticsearch accepts as an array natively, and the existing nested match /
field_value_factor query in ElasticRequestHandler works unchanged against the
grouped structure.

Feature toggle

Guarded by FT_OAK-12353, enabled by default. Set to false at runtime to revert
to the previous one-nested-document-per-value behavior.

A time-bombed test (ElasticDocumentTest#ft_oak_12353_toggleShouldBeRemoved) will start
failing after 2027-08-12 as a reminder to remove the toggle and its guards once the
grouped format has been running in production long enough.

Changes

  • ElasticDocument: introduces the FT_OAK-12353 toggle; groups dynamic boost values
    by boost score in addDynamicBoostField/getProperties when enabled.
  • ElasticIndexProviderService: registers the toggle on the OSGi Whiteboard.
  • Tests: unit coverage for the grouping logic in ElasticDocumentTest, and end-to-end
    query coverage (grouped and ungrouped) in ElasticDynamicBoostTest.

Test plan

  • ElasticDocumentTest — verifies grouping by boost, single-group unwrapping, and
    toggle-disabled fallback to the original per-value nested documents
  • ElasticDynamicBoostTest — end-to-end query tests against a real Elasticsearch
    instance, confirming queries still match on values grouped into a shared nested
    document, both with grouping enabled (default) and disabled

Dynamic boost properties are mapped as nested fields, with one nested
document per value. When many values share the same boost score, this
generates a lot of nested documents which is expensive in Elasticsearch.

Group values sharing the same boost score into a single nested document
with an array value, behind FT_OAK-12353 (disabled by default). Querying
is unaffected since text fields accept arrays natively.
Cover both the default (grouped) and toggle-disabled (one nested doc
per value) behaviour, verifying queries still match on any value
grouped into a shared nested document.
Similar to the FT_OAK-12206 test in ElasticIndexWriterTest: fails once
the deadline passes, as a reminder to remove FT_OAK-12353 and its
guards once the dynamic-boost grouping default has been in production
long enough.
… from array to object for downstream consumers
Grouping values by boost score makes the nested "value" field's token
count vary with group size, which would otherwise skew BM25 length
normalization and change ranking based on how many tags happen to
share a boost score. Boost is already applied explicitly via
field_value_factor, so length normalization on this field isn't
meaningful; disabling norms keeps matching scores stable regardless of
group size.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant