Skip to content

feat(predicate): add full-text match and score projection semantics - #451

Merged
lxy-9602 merged 3 commits into
apache:mainfrom
lszskye:file_index_predicate_tree
Oct 23, 2026
Merged

lxy-9602 merged 3 commits into
apache:mainfrom
lszskye:file_index_predicate_tree

Conversation

@lszskye

@lszskye lszskye commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #452
Existing search setters cannot represent full-text conditions inside mixed
AND/OR filters, such as body matches "apple" OR price < 10.

This PR introduces the predicate semantics and interfaces for these queries:

  • Add FullTextMatch as a leaf in the existing predicate tree, using the
    same JSON DSL as FullTextSearch without TopK truncation.
  • Preserve full-text conditions during projection, field reordering, and
    schema renaming without modifying the original shared request.
  • Retain the root TopK setters and validate positive limits and nonempty
    search inputs.
  • Resolve explicit _<field>_INDEX_SCORE projections and bind each output
    position to a search type and original table field name.
  • Reject missing or ambiguous score sources and score-producing full-text
    conditions beneath OR. Root TopK requires internal scores regardless
    of projection.

This PR covers construction, validation, and binding. Full-text predicate
execution, scan pruning with full-text predicates, and per-field score
materialization return NotImplemented. Mixed-query execution, cross-file
TopK merging, and selectivity-based scheduling will follow separately.

Tests

Added coverage for:

  • Predicate construction, equality, AND/OR composition, and unsupported
    scalar/statistics evaluation.
  • Projection validation and field rename/reorder mapping.
  • Score bindings, projection precedence, duplicate sources, and OR restrictions.
  • Root TopK score requirements and search-type isolation.
  • Explicit rejection of unsupported scan and read execution.
  • Cross-module casts from the new predicate to LeafPredicate.

API and Format

Adds:

  • FullTextMatch
  • PredicateBuilder::FullTextMatch
  • PredicateUtils::ContainsFullTextMatch
  • Function::Type::FULL_TEXT_MATCH
  • FileIndexReader::VisitFullTextMatch

Documentation

Document predicate composition, root TopK semantics, score projection rules,
and current execution limitations in docs/source/api/file_index.rst.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@lszskye
lszskye force-pushed the file_index_predicate_tree branch from 29e364d to 7491b9a Compare October 21, 2026 09:27
Comment thread include/paimon/predicate/full_text_match.h
Comment thread include/paimon/read_context.h Outdated
Comment thread src/paimon/core/utils/field_mapping.cpp Outdated
Comment thread src/paimon/core/operation/internal_read_context.h
Comment thread docs/source/api/file_index.rst Outdated

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit d21122e into apache:main Oct 23, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants