Skip to content

feat: centralizing higher-order list lambda evaluation helpers (DRAFT)#23911

Draft
saadtajwar wants to merge 5 commits into
apache:mainfrom
saadtajwar:saadtajwar/abstract-hof-list-lambda
Draft

feat: centralizing higher-order list lambda evaluation helpers (DRAFT)#23911
saadtajwar wants to merge 5 commits into
apache:mainfrom
saadtajwar:saadtajwar/abstract-hof-list-lambda

Conversation

@saadtajwar

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

array_filter, array_any_match, and array_first each implement their own version of the same higher-order list execution pattern:

  • normalize a list-like argument to List / LargeList,
  • extract the flattened child values with slice-aware semantics,
  • evaluate a lambda once over the flattened values,
  • spread captured outer columns to flattened row cardinality with list_values_row_number,
  • map the flattened lambda result back to one output value per input row using adjusted offsets and null-row handling.

This PR introduces shared helpers that allow for one area where the shared logic lives

What changes are included in this PR?

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jul 26, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.61905% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.68%. Comparing base (551c592) to head (29ffb87).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-nested/src/lambda_utils.rs 84.83% 12 Missing and 15 partials ⚠️
datafusion/functions-nested/src/array_any_match.rs 90.90% 2 Missing and 2 partials ⚠️
datafusion/functions-nested/src/array_filter.rs 92.45% 2 Missing and 2 partials ⚠️
datafusion/functions-nested/src/array_first.rs 90.00% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23911      +/-   ##
==========================================
+ Coverage   80.65%   80.68%   +0.03%     
==========================================
  Files        1093     1094       +1     
  Lines      371629   371991     +362     
  Branches   371629   371991     +362     
==========================================
+ Hits       299725   300129     +404     
+ Misses      54012    53961      -51     
- Partials    17892    17901       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Centralize higher-order list lambda evaluation helpers

2 participants