Skip to content

Expose list membership with SQL null semantics in Java and Python - #10053

Open
robert3005 wants to merge 9 commits into
claude/friendly-knuth-dj6ed1-list-containsfrom
rk/list-contains-bindings
Open

robert3005 wants to merge 9 commits into
claude/friendly-knuth-dj6ed1-list-containsfrom
rk/list-contains-bindings

Conversation

@robert3005

@robert3005 robert3005 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Expose the membership expressions from #10052 in Java/JNI and Python, so callers can construct one set-membership expression with either Vortex or SQL null semantics.

Stacked on #10052, which builds on the SQL null semantics in #10057. DuckDB integration follows in #10054.

Changes

  • Add Java listContains, in, and notIn, plus nonempty, empty, and null list literals and their JNI implementations.
  • Add Python list_contains(..., sql_null_semantics=True) and in_list(value, list); negate in_list for SQL NOT IN.
  • Fix Python typed-list construction and infer nullable list element types regardless of null position. Update exports, type stubs, and API documentation.
  • Add Java scan and literal-construction coverage, and Python tests for null semantics, empty/null lists, serialization, scan filters, and large sets.

Validation: cargo clippy -p vortex-jni -p vortex-python --all-targets --all-features -- -D warnings, uvx ty check vortex-python, and Ruff on the changed expression stub passed. Java main and test sources compiled with ./gradlew :vortex-jni:compileJava :vortex-jni:compileTestJava -x buildJniLibrary -x makeTestFiles --console=plain (native compilation checked separately with Cargo). Runtime tests and benchmarks were not executed.

@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 1 regressed benchmark
✅ 2058 untouched benchmarks
🆕 4 new benchmarks
⏩ 525 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime dict_canonicalize_gt_u8_avx512[1000000] 534.5 µs 836.7 µs -36.12%
⚡ Simulation nested_list_random[4] 1,926.8 µs 633.5 µs ×3
⚡ Simulation utf8_random[4] 732.2 µs 278.5 µs ×2.6
⚡ Simulation i64_random_chunked[4] 406.2 µs 178.4 µs ×2.3
⚡ Simulation i64_random[4] 146.5 µs 82.4 µs +77.74%
⚡ Simulation density_sweep_single_slice[0.9] 48.8 µs 32.6 µs +49.75%
⚡ WallTime dict_canonicalize_gt_u8_neon[1000000] 561.6 µs 492 µs +14.15%
⚡ WallTime words_gather_dispatch_neon[65536] 2.3 µs 2 µs +10.56%
🆕 Simulation i64_random_chunked[256] N/A 320.4 µs N/A
🆕 Simulation i64_random[256] N/A 225.8 µs N/A
🆕 Simulation nested_list_random[32] N/A 1 ms N/A
🆕 Simulation utf8_random[256] N/A 508.9 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/list-contains-bindings (2a51612) with claude/friendly-knuth-dj6ed1-list-contains (5bea57b)2

Open in CodSpeed

Footnotes

  1. 525 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on claude/friendly-knuth-dj6ed1-list-contains (159aaff) during the generation of this report, so a7d2c7f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@robert3005
robert3005 removed this pull request from stack #10055 September 25, 2026 14:05
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 2aaa1b6 to 1b63fd8 Compare September 25, 2026 14:06
@robert3005
robert3005 added this pull request to stack #10058 September 25, 2026 14:06
@robert3005
robert3005 removed this pull request from stack #10058 September 25, 2026 14:09
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 1b63fd8 to 3151b45 Compare September 25, 2026 14:09
@robert3005
robert3005 added this pull request to stack #10061 September 25, 2026 14:09
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10053 September 25, 2026 14:21 Inactive
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 3151b45 to b6ffdda Compare September 25, 2026 15:53
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10053 September 25, 2026 16:04 Inactive
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from b6ffdda to ed99a18 Compare September 25, 2026 16:28
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10053 September 25, 2026 16:39 Inactive
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from ed99a18 to 0e3443a Compare September 25, 2026 17:10
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10053 September 25, 2026 17:20 Inactive
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 0e3443a to 2242462 Compare September 25, 2026 17:26
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10053 September 25, 2026 17:37 Inactive
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 2242462 to 828a8f7 Compare September 25, 2026 20:28
@robert3005
robert3005 force-pushed the rk/list-contains-bindings branch from 828a8f7 to 3dd8f74 Compare September 28, 2026 20:43
@robert3005
robert3005 marked this pull request as ready for review September 28, 2026 22:16
Replace comparison/OR chains with prepared membership probes, normalize
constant sets, and evaluate list and needle columns together. Include
canonicalization support and probe regressions.

Build on the separate SQL null-semantics and benchmark changes.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Import the execution-context trait, remove trailing macro commas, and
construct the struct fixture outside the generated fallible test wrapper.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Add Java/JNI list literals, listContains and IN/NOT IN builders. Expose the
SQL null-semantics option and in_list in Python, including list literal
coercion, public exports, typing, documentation and regression coverage.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch was successfully deployed

1 active deployment
docs-preview/pr-10053 — 2a516129 Deployed Sep 29, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant