[kz930]

What changes were proposed in this PR?

Filter's exported Python compared the column to the value with pandas operators. The engine's FilterPredicate.evaluate compares by the column's type instead, so the two disagreed on numbers stored as text, booleans, NaN and -0.0.

Each comparison in the export now goes through _texera_filter_keep, a transcription of FilterPredicate.evaluate:

Column type	Comparison
STRING	as Java doubles when both sides parse as one, otherwise by UTF-16 text
BOOLEAN	the lower-case text "true"/"false"
INTEGER, DOUBLE	by Double.compareTo: NaN equals NaN and sorts above every number, -0.0 sorts below 0.0; a hex value past the largest double is infinity, as in Java
LONG	as integers, rejecting what Long.valueOf rejects
TIMESTAMP	by whole milliseconds; a value with an offset is read as the wall clock it shows in the machine's zone, with the offset that zone had at that instant

The column type comes from the input schema. Without one, it is inferred from the pandas dtype. is null and is not null are unchanged.

One approximation remains. The engine reads a TIMESTAMP value with DateParserUtils, which accepts more formats than pandas. A value pandas cannot read raises an error, so the export never gives a silently wrong answer.

A value with an offset is placed in the machine's zone by _texera_local_wall_clock from #8597. It uses the zone's IANA history, as the JVM does, and not Windows' localtime. Before 1900 it takes the zone's standard offset, as the engine does.

Any related issues, documentation, discussions?

Closes #8977

Depends on #8597, which adds _texera_local_wall_clock, and on #9035, which reads a column named in a different case.

How was this PR tested?

A new test in SpecializedFilterOpDescSpec runs 33 predicates across the six column types through SpecializedFilterOpExec and through the export, and compares the kept rows by input position. A second test does the same for timestamp values with an offset, with both the JVM and Python set to Asia/Kolkata. Both fail on main and pass with this change. A third test sets both to America/Mexico_City and compares 2021-07-01T00:00:00Z (summer time) and 1931-01-01T00:00:00Z (an older offset). With #8597 merged in, the Filter specs pass (42 tests). A fourth test compares a LONG column Key through a predicate on KEY, past 2^53, and passes with #8597 and #9035 merged in.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.06%. Comparing base (05c19a4) to head (fdcc225).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8980      +/-   ##
============================================
- Coverage     92.17%   92.06%   -0.12%     
+ Complexity     5043     4993      -50     
============================================
  Files          1287     1281       -6     
  Lines         55799    55395     -404     
  Branches       7144     7069      -75     
============================================
- Hits          51434    50998     -436     
- Misses         2598     2657      +59     
+ Partials       1767     1740      -27     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (ø)
agent-service	99.54% <ø> (ø)		Carriedforward from c57e6b0
amber	86.53% <ø> (-0.38%)	⬇️	Carriedforward from c57e6b0
computing-unit-managing-service	64.02% <ø> (ø)
config-service	87.61% <ø> (ø)
file-service	82.29% <ø> (ø)
frontend	96.53% <ø> (+0.05%)	⬆️	Carriedforward from c57e6b0
notebook-migration-service	83.77% <ø> (ø)
pyamber	98.54% <ø> (+0.06%)	⬆️	Carriedforward from c57e6b0
workflow-compiling-service	77.95% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 3 better · 🔴 2 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main 05c19a4 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🟢	bs=10 sw=10 sl=64	438	0.267	22,105/32,293/32,293 us	🟢 -5.0% / 🔴 +127.4%
🟢	bs=100 sw=10 sl=64	972	0.593	103,033/123,575/123,575 us	🟢 -23.8% / 🔴 +28.1%
🔴	bs=1000 sw=10 sl=64	1,098	0.67	907,404/1,060,736/1,060,736 us	🔴 +15.1% / 🔴 +13.7%

Baseline details

Latest main 05c19a4 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	438 tuples/sec	423 tuples/sec	880.32 tuples/sec	+3.5%	-50.2%
bs=10 sw=10 sl=64	MB/s	0.267 MB/s	0.258 MB/s	0.537 MB/s	+3.5%	-50.3%
bs=10 sw=10 sl=64	p50	22,105 us	23,275 us	11,404 us	-5.0%	+93.8%
bs=10 sw=10 sl=64	p95	32,293 us	33,833 us	14,198 us	-4.5%	+127.4%
bs=10 sw=10 sl=64	p99	32,293 us	33,833 us	17,818 us	-4.5%	+81.2%
bs=100 sw=10 sl=64	throughput	972 tuples/sec	927 tuples/sec	1,146 tuples/sec	+4.9%	-15.2%
bs=100 sw=10 sl=64	MB/s	0.593 MB/s	0.566 MB/s	0.7 MB/s	+4.8%	-15.3%
bs=100 sw=10 sl=64	p50	103,033 us	100,847 us	90,069 us	+2.2%	+14.4%
bs=100 sw=10 sl=64	p95	123,575 us	162,226 us	96,453 us	-23.8%	+28.1%
bs=100 sw=10 sl=64	p99	123,575 us	162,226 us	109,096 us	-23.8%	+13.3%
bs=1000 sw=10 sl=64	throughput	1,098 tuples/sec	1,137 tuples/sec	1,174 tuples/sec	-3.4%	-6.5%
bs=1000 sw=10 sl=64	MB/s	0.67 MB/s	0.694 MB/s	0.717 MB/s	-3.5%	-6.5%
bs=1000 sw=10 sl=64	p50	907,404 us	880,347 us	886,710 us	+3.1%	+2.3%
bs=1000 sw=10 sl=64	p95	1,060,736 us	921,823 us	933,314 us	+15.1%	+13.7%
bs=1000 sw=10 sl=64	p99	1,060,736 us	921,823 us	979,175 us	+15.1%	+8.3%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,456.45,200,128000,438,0.267,22105.22,32293.31,32293.31
1,100,10,64,20,2057.98,2000,1280000,972,0.593,103032.83,123575.29,123575.29
2,1000,10,64,20,18207.88,20000,12800000,1098,0.670,907404.30,1060736.49,1060736.49

[kz930]

@carloea2 Could you review this? Filter's export now compares each predicate by the column's type, as FilterPredicate.evaluate does, and a new spec test checks 26 predicates against the engine.

[carloea2]

The 40 existing Filter tests and scalafmtCheckAll pass against current upstream main on JDK 17. Two additional native versus export parity cases fail below. Finite hexadecimal, unzoned timestamp and null controls pass.

Commands: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec" scalafmtCheckAll; sbt -batch "WorkflowOperator/testOnly *FilterBoundaryReviewSpec"

[kz930]

@carloea2 Thanks, both findings are fixed in 55ce604 with regressions against the native executor. Could you take another look?

[carloea2]

Both original regressions now pass. A remaining Windows timezone problem is reproduced below. Of 47 focused tests, 44 pass and 3 fail: the two historical cases and the new Asia/Kolkata test, whose TZ setting is not honored as an IANA zone by Windows localtime. scalafmtCheckAll passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec" scalafmtCheckAll

[carloea2]

The historical timezone finding is fixed. With #8597 at f0eedd9 merged locally into current upstream main, all 61 focused tests and scalafmtCheckAll pass on Windows. That includes my original hexadecimal, offset and historical regressions. The PR head alone lacks _texera_local_zone, so the CI dependency is real. Leaving the draft unchanged until #8597 lands and CI passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec *TypeCastingOpDescSpec" scalafmtCheckAll

[carloea2]

Tested together with #8597 and #9035 after resolving only the import conflicts. Nine existing tests pass. The #9035 generated-code assertion needs updating for the new helper, and an added native-versus-export regression finds a LONG comparison mismatch.

sbt -batch 'WorkflowOperator/testOnly *SpecializedFilterOpDescSpec -- -oDF'

[carloea2]

The LONG Key and KEY regression is fixed. All 12 SpecializedFilterOpDescSpec tests pass with the #9035 lookup API and timestamp helpers supplied. The native filter and export now keep only the correct row beyond 2^53. scalafmtCheckAll passes. The combined #9035 generated-code assertion was updated mechanically for the new helper. CI is still red, so the dependencies and checks need clearing before merge.

[kz930]

@carloea2 Since your review, Filter orders doubles with the shared _texera_double_key and its helpers are spaced like the rest of the export. Could you take another look?

[carloea2]

Rechecked the updated predicates. All 44 native and export cases pass, and CI is green.

[aglinxinyuan]

LGTM!