[kz930]

What changes were proposed in this PR?

Type Casting to a timestamp read text with pd.to_datetime(col, errors="coerce") in the exported script. That is wrong in two ways, and both empty a cell the run itself filled.

pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a java.sql.Timestamp, where 2500-01-01 00:00:00 is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for 1500-06-15 08:30:00 and 9999-12-31 23:59:59.

pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands DateParserUtils one field at a time, so a row states its own format. A column holding 2024-03-05 14:09:07 and March 5, 2024 kept the first and emptied the second.

The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. Text no format reads now ends the run, as the engine's parseField ends its own. The script raises the engine's message, Failed to parse type str to Timestamp, where it used to leave the cell empty. dateutil reads every format the verification hands the engine, so the two refuse the same text.

Review and the verification's edge values found more places where the cast parted from the engine:

• A reading that states an offset, such as Z or +05:30, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same.
• A column that already held timestamps went through the text parse and lost its nanoseconds. parseField returns a java.sql.Timestamp untouched, so the script now hands such a column back at the resolution it arrived in.
• Text with digits past the millisecond kept them. DateParserUtils reads into a java.util.Date, which counts milliseconds, so the script now cuts there too.
• Casting to text wrote a value the way Python's str() does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least .0. A shared _texera_java_text now writes each one as Java's toString does.
• A double column holding a NaN as a value, as an Arrow or Parquet source hands one over, lost it. The casts asked each cell whether it was empty, and pandas hands an empty cell over as NaN too, so a stored NaN came out null where the engine writes NaN as text, 0 as an integer and true as a boolean. The casts now read empty cells from the column's own mask, and the double cast builds a nullable Float64, so a NaN read from text stays one through the next cast. A double past an integer's bounds now saturates as toInt and toLong do, where int() raised on an infinity.
• A moment was read with dateutil's tzlocal(), which knows only today's two offsets, so an instant from before a zone changed its rules came out an hour or more off on every platform. The script now reads the machine's zone by its IANA name and, before 1900, keeps today's standard offset as java.util.Date does.
• On Windows, localtime refuses an instant before 1970, which stopped the cast. The same helper now falls back to the zone's standard offset there.

That Float64 column changed what a Python operator after this cast is handed. pyamber hands a Python operator a DOUBLE column as float64, where a NaN and a null are one value, and writes a NaN out as null. Given Float64 instead, Sort ordered a NaN apart from the nulls, a chart drew points the engine dropped, and a passed-through NaN stayed NaN. StandaloneCodeGenerator gains standaloneInputAdapter(), and for a Python operator the translator feeds the fragment a float64 copy of each input, leaving the upstream frame as it was.

The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where new Timestamp(long) is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it.

Any related issues, documentation, discussions?

Closes #8595. #8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot.

#8341, #8704, #8705 and #8709 call the timestamp and text helpers this adds, so this PR merges before them.

How was this PR tested?

Eight tests in TypeCastingOpDescSpec run the generated Python and compare it cell by cell against AttributeTypeUtils.parseField. They are tagged to run in the integration job, which has pandas.

• The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, 2500-01-01, 1500-06-15, 9999-12-31, readings with Z and +05:30, and readings past the millisecond. It fails on the old generator.
• One test gives a column that already holds timestamps and checks its nanoseconds are kept.
• The cast-to-string test covers a double at 1e7 and 1e-4.
• One test checks that unreadable text ends the script with the engine's message, as the engine refuses it.
• Two tests cast a column holding a stored NaN, both infinities and an empty cell to each type and compare against AttributeTypeUtils, and check that a NaN read from text survives a second cast.
• Two tests read moments from America/Mexico_City in 1899, 1921, 1931 and 2021, at the offset the zone had then, and the same moments with localtime refusing those before 1970.
• Three tests in WorkflowToPythonTranslatorSpec check that a Python operator reads a converted copy of its input and an operator the engine runs on the JVM reads its upstream as it is. Two of them fail without the change.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 7 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 7b8422e benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	740	0.451	12,965/22,176/22,176 us	🔴 +19.7% / 🔴 +57.4%
🟢	bs=100 sw=10 sl=64	1,805	1.101	54,761/61,632/61,632 us	🟢 -26.1% / 🟢 +53.8%
🔴	bs=1000 sw=10 sl=64	2,104	1.284	463,525/629,266/629,266 us	🔴 +22.4% / 🟢 +74.7%

Baseline details

Latest main 7b8422e from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	740 tuples/sec	820 tuples/sec	888.52 tuples/sec	-9.8%	-16.7%
bs=10 sw=10 sl=64	MB/s	0.451 MB/s	0.5 MB/s	0.542 MB/s	-9.8%	-16.8%
bs=10 sw=10 sl=64	p50	12,965 us	10,828 us	11,467 us	+19.7%	+13.1%
bs=10 sw=10 sl=64	p95	22,176 us	19,017 us	14,087 us	+16.6%	+57.4%
bs=10 sw=10 sl=64	p99	22,176 us	19,017 us	17,519 us	+16.6%	+26.6%
bs=100 sw=10 sl=64	throughput	1,805 tuples/sec	1,754 tuples/sec	1,174 tuples/sec	+2.9%	+53.8%
bs=100 sw=10 sl=64	MB/s	1.101 MB/s	1.07 MB/s	0.716 MB/s	+2.9%	+53.7%
bs=100 sw=10 sl=64	p50	54,761 us	53,259 us	89,952 us	+2.8%	-39.1%
bs=100 sw=10 sl=64	p95	61,632 us	83,425 us	96,426 us	-26.1%	-36.1%
bs=100 sw=10 sl=64	p99	61,632 us	83,425 us	110,188 us	-26.1%	-44.1%
bs=1000 sw=10 sl=64	throughput	2,104 tuples/sec	2,128 tuples/sec	1,204 tuples/sec	-1.1%	+74.7%
bs=1000 sw=10 sl=64	MB/s	1.284 MB/s	1.299 MB/s	0.735 MB/s	-1.2%	+74.7%
bs=1000 sw=10 sl=64	p50	463,525 us	468,494 us	886,711 us	-1.1%	-47.7%
bs=1000 sw=10 sl=64	p95	629,266 us	513,948 us	928,644 us	+22.4%	-32.2%
bs=1000 sw=10 sl=64	p99	629,266 us	513,948 us	972,356 us	+22.4%	-35.3%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,270.40,200,128000,740,0.451,12964.52,22175.70,22175.70
1,100,10,64,20,1108.24,2000,1280000,1805,1.101,54761.23,61631.97,61631.97
2,1000,10,64,20,9507.40,20000,12800000,2104,1.284,463524.82,629266.39,629266.39

[codecov-commenter]

Codecov Report

❌ Patch coverage is 63.33333% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.18%. Comparing base (7b8422e) to head (1ed5465).

Files with missing lines	Patch %	Lines
.../amber/translator/WorkflowToPythonTranslator.scala	72.72%	0 Missing and 6 partials ⚠️
...exera/amber/operator/StandaloneCodeGenerator.scala	42.85%	4 Missing ⚠️
...ache/texera/amber/operator/StandaloneHelpers.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #8597   +/-   ##
=========================================
  Coverage     92.18%   92.18%           
- Complexity     5051     5056    +5     
=========================================
  Files          1286     1286           
  Lines         55937    55963   +26     
  Branches       7140     7146    +6     
=========================================
+ Hits          51563    51589   +26     
+ Misses         2612     2609    -3     
- Partials       1762     1765    +3     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (ø)
agent-service	99.16% <ø> (ø)		Carriedforward from 7b8422e
amber	86.91% <37.50%> (-0.03%)	⬇️
computing-unit-managing-service	64.02% <ø> (ø)
config-service	87.61% <ø> (ø)
file-service	82.29% <ø> (ø)
frontend	96.48% <ø> (ø)		Carriedforward from 7b8422e
notebook-migration-service	83.77% <ø> (ø)
pyamber	98.54% <ø> (+0.06%)	⬆️	Carriedforward from 7b8422e
workflow-compiling-service	77.95% <72.72%> (+3.70%)	⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[carloea2]

The new timestamp helper crashes on 2024-03-05T14:09:07Z. I ran the generated helper with pandas 2.2.3. Parsing keeps the timezone, then astype rejects the conversion to a timestamp without a timezone. The previous code accepts this value. Please handle timestamps with a timezone and add a test with Z and an explicit offset.

[kz930]

Fixed in c07493387. The helper now does what the engine does with an offset — DateParserUtils reads it and java.sql.Timestamp keeps no zone, so the moment is held as the wall clock of the machine's own zone (...T14:09:07Z is 06:09:07 eight hours behind UTC) — and both spellings joined the cases the test compares against parseField cell by cell, so the expectation comes from the engine rather than being written down.

[carloea2]

The timezone crash is fixed. I ran the current helper with Z, an explicit offset, year 2500, and null. Those checks passed.

[carloea2]

Reproduced locally with the generated Python code. One correctness issue below. I did not run a frontend workflow or the full Scala suite.

[carloea2]

Retested ca67ce1. Timezone inputs, extended years and nanosecond precision pass. The previous findings are fixed. Looks good.

[kz930]

@carloea2 Since your approval, text read into a timestamp is cut to the millisecond as DateParserUtils does, and a cast to text writes a value as Java's toString does.

[carloea2]

Retested with JDK 17.0.20.1. The helper still writes 1.0E23 where the engine writes 9.999999999999999E22. The JDK 19 form would keep this mismatch. We need maintainer agreement on the supported formatting contract before accepting that difference.

[kz930]

@carloea2 Per Xinyuan's call on the thread, the helper keeps the shortest form and the JDK 17 difference is noted in a comment on _texera_java_text (2b1dcf5). Could you take another look?

[kz930]

@carloea2 Since your approval I pushed two commits. A timestamp cast now raises on text no format reads, as the engine's parseField does, and the spec checks that. Could you take another look?

[carloea2]

Retested the timestamp refusal change on fresh upstream main. All 13 TypeCastingOpDescSpec tests and 13 WorkflowToPythonTranslatorSpec tests pass, including invalid text and the input adapter. scalafmtCheckAll passes. Code looks good. The current CI failures occur before tests because the Ampel action is not allowed.

[kz930]

@carloea2 Since your approval, the script carries only the helpers it calls, one blank line apart, and a holed LONG column cast to text no longer rounds values past 2^53 (test in TypeCastingOpDescSpec). Plain ISO timestamps are also read in bulk and a LONG cast clamps inside the cast itself, with the same answers. Could you take another look?

[carloea2]

The generated timestamp helper still differs from the native parser for local times inside daylight-saving gaps. Direct comparisons against DateParserUtils 1.0.11 on JDK 17 give two failures and three passing controls. This is a new timestamp finding, not the accepted floating-point formatting difference.

[carloea2]

Approving as requested, with the separate timestamp-gap discrepancy still documented in the inline review. This approval does not mark that finding fixed. The JDK 17 floating-point formatting difference was accepted by Xinyuan and is not a blocker.

[aglinxinyuan]

Please resolve the conflict.

[aglinxinyuan]

LGTM!