feat(operator): read the whole-number parts out of a timestamp column by kz930 · Pull Request #8480 · apache/texera

kz930 · 2026-09-10T00:37:22Z

What changes were proposed in this PR?

An operator that reads the year, quarter, month, day, weekday, day of year, ISO week, hour, minute or second out of a timestamp column, each added as a whole-number column of its own.

Nothing here is parsed. The column is already a moment when it arrives, and which text became which moment was settled upstream, the way KNIME, Alteryx and Spark all separate parsing from extraction. The parts are read as ISO-8601 states them, because the two runtimes disagree by default: pandas counts Monday as 0, java.time counts it as 1.

Any related issues, documentation, discussions?

Not part of #8325: that set makes the operators Texera already had exportable. This adds a new one, which implements the trait from the start rather than gaining it. The trait is in #8327, which has merged.

Closes #8478, the task this change is the whole of.

Aggregate groups by a column's values, and a timestamp has a different value in almost every row, so grouping by one puts each row in its own group. Sales by month could not be asked for, though the platform carries a TIMESTAMP type, a Gantt chart and a time-series plot.

How was this PR tested?

Twelve unit tests in the operator's own spec, over the schema it declares, the values the executor reads and the Python it emits. The parity harness runs it on 13 configurations, one per field plus a hostile column name and an empty cell, comparing the engine's answer against the exported script's.

The operator in the editor, reading event_time and asking for the year, the month and the weekday:

The three whole-number columns beside the timestamp they came from. 2024-01-15 is a Monday, 2024-07-04 a Thursday and 2025-02-09 a Sunday, reading 1, 4 and 7 as ISO-8601 states them rather than the 0, 3 and 6 pandas would give:

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

github-actions · 2026-09-10T01:39:28Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @aglinxinyuan
You can notify them by mentioning @aglinxinyuan in a comment.

github-actions · 2026-09-10T02:54:14Z

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main e70d67e benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	523	0.319	18,207/27,164/27,164 us	🔴 -15.8% / 🔴 +106.9%
🟢	bs=100 sw=10 sl=64	1,238	0.756	80,735/90,670/90,670 us	🟢 -7.7% / 🟢 -11.4%
⚪	bs=1000 sw=10 sl=64	1,455	0.888	688,791/821,579/821,579 us	⚪ within ±5% / 🟢 -18.1%

Baseline details

Latest main e70d67e from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	523 tuples/sec	621 tuples/sec	931.98 tuples/sec	-15.8%	-43.9%
bs=10 sw=10 sl=64	MB/s	0.319 MB/s	0.379 MB/s	0.569 MB/s	-15.8%	-43.9%
bs=10 sw=10 sl=64	p50	18,207 us	15,843 us	10,818 us	+14.9%	+68.3%
bs=10 sw=10 sl=64	p95	27,164 us	23,819 us	13,127 us	+14.0%	+106.9%
bs=10 sw=10 sl=64	p99	27,164 us	23,819 us	16,504 us	+14.0%	+64.6%
bs=100 sw=10 sl=64	throughput	1,238 tuples/sec	1,250 tuples/sec	1,210 tuples/sec	-1.0%	+2.3%
bs=100 sw=10 sl=64	MB/s	0.756 MB/s	0.763 MB/s	0.739 MB/s	-0.9%	+2.3%
bs=100 sw=10 sl=64	p50	80,735 us	81,205 us	85,326 us	-0.6%	-5.4%
bs=100 sw=10 sl=64	p95	90,670 us	98,188 us	91,092 us	-7.7%	-0.5%
bs=100 sw=10 sl=64	p99	90,670 us	98,188 us	102,379 us	-7.7%	-11.4%
bs=1000 sw=10 sl=64	throughput	1,455 tuples/sec	1,438 tuples/sec	1,240 tuples/sec	+1.2%	+17.4%
bs=1000 sw=10 sl=64	MB/s	0.888 MB/s	0.878 MB/s	0.757 MB/s	+1.1%	+17.4%
bs=1000 sw=10 sl=64	p50	688,791 us	689,465 us	840,622 us	-0.1%	-18.1%
bs=1000 sw=10 sl=64	p95	821,579 us	788,856 us	889,172 us	+4.1%	-7.6%
bs=1000 sw=10 sl=64	p99	821,579 us	788,856 us	931,615 us	+4.1%	-11.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,382.58,200,128000,523,0.319,18206.71,27163.76,27163.76
1,100,10,64,20,1615.17,2000,1280000,1238,0.756,80735.35,90670.36,90670.36
2,1000,10,64,20,13741.78,20000,12800000,1455,0.888,688790.67,821579.10,821579.10

Aggregate groups by a column's values, and a timestamp has a different value in almost every row, so grouping by one puts each row in its own group. Sales by month could not be asked for, though the platform carries a TIMESTAMP type, a Gantt chart and a time-series plot. Nothing here is parsed: the column is already a moment by the time it arrives, and which text became which moment was settled upstream, the way KNIME, Alteryx and Spark all separate parsing from extraction. The parts are read as ISO-8601 states them, since the two runtimes do not agree by default — pandas counts Monday as 0 and java.time counts it as 1. Each part is added under a name derived from the column it came from, so reading two timestamp columns names four distinct results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

codecov-commenter · 2026-09-10T05:21:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.20%. Comparing base (e70d67e) to head (a257026).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #8480   +/-   ##
=========================================
  Coverage     92.20%   92.20%           
  Complexity     4977     4977           
=========================================
  Files          1275     1275           
  Lines         55568    55568           
  Branches       7163     7163           
=========================================
+ Hits          51237    51238    +1     
  Misses         2555     2555           
+ Partials       1776     1775    -1

Flag	Coverage Δ		*Carryforward flag
access-control-service	`77.44% <ø> (ø)`
agent-service	`99.53% <ø> (ø)`		Carriedforward from cf6faeb
amber	`86.77% <ø> (ø)`		Carriedforward from cf6faeb
computing-unit-managing-service	`64.02% <ø> (ø)`
config-service	`87.61% <ø> (ø)`
file-service	`82.29% <ø> (ø)`
frontend	`96.57% <ø> (+<0.01%)`	⬆️
notebook-migration-service	`83.77% <ø> (ø)`
pyamber	`98.54% <ø> (ø)`		Carriedforward from cf6faeb
workflow-compiling-service	`77.95% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kz930 · 2026-09-11T03:03:37Z

@carloea2 this adds an operator that reads the year, month, weekday and the rest out of a timestamp column, each as a whole-number column of its own. Would you take a look when you have a moment?

carloea2

The timestamp fields match between the native and exported paths. Looks good.

carloea2

The standalone path cannot handle timestamps outside pandas nanosecond range. A timestamp in year 2500 works in the native Java path, but pd.to_datetime raises before extracting the year. Please preserve the supported timestamp range or reject it consistently, and add boundary tests.

pandas parses into nanoseconds by default, which reach only 1677 to 2262, so the exported script ended the run on a moment the executor reads like any other: it holds a java.sql.Timestamp, where the year 2500 is ordinary. Microseconds cover the whole range the executor answers for, and the fields are compared against it at and past the nanosecond edge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-18T19:34:32Z

Fixed in c4bb488: the exported script now parses the column at microsecond resolution, which covers the whole range the executor answers for, so the year 2500 no longer ends the run. The boundary test reads the ten fields at 1677-09-22, 2262-04-11, 2500-01-01, 1500-06-15 and 9999-12-31, and compares each one against the executor's own answer for the same moment.

carloea2

The timestamp range issue is fixed. I checked the wider dates and null handling with pandas 2.2.3. The added test also compares the fields against the native executor. Looks good.

…-integration Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

carloea2

Reviewed the latest test tagging change. All 14 focused tests and scalafmtCheckAll pass locally, including native versus exported date fields from year 1500 through 9999. No new findings. Leaving the draft status unchanged.

carloea2

All 14 focused tests and formatting checks pass on a fresh detached checkout. Native and exported date fields agree, including nulls, ISO weekdays and years 1500 through 9999. Current build checks are green. Looks good. Leaving the draft status unchanged.

sbt -batch 'WorkflowOperator/testOnly *ExtractDateTimeOpDescSpec -- -oDF' 'scalafmtCheckAll'

…es (apache#8597) ### What changes were proposed in this PR? Type Casting to a timestamp read text with `pd.to_datetime(col, errors="coerce")` in the exported script. That is wrong in two ways, and both empty a cell the run itself filled. pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a `java.sql.Timestamp`, where `2500-01-01 00:00:00` is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for `1500-06-15 08:30:00` and `9999-12-31 23:59:59`. pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands `DateParserUtils` one field at a time, so a row states its own format. A column holding `2024-03-05 14:09:07` and `March 5, 2024` kept the first and emptied the second. The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. Text no format reads now ends the run, as the engine's `parseField` ends its own. The script raises the engine's message, `Failed to parse type str to Timestamp`, where it used to leave the cell empty. dateutil reads every format the verification hands the engine, so the two refuse the same text. Review and the verification's edge values found more places where the cast parted from the engine: - A reading that states an offset, such as `Z` or `+05:30`, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same. - A column that already held timestamps went through the text parse and lost its nanoseconds. `parseField` returns a `java.sql.Timestamp` untouched, so the script now hands such a column back at the resolution it arrived in. - Text with digits past the millisecond kept them. `DateParserUtils` reads into a `java.util.Date`, which counts milliseconds, so the script now cuts there too. - Casting to text wrote a value the way Python's `str()` does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least `.0`. A shared `_texera_java_text` now writes each one as Java's `toString` does. - A double column holding a NaN as a value, as an Arrow or Parquet source hands one over, lost it. The casts asked each cell whether it was empty, and pandas hands an empty cell over as NaN too, so a stored NaN came out null where the engine writes `NaN` as text, 0 as an integer and `true` as a boolean. The casts now read empty cells from the column's own mask, and the double cast builds a nullable `Float64`, so a NaN read from text stays one through the next cast. A double past an integer's bounds now saturates as `toInt` and `toLong` do, where `int()` raised on an infinity. - A moment was read with dateutil's `tzlocal()`, which knows only today's two offsets, so an instant from before a zone changed its rules came out an hour or more off on every platform. The script now reads the machine's zone by its IANA name and, before 1900, keeps today's standard offset as `java.util.Date` does. - On Windows, `localtime` refuses an instant before 1970, which stopped the cast. The same helper now falls back to the zone's standard offset there. That `Float64` column changed what a Python operator after this cast is handed. pyamber hands a Python operator a DOUBLE column as `float64`, where a NaN and a null are one value, and writes a NaN out as null. Given `Float64` instead, Sort ordered a NaN apart from the nulls, a chart drew points the engine dropped, and a passed-through NaN stayed NaN. `StandaloneCodeGenerator` gains `standaloneInputAdapter()`, and for a Python operator the translator feeds the fragment a `float64` copy of each input, leaving the upstream frame as it was. The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where `new Timestamp(long)` is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it. ### Any related issues, documentation, discussions? Closes apache#8595. apache#8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot. apache#8341, apache#8704, apache#8705 and apache#8709 call the timestamp and text helpers this adds, so this PR merges before them. ### How was this PR tested? Eight tests in `TypeCastingOpDescSpec` run the generated Python and compare it cell by cell against `AttributeTypeUtils.parseField`. They are tagged to run in the integration job, which has pandas. - The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, `2500-01-01`, `1500-06-15`, `9999-12-31`, readings with `Z` and `+05:30`, and readings past the millisecond. It fails on the old generator. - One test gives a column that already holds timestamps and checks its nanoseconds are kept. - The cast-to-string test covers a double at 1e7 and 1e-4. - One test checks that unreadable text ends the script with the engine's message, as the engine refuses it. - Two tests cast a column holding a stored NaN, both infinities and an empty cell to each type and compare against `AttributeTypeUtils`, and check that a NaN read from text survives a second cast. - Two tests read moments from `America/Mexico_City` in 1899, 1921, 1931 and 2021, at the offset the zone had then, and the same moments with `localtime` refusing those before 1970. - Three tests in `WorkflowToPythonTranslatorSpec` check that a Python operator reads a converted copy of its input and an operator the engine runs on the JVM reads its upstream as it is. Two of them fail without the change. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

# Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala

kz930 mentioned this pull request Sep 10, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

20 of 31 tasks

kz930 marked this pull request as draft September 10, 2026 01:31

github-actions Bot added feature dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI common platform Non-amber Scala service paths labels Sep 10, 2026

github-actions Bot assigned kz930 Sep 10, 2026

kz930 force-pushed the feat/extract-datetime-fields branch from d6f6a34 to 535f06e Compare September 10, 2026 05:15

github-actions Bot removed dependencies Pull requests that update a dependency file platform Non-amber Scala service paths labels Sep 10, 2026

Merge remote-tracking branch 'upstream/main' into HEAD

95aadac

carloea2 approved these changes Sep 18, 2026

View reviewed changes

carloea2 suggested changes Sep 18, 2026

View reviewed changes

This was referenced Sep 18, 2026

Casting text to a timestamp empties a moment the engine reads #8595

Closed

feat(operator): read a timestamp cast over the years the engine reaches #8597

Merged

carloea2 approved these changes Sep 18, 2026

View reviewed changes

test(operator): run the cases that execute the exported code in amber…

cde6b0b

…-integration Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

carloea2 reviewed Oct 13, 2026

View reviewed changes

carloea2 approved these changes Oct 14, 2026

View reviewed changes

Merge apache/main into feat/extract-datetime-fields

a257026

# Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala

Navigation Menu

Repository navigation

feat(operator): read the whole-number parts out of a timestamp column - #8480

feat(operator): read the whole-number parts out of a timestamp column#8480
kz930 wants to merge 5 commits into
apache:mainfrom
kz930:feat/extract-datetime-fields

kz930 commented Sep 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 10, 2026 •

edited

Loading

Uh oh!

kz930 commented Sep 11, 2026

Uh oh!

carloea2 left a comment

Uh oh!

carloea2 left a comment

Uh oh!

kz930 commented Sep 18, 2026

Uh oh!

carloea2 left a comment

Uh oh!

carloea2 left a comment

Uh oh!

carloea2 left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kz930 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

github-actions Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

codecov-commenter commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kz930 commented Sep 11, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Sep 18, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kz930 commented Sep 10, 2026 •

edited

Loading

github-actions Bot commented Sep 10, 2026 •

edited

Loading

github-actions Bot commented Sep 10, 2026 •

edited

Loading

codecov-commenter commented Sep 10, 2026 •

edited

Loading