feat(workflow-operator): export the line sources and the URL fetcher as Python by kz930 · Pull Request #8705 · apache/texera

kz930 · 2026-09-27T03:23:23Z

What changes were proposed in this PR?

File Scan, the file-scan source, Text Input and URL Fetcher implement StandaloneCodeGenerator. Each slices the raw lines before parsing, as the engine does, so a line outside the window is never converted. A boolean line is read as parseField reads one, where 1 is true and other text ends the run. A timestamp line is kept as text and parsed as a column with the helper #8597 adds, as the CSV readers parse theirs. Every file is read as bytes and decoded the way the engine's reader decodes it. Bytes the charset cannot hold become U+FFFD, and UTF-16 with no byte-order mark reads as big-endian. Python's codecs refuse both, so a file read under the wrong Encoding used to end the script while the engine read on.

URL Fetcher is reported as unverifiable rather than exported blind. It reads a live URL, so two runs are not required to agree and a comparison would only measure the network. File Scan takes its filenames from an input port, so the verification feeds it a table that names the files to read, as it feeds a transform. That lands with #8364 and #8363.

Any related issues, documentation, discussions?

Part of #8325, step 12c; that issue lists the set in order. It needs #8341 for the shared source code and #8597 for the timestamp helpers, so both merge first.

Closes #8703, the task this change is the whole of.

One engine defect was found while matching these readers, and it is fixed here rather than worked around: the export alone could not close it. #8598, where a file scan extracting an archive could not also include the filename, because the line-by-line branch emitted a one-field row against the two-column schema its own flag declares. A second, #8596, where the Encoding field never reached the executor, was fixed upstream by #8605, and this branch takes that fix.

Closes #8598.

How was this PR tested?

Each operator asserts the block it emits in its own spec. Where a difference is one of behaviour rather than of text, the spec runs the engine's own executor and the generated script over the same input and compares what each produced: the boolean parsing rules and the entries read out of an archive. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-27T03:23:47Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @Ma77Ball, @aglinxinyuan, @suyashj1231
You can notify them by mentioning @Ma77Ball, @aglinxinyuan, @suyashj1231 in a comment.

github-actions · 2026-09-27T03:26:53Z

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 7b8422e benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	384	0.235	25,235/34,034/34,034 us	🟢 -28.8% / 🔴 +159.3%
🔴	bs=100 sw=10 sl=64	844	0.515	115,695/144,777/144,777 us	🔴 +8.2% / 🔴 +58.9%
🔴	bs=1000 sw=10 sl=64	970	0.592	1,029,900/1,186,559/1,186,559 us	🔴 +5.0% / 🔴 +33.4%

Baseline details

Latest main 7b8422e from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	384 tuples/sec	387 tuples/sec	931.98 tuples/sec	-0.8%	-58.8%
bs=10 sw=10 sl=64	MB/s	0.235 MB/s	0.236 MB/s	0.569 MB/s	-0.4%	-58.7%
bs=10 sw=10 sl=64	p50	25,235 us	22,517 us	10,818 us	+12.1%	+133.3%
bs=10 sw=10 sl=64	p95	34,034 us	47,768 us	13,127 us	-28.8%	+159.3%
bs=10 sw=10 sl=64	p99	34,034 us	47,768 us	16,504 us	-28.8%	+106.2%
bs=100 sw=10 sl=64	throughput	844 tuples/sec	881 tuples/sec	1,210 tuples/sec	-4.2%	-30.3%
bs=100 sw=10 sl=64	MB/s	0.515 MB/s	0.537 MB/s	0.739 MB/s	-4.1%	-30.3%
bs=100 sw=10 sl=64	p50	115,695 us	112,358 us	85,326 us	+3.0%	+35.6%
bs=100 sw=10 sl=64	p95	144,777 us	133,776 us	91,092 us	+8.2%	+58.9%
bs=100 sw=10 sl=64	p99	144,777 us	133,776 us	102,379 us	+8.2%	+41.4%
bs=1000 sw=10 sl=64	throughput	970 tuples/sec	979 tuples/sec	1,240 tuples/sec	-0.9%	-21.8%
bs=1000 sw=10 sl=64	MB/s	0.592 MB/s	0.598 MB/s	0.757 MB/s	-1.0%	-21.8%
bs=1000 sw=10 sl=64	p50	1,029,900 us	1,010,680 us	840,622 us	+1.9%	+22.5%
bs=1000 sw=10 sl=64	p95	1,186,559 us	1,129,663 us	889,172 us	+5.0%	+33.4%
bs=1000 sw=10 sl=64	p99	1,186,559 us	1,129,663 us	931,615 us	+5.0%	+27.4%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,520.34,200,128000,384,0.235,25234.96,34034.37,34034.37
1,100,10,64,20,2370.34,2000,1280000,844,0.515,115695.33,144777.09,144777.09
2,1000,10,64,20,20627.44,20000,12800000,970,0.592,1029900.46,1186559.02,1186559.02

kz930 · 2026-09-27T03:44:41Z

@carloea2 Split out of #8341 so each part reads in one sitting; the code is what you reviewed there, plus today's timestamp fixes.

carloea2

One text-content mismatch remains in the single-string read. Checked Python's text reader locally against the native byte-decoding path.

… export it Read as a single string, the export opened the file in text mode, which turns a CRLF into a LF. The engine decodes the file's bytes as they are, so the two disagreed on any file with Windows line endings. Open it with newline="" for both File Scan operators; reading by line keeps the default, which already ends a line where the engine does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov-commenter · 2026-10-08T08:27:38Z

Codecov Report

❌ Patch coverage is 68.63905% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.10%. Comparing base (7b8422e) to head (97f1b20).
⚠️ Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
...rator/source/scan/text/TextInputSourceOpDesc.scala	7.40%	25 Missing ⚠️
...erator/source/scan/file/FileScanSourceOpDesc.scala	67.27%	4 Missing and 14 partials ⚠️
...ber/operator/source/scan/file/FileScanOpDesc.scala	88.52%	0 Missing and 7 partials ⚠️
...r/operator/source/scan/text/TextSourceOpDesc.scala	80.00%	0 Missing and 2 partials ⚠️
...ber/operator/source/fetcher/URLFetcherOpDesc.scala	93.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8705      +/-   ##
============================================
- Coverage     92.18%   92.10%   -0.08%     
+ Complexity     5051     5006      -45     
============================================
  Files          1286     1267      -19     
  Lines         55937    55886      -51     
  Branches       7140     7179      +39     
============================================
- Hits          51563    51475      -88     
- Misses         2612     2621       +9     
- Partials       1762     1790      +28

Flag	Coverage Δ		*Carryforward flag
access-control-service	`77.44% <ø> (ø)`
agent-service	`99.16% <ø> (ø)`		Carriedforward from 5b2f864
amber	`86.67% <68.63%> (-0.27%)`	⬇️	Carriedforward from 5b2f864
computing-unit-managing-service	`64.02% <ø> (ø)`
config-service	`87.61% <ø> (ø)`
file-service	`82.29% <ø> (ø)`		Carriedforward from 5b2f864
frontend	`96.48% <ø> (ø)`		Carriedforward from 5b2f864
notebook-migration-service	`83.77% <ø> (ø)`
pyamber	`98.54% <ø> (+0.06%)`	⬆️	Carriedforward from 5b2f864
workflow-compiling-service	`77.95% <ø> (+3.70%)`	⬆️	Carriedforward from 5b2f864

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

carloea2

The CRLF change addresses the earlier finding. A new native versus generated Text Input test fails on a Unicode separator; its LF and CRLF controls pass. The test ran with the declared #8341 shared API supplied locally.

The engine splits with linesIterator, which ends a line only at LF, CR and CRLF. The export used splitlines, which also ends one at eight other characters such as U+2028 and NEL, so a row holding one came out as two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-12T22:08:57Z

@carloea2 Fixed the U+2028 split in e9d9824. The export now ends a line only at LF, CR and CRLF, like linesIterator, with a parity test over every character splitlines breaks at. Could you take another look?

carloea2

Retested the current head. The Unicode line separator regression is fixed. All 62 focused tests and scalafmtCheckAll pass with the #8341 source API and the existing Windows test cleanup from #8796 supplied locally. Without #8796, only that UTF-16 cleanup test fails on Windows.

Command: sbt -batch "WorkflowOperator/testOnly *FileScanOpDescSpec *FileScanSourceOpDescSpec *TextInputSourceOpDescSpec *URLFetcherOpDescSpec -- -oDF" "scalafmtCheckAll".

No new finding in the line source changes. The current checks only cover metadata, so this is not a green build confirmation. #8341 and #8597 remain prerequisites.

int() and float() read "1_0", "nan", "inf" and an INTEGER past 32 bits, which end the engine's run, and float() refused "2d" and "0x1p3", which the engine reads. Each line is now trimmed as Java trims and checked against the grammar parseInt, parseLong or parseDouble accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Every file now opens binary and goes through _texera_decode. Bytes that cannot be decoded become U+FFFD, and UTF-16 with no byte-order mark reads as big-endian, as Java's InputStreamReader does. A UTF-8 file read as UTF-16 used to end the script, while the engine read on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-22T08:02:44Z

@carloea2 Two commits since your review: a number line is read the way the engine parses it, and File Scan decodes bytes the way the engine's reader does. Could you take another look?

carloea2

The 19 focused export tests and scalafmtCheckAll pass with #8341 and #8597 supplied. The full four-suite run gives 62 passes and one Windows file-lock failure in the unchanged UTF-16 test cleanup. Rebase must retain upstream NoLoopVariable alongside the new TextSourceOpDesc import. No new export blocker found.

…rted script _texera_parse_int and _texera_parse_double explained themselves in Python comments, which every exported text-source script printed. The notes now sit in the Scaladoc, one line per function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…parsers NumberParsers repeated String.trim and Double.parseDouble beside the copies in AttributeCasts. It is gone: a line is read with _texera_parse_int and _texera_java_double from there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ean cast Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-26T22:20:00Z

@carloea2 Since your approval, the line sources read numbers and booleans with the shared Java parsers from #8341, so _texera_parse_bool is gone, and the exported Python carries no comments. Could you take another look?

…rces-lines # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextSourceOpDesc.scala

…es (apache#8597) ### What changes were proposed in this PR? Type Casting to a timestamp read text with `pd.to_datetime(col, errors="coerce")` in the exported script. That is wrong in two ways, and both empty a cell the run itself filled. pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a `java.sql.Timestamp`, where `2500-01-01 00:00:00` is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for `1500-06-15 08:30:00` and `9999-12-31 23:59:59`. pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands `DateParserUtils` one field at a time, so a row states its own format. A column holding `2024-03-05 14:09:07` and `March 5, 2024` kept the first and emptied the second. The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. Text no format reads now ends the run, as the engine's `parseField` ends its own. The script raises the engine's message, `Failed to parse type str to Timestamp`, where it used to leave the cell empty. dateutil reads every format the verification hands the engine, so the two refuse the same text. Review and the verification's edge values found more places where the cast parted from the engine: - A reading that states an offset, such as `Z` or `+05:30`, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same. - A column that already held timestamps went through the text parse and lost its nanoseconds. `parseField` returns a `java.sql.Timestamp` untouched, so the script now hands such a column back at the resolution it arrived in. - Text with digits past the millisecond kept them. `DateParserUtils` reads into a `java.util.Date`, which counts milliseconds, so the script now cuts there too. - Casting to text wrote a value the way Python's `str()` does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least `.0`. A shared `_texera_java_text` now writes each one as Java's `toString` does. - A double column holding a NaN as a value, as an Arrow or Parquet source hands one over, lost it. The casts asked each cell whether it was empty, and pandas hands an empty cell over as NaN too, so a stored NaN came out null where the engine writes `NaN` as text, 0 as an integer and `true` as a boolean. The casts now read empty cells from the column's own mask, and the double cast builds a nullable `Float64`, so a NaN read from text stays one through the next cast. A double past an integer's bounds now saturates as `toInt` and `toLong` do, where `int()` raised on an infinity. - A moment was read with dateutil's `tzlocal()`, which knows only today's two offsets, so an instant from before a zone changed its rules came out an hour or more off on every platform. The script now reads the machine's zone by its IANA name and, before 1900, keeps today's standard offset as `java.util.Date` does. - On Windows, `localtime` refuses an instant before 1970, which stopped the cast. The same helper now falls back to the zone's standard offset there. That `Float64` column changed what a Python operator after this cast is handed. pyamber hands a Python operator a DOUBLE column as `float64`, where a NaN and a null are one value, and writes a NaN out as null. Given `Float64` instead, Sort ordered a NaN apart from the nulls, a chart drew points the engine dropped, and a passed-through NaN stayed NaN. `StandaloneCodeGenerator` gains `standaloneInputAdapter()`, and for a Python operator the translator feeds the fragment a `float64` copy of each input, leaving the upstream frame as it was. The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where `new Timestamp(long)` is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it. ### Any related issues, documentation, discussions? Closes apache#8595. apache#8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot. apache#8341, apache#8704, apache#8705 and apache#8709 call the timestamp and text helpers this adds, so this PR merges before them. ### How was this PR tested? Eight tests in `TypeCastingOpDescSpec` run the generated Python and compare it cell by cell against `AttributeTypeUtils.parseField`. They are tagged to run in the integration job, which has pandas. - The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, `2500-01-01`, `1500-06-15`, `9999-12-31`, readings with `Z` and `+05:30`, and readings past the millisecond. It fails on the old generator. - One test gives a column that already holds timestamps and checks its nanoseconds are kept. - The cast-to-string test covers a double at 1e7 and 1e-4. - One test checks that unreadable text ends the script with the engine's message, as the engine refuses it. - Two tests cast a column holding a stored NaN, both infinities and an empty cell to each type and compare against `AttributeTypeUtils`, and check that a NaN read from text survives a second cast. - Two tests read moments from `America/Mexico_City` in 1899, 1921, 1931 and 2021, at the offset the zone had then, and the same moments with `localtime` refusing those before 1970. - Three tests in `WorkflowToPythonTranslatorSpec` check that a Python operator reads a converted copy of its input and an operator the engine runs on the JVM reads its upstream as it is. Two of them fail without the change. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

carloea2

File Scan Source still decodes differently from the engine.

Python's utf-16 codec refuses a file with no byte-order mark and reads a zip entry as little-endian. The engine reads both as big-endian. Every text read, archive entries included, now opens binary and goes through _texera_decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-11-04T21:07:13Z

@carloea2 File Scan Source and its archive branches now decode with the shared _texera_decode, so a BOM-less UTF-16BE file matches the engine. Could you take another look?

carloea2

Retested BOM-less UTF16BE, replacement decoding and ZIP entries against native execution in 42 cases; the outputs match. The one Windows fixture cleanup failure also reproduces on main.

aglinxinyuan

LGTM!

feat(operator): export the line sources and the URL fetcher

efb2dfc

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions Bot assigned kz930 Sep 27, 2026

github-actions Bot added feature common labels Sep 27, 2026

This was referenced Sep 27, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

test(verify): run the source operators, and report what is covered #8364

Draft

feat(operator): read a timestamp cast over the years the engine reaches #8597

Merged

kz930 mentioned this pull request Sep 27, 2026

feat(workflow-operator): export the CSV sources as Python #8341

Merged

carloea2 suggested changes Oct 7, 2026

View reviewed changes

Comment thread .../src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala Resolved

kz930 requested a review from carloea2 October 8, 2026 07:53

carloea2 suggested changes Oct 12, 2026

View reviewed changes

Comment thread ...src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextInputSourceOpDesc.scala Resolved

carloea2 reviewed Oct 13, 2026

View reviewed changes

kz930 marked this pull request as draft October 15, 2026 22:58

carloea2 approved these changes Oct 23, 2026

View reviewed changes

kz930 and others added 3 commits October 23, 2026 15:37

feat(workflow-operator): read text-line booleans with the shared bool…

68872da

…ean cast Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'upstream/main' into feat/standalone-sou…

2b63b7c

…rces-lines # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextSourceOpDesc.scala

kz930 marked this pull request as ready for review November 2, 2026 20:25

carloea2 suggested changes Nov 4, 2026

View reviewed changes

Comment thread .../src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala Resolved

kz930 and others added 2 commits November 4, 2026 02:14

ci: re-run CI after a GitHub API failure in precheck

97f1b20

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 requested a review from carloea2 November 4, 2026 21:17

carloea2 approved these changes Nov 4, 2026

View reviewed changes

aglinxinyuan approved these changes Nov 5, 2026

View reviewed changes

aglinxinyuan enabled auto-merge November 5, 2026 04:44

aglinxinyuan added this pull request to the merge queue Nov 5, 2026

Merged via the queue into apache:main with commit 241b504 Nov 5, 2026
34 checks passed

Conversation

kz930 commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

github-actions Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

kz930 commented Sep 27, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kz930 commented Oct 12, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Oct 22, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Oct 26, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kz930 commented Nov 4, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

aglinxinyuan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kz930 commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

github-actions Bot commented Sep 27, 2026 •

edited

Loading

codecov-commenter commented Oct 8, 2026 •

edited

Loading