Conversation
Part of apache#19524. Remove the changes job and every needs and step-level if that depended on it; the docs-only skip is a trigger-level paths filter, evaluated before any job exists. validate-source and validate-ci-baseline move verbatim to validate_source.yml, which is never path filtered because both are required status checks. The Flink, bundle-validation and integration-test jobs move verbatim to java_ci_engines.yml so each workflow stays under the ASF target of 15 concurrent jobs. The concurrency group carries the workflow name so the three workflows cancel only their own superseded runs. Review follow-ups: .github/** added to the positive paths list, a note in .asf.yaml on why only always-on workflows may contribute required contexts, and a shorter Codecov range comment.
Part of apache#19524, on top of the flattened job graph. - Surefire runs with hudi.surefire.forkCount=0.5C: two forks on a 4-core GitHub runner, one on a 2-core Azure agent. The two fixed test ports, HiveServer2 and the test ZooKeeper, are picked per JVM so forks do not collide; forks share one JaCoCo exec file. - TestHoodieDeltaStreamer is split into four classes by area, with bodies moved verbatim, so a second fork can help the utilities job. Three helpers move to HoodieDeltaStreamerTestBase. - The bloom filter benchmark stays out of unit runs. - The Spark workflow is resharded by measured time: Java UT split into the functional package and the rest, the three fat classes sharing a job with FTC, FTA and FTB one job each, Scala DML and Scala other split by package, utilities split into the DeltaStreamer classes and the rest. 14 jobs, under the ASF target of 15. Measured: 615 to 577 runner-minutes per run, longest job 77 to 41 minutes.
Part of apache#19524. Workflow files only. - flink-2's failsafe filter -Dit.test=!ITTestHoodieDataSource dropped failsafe's IT* include and ran 170 unit-test classes the docker job already runs. New filter: ITTest*,!ITTestHoodieDataSource. - The docker job built the whole reactor, 7.8 minutes, of which the 15 bundles and the examples are 6. The build, UT and FT reactors now skip the bundles, the examples and hudi-cli, each of which is built, tested or validated in the job that owns it. - hudi-examples-spark is built only in the jobs that run its quickstart. Measured over three runs: 577 to a mean of 548 runner-minutes, flink-2 39 to 23 minutes, the docker job 37 to about 30.
Closes apache#19528. Part of apache#19524. scripts/ci/test_timing_summary.py walks every surefire-reports directory, which surefire and the scalatest plugin both fill with JUnit XML, and prints the slowest classes and the slowest tests. Every job that runs tests, 14 in bot.yml and 3 in java_ci_engines.yml, gets one step after its test steps with if: always(), so the tables appear on the run summary for green and red jobs alike. Under a second of runner time per job.
Part of apache#19524. IT_MODULES names the four modules with integration-test classes outside hudi-flink: hudi-cli, hudi-integ-test, hudi-aws and hudi-azure. The build passes them with -am, which Maven resolves to 30 modules; the IT step verifies just the four instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it. Measured: the job from 37 to 28 minutes, same 25 test classes.
Add hudi.spark.test.sharedSession (default false), forwarded by the scalatest plugin. When set, every suite on HoodieSparkSqlTestBase shares one never-stopped SparkContext and works in its own SparkSession.newSession() child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse. A suite's conf deltas (extraConf, sparkConf overrides) are validated first and then applied to the child session; spark.hadoop.* keys go to the shared Hadoop conf and are restored in afterAll, any other context-level or static key is rejected before anything mutates. The child session is pinned as the active session on the suite thread, with a check that its HoodieCatalog is bound to it. The per-test cleanup drops only the suite's own tables. With the property off nothing changes. It is switched on for the three Java 17 Scala shards whose packages hold nothing but HoodieSparkSqlTestBase suites (dml-1, dml-2, other-1); other-2 mixes in suites that create their own SparkContext. TestHoodieDataUtils no longer closes its JavaSparkContext in afterAll, which stopped the context. Supersedes the proof of concept in apache#19921, where this ran green serially on the dml packages at master's serial time.
3 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19928 +/- ##
============================================
- Coverage 80.20% 80.20% -0.01%
- Complexity 34661 34668 +7
============================================
Files 2546 2546
Lines 142452 142615 +163
Branches 17330 17483 +153
============================================
+ Hits 114260 114381 +121
- Misses 20281 20284 +3
- Partials 7911 7950 +39
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
ParquetUtils.readAvroRecords set the Avro read schema on the caller's Hadoop Configuration, which is usually the SparkContext wide one, and the schema-less overload built its reader on that same object. Whatever read schema an earlier caller left behind was silently applied to later schema-less reads. With the SQL suites sharing one SparkContext, the hdfs parquet import procedure (HDFSParquetImporterUtils sets the schema on jsc.hadoopConfiguration after Job.getInstance has already copied it) leaked a schema without Hudi meta fields into the JVM, and repair_deduplicate then failed with "Avro field '_hoodie_commit_time' not found". Both overloads now read on a private copy of the configuration. The schema-less one also drops any read schema or projection the copy carried, so it reads with the file's own schema.
Prepares HoodieSparkSqlTestBase suites for running concurrently in one JVM: - Literal table and database names in the SQL suites come from generateTableName, which is unique per suite; path assertions that embedded a literal build from the same value. - HoodieInMemoryHashIndex keeps one map per table (base path) and can clear a single table; the base clears globally only in per-suite mode and per table on drop in shared mode. - ExclusiveSuite marks suites that mutate JVM-wide state (shared Hadoop conf, persisted RDDs, metrics registry, Hudi's global properties). Every suite holds the read side of a fair read/write lock for its whole run; an exclusive suite holds the write side. No effect until suites run in parallel. - TestInsertTable4 scopes its stage listener to the jobs the test started, through a local property, and removes it in a finally. - TestRepairsProcedure no longer closes the JVM-cached FileSystem. Supersedes the proof of concept in apache#19923.
The eight catalog-wide `show tables` assertions saw the tables of suites running at the same time on the shared SparkContext. They now list `show tables like '<prefix>*'`, where the prefix is the one generateTableName uses for this suite, so the expected rows are unchanged and other suites' tables are ignored.
Add hudi.scalatest.parallel (default false) and hudi.scalatest.threads (default 0), mapped to the scalatest plugin's parallel and threadCount parameters: suites run concurrently in one JVM, tests within a suite stay sequential. Set to true and 2 on the three Java 17 Scala shards that already share one SparkContext (dml-1, dml-2, other-1). Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's generateTableName prefix or as a session temp view, since the earlier "not a generated name" clause let one suite drop another's derived names mid-test; and two test helpers that sorted log file paths as strings now sort by the log file comparator, because the write token precedes the instant in the file name and stage ids compare as text (latent on master). Supersedes the proof of concept in apache#19925, where the dml shard at two threads measured 27:58, 35:27 and 35:32 against 41 to 56 minutes serial, with the same 326 tests each time.
voonhous
force-pushed
the
scala-shared-session-c
branch
from
September 12, 2026 18:35
ba4b646 to
3aab942
Compare
Phase 4a measurement: the three shards ran two suites at a time with both threads busy 87 to 92 percent of the wall clock, so the thread count goes to four to see whether the shards still scale on a 4 vCPU runner. Same three shards, same flags.
Four threads were measured twice on the three shards: dml-1 17:20 and 13:35 against 19:36 to 20:19 at two threads, dml-2 and other-1 inside their two-thread spread, all green. The gain is on a shard that does not set the run's wall clock, and every suite runs about 60 percent longer under four-way contention on the 4 vCPU runner, so two threads stays the default. Four is a measured, safe setting for a shard that becomes the critical path. This reverts commit b8abd45.
3 tasks
Collaborator
voonhous
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Part of #19524, stacked on the shared-session and state-scoping PRs (successors of #19921 and #19923). Supersedes the proof of concept #19925. The Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork count, so the lever is running suites concurrently in one JVM. This PR turns that on for the three SQL-only shards and measures it.
Summary and Changelog
hudi.scalatest.parallel(default false) andhudi.scalatest.threads(default 0) mapped to the scalatest plugin'sparallelandthreadCount: suites run concurrently, tests within a suite stay sequential. Set to true and 2 on dml-1, dml-2 and other-1, next to the shared-session flag they carry.<generated>_ptwere dropped by other suites), and two helpers that sorted log file paths as strings now sort by the log file comparator (the write token precedes the instant in the name, so stage ids compared as text). The second is latent on master.Measurement, over three runs of each of the three shards: the scalatest "Run completed" wall clock against the same job serial. dml-1 and other-1 are the two jobs setting the run's critical runtime of about 41 minutes under #19892, so this is where the wall clock should move. A result inside the serial range is no gain.
Serial times are the scalatest wall clocks of #19892's runs 34581330915 and 34499656524, plus #19931's first run, which has the same layout without this stack; every sample ran the same test names as those runs.
One more two-thread sample after the four-thread experiment, on the revert commit (run 34742475247): dml-1 15:25, dml-2 16:21, other-1 19:43, longest job 37 (ut-fat-and-ftc), all green with the same test names.
Verdict after three or more samples per shard: dml-1 30 to 31 -> 15:25 to 20:19 (-34% to -50%), dml-2 19 to 23 -> 13:27 to 16:56 (-23%), other-1 32 to 33 -> 17:14 to 22:34 (-36%), every sample below every serial sample. The run's longest job moved from the Scala shards (39 to 44) to the serial scala-other-2 at 35 to 37, so the next step is that shard.
Phase 4a, four threads (commit b8abd45, reverted by fe4e203; the PR stays at two threads): does the shard still scale on a 4 vCPU runner once two threads keep both busy 87 to 92 percent of the time? Same measurement, two samples (the second decided it).
Four threads keep 3.6 of them busy on dml-1 and dml-2 and 3.0 on other-1, at the price of every suite running about 60 percent longer than at two threads (four sessions on four vCPUs). dml-1 gains on both samples; dml-2 and other-1 stay inside their two-thread spread. Test names identical to serial on all six shard runs. Decision: stay at two. The gain is on a shard that does not set the run's wall clock, and four-way contention on a 4 vCPU runner narrows every suite's timing margin; four threads is recorded here as a measured, safe setting for a shard that becomes the critical path. other-1's first run (job 103593487295, 21:53) failed ten tests: a shared-session leak in the parquet reader, fixed in the shared-session PR, and a catalog-wide
show tablesassertion, fixed in the state-scoping PR. It is not counted; other-1's third sample comes from one more rerun after attempt 3. (a) On attempt 3 the Java job ut-other hung after finishing all its hudi-common test classes (27 to 34 min on eleven earlier runs); it was cancelled at 76 min and rerun alone as attempt 4 in 34 min, which is the figure used. The Scala shards are unaffected. Both threads are busy 91%, 92% and 87% of the wall clock on dml-1, dml-2 and other-1 (sum of suite spans over wall).Evidence from the proof of concept (#19925)
On master's layout the single dml job (dml-1 plus dml-2 together) at two threads: 27:58, 35:27 and 35:32 (run 34697891947, attempts 1 to 3) against 41:22 on master's fastest runner and 52 to 56 on its typical one, the same 326 test names each time, both threads busy 96 to 98% of the time. Locally, twenty iterations of twelve interaction-heavy suites at two threads were green, and the whole dml tree at two threads passed in 22 minutes.
Impact
CI configuration and test infrastructure only.
Risk Level
low. Off by default; on three shards whose suites were prepared for it in the two PRs below and looped locally at two threads.
Documentation Update
none
Contributor's checklist