Conversation
Part of apache#19524. Remove the changes job and every needs and step-level if that depended on it; the docs-only skip is a trigger-level paths filter, evaluated before any job exists. validate-source and validate-ci-baseline move verbatim to validate_source.yml, which is never path filtered because both are required status checks. The Flink, bundle-validation and integration-test jobs move verbatim to java_ci_engines.yml so each workflow stays under the ASF target of 15 concurrent jobs. The concurrency group carries the workflow name so the three workflows cancel only their own superseded runs. Review follow-ups: .github/** added to the positive paths list, a note in .asf.yaml on why only always-on workflows may contribute required contexts, and a shorter Codecov range comment.
Part of apache#19524, on top of the flattened job graph. - Surefire runs with hudi.surefire.forkCount=0.5C: two forks on a 4-core GitHub runner, one on a 2-core Azure agent. The two fixed test ports, HiveServer2 and the test ZooKeeper, are picked per JVM so forks do not collide; forks share one JaCoCo exec file. - TestHoodieDeltaStreamer is split into four classes by area, with bodies moved verbatim, so a second fork can help the utilities job. Three helpers move to HoodieDeltaStreamerTestBase. - The bloom filter benchmark stays out of unit runs. - The Spark workflow is resharded by measured time: Java UT split into the functional package and the rest, the three fat classes sharing a job with FTC, FTA and FTB one job each, Scala DML and Scala other split by package, utilities split into the DeltaStreamer classes and the rest. 14 jobs, under the ASF target of 15. Measured: 615 to 577 runner-minutes per run, longest job 77 to 41 minutes.
Part of apache#19524. Workflow files only. - flink-2's failsafe filter -Dit.test=!ITTestHoodieDataSource dropped failsafe's IT* include and ran 170 unit-test classes the docker job already runs. New filter: ITTest*,!ITTestHoodieDataSource. - The docker job built the whole reactor, 7.8 minutes, of which the 15 bundles and the examples are 6. The build, UT and FT reactors now skip the bundles, the examples and hudi-cli, each of which is built, tested or validated in the job that owns it. - hudi-examples-spark is built only in the jobs that run its quickstart. Measured over three runs: 577 to a mean of 548 runner-minutes, flink-2 39 to 23 minutes, the docker job 37 to about 30.
Closes apache#19528. Part of apache#19524. scripts/ci/test_timing_summary.py walks every surefire-reports directory, which surefire and the scalatest plugin both fill with JUnit XML, and prints the slowest classes and the slowest tests. Every job that runs tests, 14 in bot.yml and 3 in java_ci_engines.yml, gets one step after its test steps with if: always(), so the tables appear on the run summary for green and red jobs alike. Under a second of runner time per job.
Part of apache#19524. IT_MODULES names the four modules with integration-test classes outside hudi-flink: hudi-cli, hudi-integ-test, hudi-aws and hudi-azure. The build passes them with -am, which Maven resolves to 30 modules; the IT step verifies just the four instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it. Measured: the job from 37 to 28 minutes, same 25 test classes.
Add hudi.spark.test.sharedSession (default false), forwarded by the scalatest plugin. When set, every suite on HoodieSparkSqlTestBase shares one never-stopped SparkContext and works in its own SparkSession.newSession() child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse. A suite's conf deltas (extraConf, sparkConf overrides) are validated first and then applied to the child session; spark.hadoop.* keys go to the shared Hadoop conf and are restored in afterAll, any other context-level or static key is rejected before anything mutates. The child session is pinned as the active session on the suite thread, with a check that its HoodieCatalog is bound to it. The per-test cleanup drops only the suite's own tables. With the property off nothing changes. It is switched on for the three Java 17 Scala shards whose packages hold nothing but HoodieSparkSqlTestBase suites (dml-1, dml-2, other-1); other-2 mixes in suites that create their own SparkContext. TestHoodieDataUtils no longer closes its JavaSparkContext in afterAll, which stopped the context. Supersedes the proof of concept in apache#19921, where this ran green serially on the dml packages at master's serial time.
ParquetUtils.readAvroRecords set the Avro read schema on the caller's Hadoop Configuration, which is usually the SparkContext wide one, and the schema-less overload built its reader on that same object. Whatever read schema an earlier caller left behind was silently applied to later schema-less reads. With the SQL suites sharing one SparkContext, the hdfs parquet import procedure (HDFSParquetImporterUtils sets the schema on jsc.hadoopConfiguration after Job.getInstance has already copied it) leaked a schema without Hudi meta fields into the JVM, and repair_deduplicate then failed with "Avro field '_hoodie_commit_time' not found". Both overloads now read on a private copy of the configuration. The schema-less one also drops any read schema or projection the copy carried, so it reads with the file's own schema.
Prepares HoodieSparkSqlTestBase suites for running concurrently in one JVM: - Literal table and database names in the SQL suites come from generateTableName, which is unique per suite; path assertions that embedded a literal build from the same value. - HoodieInMemoryHashIndex keeps one map per table (base path) and can clear a single table; the base clears globally only in per-suite mode and per table on drop in shared mode. - ExclusiveSuite marks suites that mutate JVM-wide state (shared Hadoop conf, persisted RDDs, metrics registry, Hudi's global properties). Every suite holds the read side of a fair read/write lock for its whole run; an exclusive suite holds the write side. No effect until suites run in parallel. - TestInsertTable4 scopes its stage listener to the jobs the test started, through a local property, and removes it in a finally. - TestRepairsProcedure no longer closes the JVM-cached FileSystem. Supersedes the proof of concept in apache#19923.
The eight catalog-wide `show tables` assertions saw the tables of suites running at the same time on the shared SparkContext. They now list `show tables like '<prefix>*'`, where the prefix is the one generateTableName uses for this suite, so the expected rows are unchanged and other suites' tables are ignored.
Add hudi.scalatest.parallel (default false) and hudi.scalatest.threads (default 0), mapped to the scalatest plugin's parallel and threadCount parameters: suites run concurrently in one JVM, tests within a suite stay sequential. Set to true and 2 on the three Java 17 Scala shards that already share one SparkContext (dml-1, dml-2, other-1). Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's generateTableName prefix or as a session temp view, since the earlier "not a generated name" clause let one suite drop another's derived names mid-test; and two test helpers that sorted log file paths as strings now sort by the log file comparator, because the write token precedes the instant in the file name and stage ids compare as text (latent on master). Supersedes the proof of concept in apache#19925, where the dml shard at two threads measured 27:58, 35:27 and 35:32 against 41 to 56 minutes serial, with the same 326 tests each time.
Phase 4a measurement: the three shards ran two suites at a time with both threads busy 87 to 92 percent of the wall clock, so the thread count goes to four to see whether the shards still scale on a 4 vCPU runner. Same three shards, same flags.
Four threads were measured twice on the three shards: dml-1 17:20 and 13:35 against 19:36 to 20:19 at two threads, dml-2 and other-1 inside their two-thread spread, all green. The gain is on a shard that does not set the run's wall clock, and every suite runs about 60 percent longer under four-way contention on the 4 vCPU runner, so two threads stays the default. Four is a measured, safe setting for a shard that becomes the critical path. This reverts commit b8abd45.
scala-other-2 was the run's longest job at 35 to 37 minutes and the only Scala shard still serial, because its packages mix 33 suites on HoodieSparkSqlTestBase (26.5 of its 28.4 minutes) with 10 suites that build their own SparkContext (1.9 minutes). A ScalaTest tag annotation, SharedSessionSuite, marked @inherited and placed on HoodieSparkSqlTestBase, now identifies the first group without listing suites: scala-other-2 runs the tagged suites in shared-session mode two at a time (-DtagsToInclude), and the untagged suites of the same packages run as a short extra step of test-spark-java17-ftb (-DtagsToExclude). A new suite lands on the serial side unless it extends the base.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19932 +/- ##
============================================
+ Coverage 80.20% 80.22% +0.01%
- Complexity 34661 34674 +13
============================================
Files 2546 2546
Lines 142452 142622 +170
Branches 17330 17483 +153
============================================
+ Hits 114260 114422 +162
+ Misses 20281 20259 -22
- Partials 7911 7941 +30
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
HoodieBackedTableMetadataIndexLookupTestBase unpersists every RDD of the SparkContext in its setup and teardown and asserts that the context holds none, so with a shared context a neighbouring suite's cached RDD fails "Should start with no persistent RDDs" (seen on the first two-thread run of the other-2 suites). The base now mixes in ExclusiveSuite, like TestSecondaryIndex and TestExpressionIndex, so its four short suites run alone.
hudi-spark's pom pins tagsToExclude to the SparkSQLCoreFlow tag in the scalatest plugin configuration, and an explicit pom value wins over -DtagsToExclude, so the ftb step's exclusion of the shared session tag was silently ignored: it ran all 239 tests of the other-2 packages serially instead of the 88 own-context ones. The pom value moves to a property, hudi.scalatest.tagsToExclude, with the same default, and the ftb step passes the core-flow tag plus the shared-session tag through it.
HadoopFSUtils.getFSDataInputStream unwraps the FSDataInputStream a
filesystem returns, buffers its inner FSInputStream and hands out
the buffered stream, so closing what Hudi returns never closes the
object the filesystem created. Filesystems that track the streams
they hand out, such as Spark's test DebugFilesystem, then report
every Hudi log file read as leaked although the OS stream is
closed. TestStreamingSource aborted on exactly that ("26 possibly
leaked file streams") once it ran in a JVM where DebugFilesystem
was the cached local filesystem; in the old serial job a plain
LocalFileSystem got cached first and the check was vacuous.
The returned stream now closes the original FSDataInputStream
after the buffered wrapper; the inner stream's close is idempotent
by the Closeable contract. A unit test checks the outer object is
closed once.
InLineFsDataInputStream wraps the outer file's FSDataInputStream in an offset-adjusting InputStream and passes that to its super constructor. Neither class overrides close(), so closing an inline stream ends at InputStream.close(), a no-op, and the outer file's stream stays open until garbage collection. Every inline HFile read (metadata table log blocks) leaked a file handle this way. Spark's DebugFilesystem in TestStreamingSource reported it as "possibly leaked file streams" once the previous commit made its accounting see Hudi's reads. close() now closes the wrapped stream and then the outer one. A unit test checks the outer stream is closed exactly once.
This was referenced Sep 13, 2026
Collaborator
voonhous
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Part of #19524, stacked on #19928 (phase 4 of the Scala parallel work). With dml-1, dml-2 and other-1 running suites two at a time, the run's longest job became scala-other-2 at 35 to 37 minutes: the only Scala shard still serial, because its packages mix 33 suites on
HoodieSparkSqlTestBase(26.5 of its 28.4 minutes of tests) with 10 suites that build their own SparkContext (1.9 minutes).Summary and Changelog
New ScalaTest tag annotation
SharedSessionSuite(@TagAnnotation,@Inherited) onHoodieSparkSqlTestBase, so every suite on that base carries the tag without being listed, including the procedure base's subclasses.scala-other-2 runs only the tagged suites of its packages (
-DtagsToInclude), in shared-session mode two at a time, the same flags as the other three SQL shards.The untagged suites of the same packages (
-DtagsToExclude) run as an extra step of test-spark-java17-ftb, the shortest Java 17 job. A new suite lands on this serial side unless it extends the base.Three things the first runs exposed, each its own commit: the four index lookup suites unpersist every RDD in the context and assert it holds none, so their base runs alone (
ExclusiveSuite); hudi-spark's pom pinned the scalatesttagsToExclude, which silently beat the-Dflag, so it is now the propertyhudi.scalatest.tagsToExcludewith the same default; and two stream-close gaps in hudi-hadoop-common that Spark's DebugFilesystem exposed once it saw Hudi's reads:HadoopFSUtils.getFSDataInputStreamnever closed the FSDataInputStream object a filesystem returned, only the inner stream it rebuffered (an accounting miss), andInLineFsDataInputStreamhad nocloseat all, so every inline HFile read of a metadata table log block leaked the outer file handle until GC (a real leak). Each has a unit test, and each is also up on its own for review, fix: close the filesystem's stream on rewrap #19935 and fix: close the outer file stream of inline reads #19936; this PR carries copies until they merge and drops them on its next rebase.Job count unchanged at fourteen.
Measurement, three samples: scala-other-2's scalatest wall clock against its serial 28:24, the ftb job against its 21 to 28, and the run's longest job against 35 to 37.
Sample 1 notes: 151 tests on other-2 plus 88 on ftb's step are exactly the 239 names of the serial job. other-2 at two threads keeps only 1.2 threads busy because TestSecondaryIndex, TestExpressionIndex and the lookup suites run alone under the exclusive lock; that is its floor now. Runner-minutes for the runs: 514, 491, 512.
Verdict after three samples: other-2 goes from 28:24 of tests and a 35 to 37 minute job to 22:56 to 23:44 and 30 to 31, every sample below serial, the same 239 test names split 151 and 88 each time. ftb absorbs the 88 own-context tests at 2:26 to 2:49 plus about a minute of Maven, and lands at 30 to 35 depending on the runner. The run's longest job is now a Java job every time: fta, fta, ut-fat-and-ftc at 33 to 39. The Scala shards no longer set the wall clock, which is where this stack stops.
Why the stream leaks only showed up now
Spark's
StreamTestsuites installDebugFilesystemforfile:and assert at teardown that every stream it handed out was closed. Hadoop caches one FileSystem per scheme for the whole JVM, keyed by scheme, not by implementation. In the serial other-2 job the SQL suites ran first and cached a plain LocalFileSystem, so DebugFilesystem never saw Hudi's reads and the check had nothing to check. In the new ftb step the twoStreamTestsuites are the first to touchfile:, the check finally covers Hudi's log and HFile reads, and TestStreamingSource aborted with "26 possibly leaked file streams". 23 of those were the rebuffering accounting miss; the remaining 3 were the inline filesystem never closing the outer file, which is a real handle leak on every inline HFile read.Why a tag rather than package filters or a suite list
The two families live in the same packages, and the plugin's
wildcardSuitescannot exclude. An explicit list of the ten own-context suites would silently route a new own-context suite into the shared-session job, where it would break. The inherited tag makes the shared-session side opt-in by base class, verified locally against the project's ScalaTest: a Java annotation with@TagAnnotationand@Inheritedon a base class is seen by-nand-lthrough two levels of subclassing.Impact
CI configuration and test infrastructure, plus two stream-close fixes in hudi-hadoop-common, one of them a real file-handle leak on inline HFile reads.
Risk Level
low. Off by default like the rest of the stack; the ten own-context suites keep a private SparkContext.
Documentation Update
none
Contributor's checklist