Conversation
Part of apache#19524. Remove the changes job and every needs and step-level if that depended on it; the docs-only skip is a trigger-level paths filter, evaluated before any job exists. validate-source and validate-ci-baseline move verbatim to validate_source.yml, which is never path filtered because both are required status checks. The Flink, bundle-validation and integration-test jobs move verbatim to java_ci_engines.yml so each workflow stays under the ASF target of 15 concurrent jobs. The concurrency group carries the workflow name so the three workflows cancel only their own superseded runs. Review follow-ups: .github/** added to the positive paths list, a note in .asf.yaml on why only always-on workflows may contribute required contexts, and a shorter Codecov range comment.
Part of apache#19524, on top of the flattened job graph. - Surefire runs with hudi.surefire.forkCount=0.5C: two forks on a 4-core GitHub runner, one on a 2-core Azure agent. The two fixed test ports, HiveServer2 and the test ZooKeeper, are picked per JVM so forks do not collide; forks share one JaCoCo exec file. - TestHoodieDeltaStreamer is split into four classes by area, with bodies moved verbatim, so a second fork can help the utilities job. Three helpers move to HoodieDeltaStreamerTestBase. - The bloom filter benchmark stays out of unit runs. - The Spark workflow is resharded by measured time: Java UT split into the functional package and the rest, the three fat classes sharing a job with FTC, FTA and FTB one job each, Scala DML and Scala other split by package, utilities split into the DeltaStreamer classes and the rest. 14 jobs, under the ASF target of 15. Measured: 615 to 577 runner-minutes per run, longest job 77 to 41 minutes.
Part of apache#19524. Workflow files only. - flink-2's failsafe filter -Dit.test=!ITTestHoodieDataSource dropped failsafe's IT* include and ran 170 unit-test classes the docker job already runs. New filter: ITTest*,!ITTestHoodieDataSource. - The docker job built the whole reactor, 7.8 minutes, of which the 15 bundles and the examples are 6. The build, UT and FT reactors now skip the bundles, the examples and hudi-cli, each of which is built, tested or validated in the job that owns it. - hudi-examples-spark is built only in the jobs that run its quickstart. Measured over three runs: 577 to a mean of 548 runner-minutes, flink-2 39 to 23 minutes, the docker job 37 to about 30.
Closes apache#19528. Part of apache#19524. scripts/ci/test_timing_summary.py walks every surefire-reports directory, which surefire and the scalatest plugin both fill with JUnit XML, and prints the slowest classes and the slowest tests. Every job that runs tests, 14 in bot.yml and 3 in java_ci_engines.yml, gets one step after its test steps with if: always(), so the tables appear on the run summary for green and red jobs alike. Under a second of runner time per job.
Part of apache#19524. IT_MODULES names the four modules with integration-test classes outside hudi-flink: hudi-cli, hudi-integ-test, hudi-aws and hudi-azure. The build passes them with -am, which Maven resolves to 30 modules; the IT step verifies just the four instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it. Measured: the job from 37 to 28 minutes, same 25 test classes.
Add hudi.spark.test.sharedSession (default false), forwarded by the scalatest plugin. When set, every suite on HoodieSparkSqlTestBase shares one never-stopped SparkContext and works in its own SparkSession.newSession() child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse. A suite's conf deltas (extraConf, sparkConf overrides) are validated first and then applied to the child session; spark.hadoop.* keys go to the shared Hadoop conf and are restored in afterAll, any other context-level or static key is rejected before anything mutates. The child session is pinned as the active session on the suite thread, with a check that its HoodieCatalog is bound to it. The per-test cleanup drops only the suite's own tables. With the property off nothing changes. It is switched on for the three Java 17 Scala shards whose packages hold nothing but HoodieSparkSqlTestBase suites (dml-1, dml-2, other-1); other-2 mixes in suites that create their own SparkContext. TestHoodieDataUtils no longer closes its JavaSparkContext in afterAll, which stopped the context. Supersedes the proof of concept in apache#19921, where this ran green serially on the dml packages at master's serial time.
ParquetUtils.readAvroRecords set the Avro read schema on the caller's Hadoop Configuration, which is usually the SparkContext wide one, and the schema-less overload built its reader on that same object. Whatever read schema an earlier caller left behind was silently applied to later schema-less reads. With the SQL suites sharing one SparkContext, the hdfs parquet import procedure (HDFSParquetImporterUtils sets the schema on jsc.hadoopConfiguration after Job.getInstance has already copied it) leaked a schema without Hudi meta fields into the JVM, and repair_deduplicate then failed with "Avro field '_hoodie_commit_time' not found". Both overloads now read on a private copy of the configuration. The schema-less one also drops any read schema or projection the copy carried, so it reads with the file's own schema.
Prepares HoodieSparkSqlTestBase suites for running concurrently in one JVM: - Literal table and database names in the SQL suites come from generateTableName, which is unique per suite; path assertions that embedded a literal build from the same value. - HoodieInMemoryHashIndex keeps one map per table (base path) and can clear a single table; the base clears globally only in per-suite mode and per table on drop in shared mode. - ExclusiveSuite marks suites that mutate JVM-wide state (shared Hadoop conf, persisted RDDs, metrics registry, Hudi's global properties). Every suite holds the read side of a fair read/write lock for its whole run; an exclusive suite holds the write side. No effect until suites run in parallel. - TestInsertTable4 scopes its stage listener to the jobs the test started, through a local property, and removes it in a finally. - TestRepairsProcedure no longer closes the JVM-cached FileSystem. Supersedes the proof of concept in apache#19923.
The eight catalog-wide `show tables` assertions saw the tables of suites running at the same time on the shared SparkContext. They now list `show tables like '<prefix>*'`, where the prefix is the one generateTableName uses for this suite, so the expected rows are unchanged and other suites' tables are ignored.
Add hudi.scalatest.parallel (default false) and hudi.scalatest.threads (default 0), mapped to the scalatest plugin's parallel and threadCount parameters: suites run concurrently in one JVM, tests within a suite stay sequential. Set to true and 2 on the three Java 17 Scala shards that already share one SparkContext (dml-1, dml-2, other-1). Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's generateTableName prefix or as a session temp view, since the earlier "not a generated name" clause let one suite drop another's derived names mid-test; and two test helpers that sorted log file paths as strings now sort by the log file comparator, because the write token precedes the instant in the file name and stage ids compare as text (latent on master). Supersedes the proof of concept in apache#19925, where the dml shard at two threads measured 27:58, 35:27 and 35:32 against 41 to 56 minutes serial, with the same 326 tests each time.
Phase 4a measurement: the three shards ran two suites at a time with both threads busy 87 to 92 percent of the wall clock, so the thread count goes to four to see whether the shards still scale on a 4 vCPU runner. Same three shards, same flags.
Four threads were measured twice on the three shards: dml-1 17:20 and 13:35 against 19:36 to 20:19 at two threads, dml-2 and other-1 inside their two-thread spread, all green. The gain is on a shard that does not set the run's wall clock, and every suite runs about 60 percent longer under four-way contention on the 4 vCPU runner, so two threads stays the default. Four is a measured, safe setting for a shard that becomes the critical path. This reverts commit b8abd45.
scala-other-2 was the run's longest job at 35 to 37 minutes and the only Scala shard still serial, because its packages mix 33 suites on HoodieSparkSqlTestBase (26.5 of its 28.4 minutes) with 10 suites that build their own SparkContext (1.9 minutes). A ScalaTest tag annotation, SharedSessionSuite, marked @inherited and placed on HoodieSparkSqlTestBase, now identifies the first group without listing suites: scala-other-2 runs the tagged suites in shared-session mode two at a time (-DtagsToInclude), and the untagged suites of the same packages run as a short extra step of test-spark-java17-ftb (-DtagsToExclude). A new suite lands on the serial side unless it extends the base.
HoodieBackedTableMetadataIndexLookupTestBase unpersists every RDD of the SparkContext in its setup and teardown and asserts that the context holds none, so with a shared context a neighbouring suite's cached RDD fails "Should start with no persistent RDDs" (seen on the first two-thread run of the other-2 suites). The base now mixes in ExclusiveSuite, like TestSecondaryIndex and TestExpressionIndex, so its four short suites run alone.
hudi-spark's pom pins tagsToExclude to the SparkSQLCoreFlow tag in the scalatest plugin configuration, and an explicit pom value wins over -DtagsToExclude, so the ftb step's exclusion of the shared session tag was silently ignored: it ran all 239 tests of the other-2 packages serially instead of the 88 own-context ones. The pom value moves to a property, hudi.scalatest.tagsToExclude, with the same default, and the ftb step passes the core-flow tag plus the shared-session tag through it.
HadoopFSUtils.getFSDataInputStream unwraps the FSDataInputStream a
filesystem returns, buffers its inner FSInputStream and hands out
the buffered stream, so closing what Hudi returns never closes the
object the filesystem created. Filesystems that track the streams
they hand out, such as Spark's test DebugFilesystem, then report
every Hudi log file read as leaked although the OS stream is
closed. TestStreamingSource aborted on exactly that ("26 possibly
leaked file streams") once it ran in a JVM where DebugFilesystem
was the cached local filesystem; in the old serial job a plain
LocalFileSystem got cached first and the check was vacuous.
The returned stream now closes the original FSDataInputStream
after the buffered wrapper; the inner stream's close is idempotent
by the Closeable contract. A unit test checks the outer object is
closed once.
InLineFsDataInputStream wraps the outer file's FSDataInputStream in an offset-adjusting InputStream and passes that to its super constructor. Neither class overrides close(), so closing an inline stream ends at InputStream.close(), a no-op, and the outer file's stream stays open until garbage collection. Every inline HFile read (metadata table log blocks) leaked a file handle this way. Spark's DebugFilesystem in TestStreamingSource reported it as "possibly leaked file streams" once the previous commit made its accounting see Hudi's reads. close() now closes the wrapped stream and then the outer one. A unit test checks the outer stream is closed exactly once.
The nine Java 17 Spark test jobs had no timeout-minutes, so a job whose surefire fork stops answering runs to GitHub's six hour default. That happened on a rerun of apache#19928: hudi-common finished all 184 test classes in 26 seconds, the fork never reported completion, and the job sat idle for 76 minutes until it was cancelled by hand. 75 minutes is what test-spark-client-and-hadoop-common already uses. The longest of these nine jobs measured 44 minutes on a slow runner, so the cap only fires on a hang.
The first run of this branch hung in test-flink-1's integration test step for the full six hour default before GitHub cancelled it. The five jobs of the engines workflow (flink-1, flink-2, the two bundle validations, integration-tests) had no timeout either; they measure 23 to 34 minutes, so 75 only fires on a hang.
test-hudi-hadoop-mr-and-hudi-java-client ran 9 minutes, 5 of them checkout and build, on the same JDK 11 lane as test-utilities-rest (18 to 22 minutes). Its test step moves into that job, whose build now also covers hudi-hadoop-mr and hudi-client/hudi-java-client, and its Codecov flag rides on the same upload. One job fewer per run, one build fewer, and the merged job stays well under the run's longest jobs at 33 to 36 minutes.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19937 +/- ##
============================================
+ Coverage 80.20% 80.22% +0.01%
- Complexity 34661 34679 +18
============================================
Files 2546 2546
Lines 142452 142622 +170
Branches 17330 17483 +153
============================================
+ Hits 114260 114414 +154
+ Misses 20281 20268 -13
- Partials 7911 7940 +29
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Collaborator
Member
Author
|
Three samples are on the description and on #19910, with the test-count parity check against master. Measurement only, closing as planned. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Do not merge. Measurement only, for #19910: every open CI change on one branch, so the end state of the campaign gets its own runner-minute and longest-job samples. It is #19932 (the Scala stack on the CI stack) plus #19931 (job timeouts) and #19933 (hadoop-mr folded into utilities-rest). No code of its own.
Summary and Changelog
Three samples, this run and two full reruns:
Three samples in: 491 to 511 runner-minutes (mean 503) against 615 on master, longest job 35 to 37 against 77, all green each time. The remaining longest jobs are Java: ut-fat-and-ftc, fta, spark-client-and-hadoop-common.
Impact
None; closed after the samples.
Risk Level
none
Documentation Update
none
Contributor's checklist