chore: cache the spark integration test archive by abhinav-phi · Pull Request #19986 · apache/hudi

abhinav-phi · 2026-09-17T13:10:00Z

Describe the issue this Pull Request addresses

This implements the caching / pre-seeding follow-up that #19883 invites after the dlcdn fast path from #19884 was merged: every integration-tests run still moves the 382 MB Spark tarball over the network. While implementing it, I found that a plain actions/cache is useless in this repository because the delete-all-caches job in scheduled_workflow.yml deletes every Actions cache every five minutes — so this PR also adds narrow exceptions to that purge for the Spark archive and Trino m2 keys.

Summary and Changelog

Add a Restore Spark archive step (actions/cache/restore@v4) to the integration-tests job in bot.yml. It restores the tarball to ~/spark-archives/${{ matrix.sparkArchive }} under the exact key spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}. There are deliberately no restore-keys prefixes, because a partially matching restore (a different Spark or Hadoop flavor) would be worse than downloading the correct archive. The purge exception for this key family lives in scheduled_workflow.yml — a pointer comment sits next to the key so the two stay in sync across key bumps.
Split the old monolithic IT step in two: a Fetch Spark archive step runs the existing dlcdn → archive fallback chain from chore(spark): bump Spark 3.5 to 3.5.9 and fast-path the CI download #19884 unchanged on a restore miss, and a Verify and extract Spark archive step checks Apache's published .sha512 on both the restored and the freshly downloaded paths, then extracts. Cold-run download behavior is identical to master today.
Persist the tarball with a separate actions/cache/save@v4 step placed immediately after a successful tar -xf and before mvn verify, gated on push-to-master plus a restore miss. The extract itself — not Hudi test health — is the integrity gate on the entry, and seeding no longer depends on a green master IT. PR runs and failed extracts never write to the cache, so the 10 GB repo quota only ever holds one ~382 MB entry per Spark archive generation, with stale generations aging out through GitHub's normal LRU eviction.
Exclude version-independent prefixes from the five-minute purge in scheduled_workflow.yml: spark-archive- and trino-m2-v2-. Pinning a literal version there would silently re-arm the purge on the next key bump. The Trino exception fixes the same dead-cache problem for the trino-m2-v2- entry in hudi_trino_ci.yml, which the purge was likewise deleting minutes after creation.

Impact

No public API or user-facing behavior change; this is CI-only. Warm runs skip the 382 MB egress entirely — measured on six master runs since dlcdn landed, the download itself is only 1–18s (median ~11s) against a ~36 minute job with a ~4.5s extrapolated restore, so the honest saving is roughly 6–14s per warm run plus removing the stale-pin tail risk: as Spark drops the pinned version from the fastly CDN, CI falls back to the slower archive origin. Cold runs keep the existing download path. No Spark version, profile, or mvn invocation is changed. The only lasting cost is at most one ~382 MB cache entry per archive generation against the 10 GB repo quota. No measured hosted-cache speedup is claimed yet; that requires a real run (tracked separately).

Risk Level

Low. The cache touches CI plumbing only: a cache miss, a failed save, or a failed checksum all fall back to — or fail loudly before — the exact download path merged in #19884. Seeding is gated on a successful tar -xf plus sha512sum -c, so a corrupt entry can neither be written nor consumed silently. The two judgment calls — keeping the Spark archive and Trino m2 caches out of the scheduled purge — are flagged here and on the issue for maintainer sign-off, and can be dropped from this PR if preferred.

Documentation Update

None.

Validation

Local plus a hosted validation run on a temporary fork branch (see the tracking issue linked below); nothing here has run on the apache/hudi cache service yet.

Behavior tests executed against the exact shell of the modified steps with mocked curl/sha512sum/mvn: exact hit (zero download calls), miss (single dlcdn call), dlcdn failure with archive success, both sources failing (non-zero exit, nothing cached), checksum mismatch on either path (non-zero exit before extraction), save skipped on non-master ref, and purge-filter cases (mixed keys delete only unprotected keys; Spark-only and Trino-only lists delete nothing; empty list deletes nothing).
Both workflow files parse as valid YAML; git diff --check is clean.
Hosted validation (temporary fork branch validate-spark-cache, tracked in Track hosted validation of the spark-archive cache seeding (follows #19986) #20011): restore-miss → download → checksum → extract → save → full IT all green on the hosted runner, and the fork cache holds spark-archive-v1-Linux-spark-3.5.9/spark-3.5.9-bin-hadoop3.tgz (382.72 MiB), proving the save gate end to end. The temporary branch (widened save gate, null-before-SHA bypass, one comment-only Java touch) was deleted after the run; none of that scaffolding is in this PR.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable (CI change; validated with the local behavior tests described above)

_{AI assistance was used for implementation, local validation, and this description.}

abhinav-phi · 2026-09-17T13:10:11Z

@voonhous — this is the caching follow-up for #19883 that you invited after merging #19884.

The design summary: the integration-tests job restores the Spark tarball under an exact key (spark-archive-v1-${{ runner.os }}-${{ matrix.sparkArchive }}), skips the download only on an exact hit, and saves it from successful master pushes only. PR runs and failed runs never write to the cache, and on a miss the existing dlcdn → archive fallback chain you added runs unchanged.

One thing needed your call, so I included it in this PR rather than deciding it myself: scheduled_workflow.yml purges every Actions cache in the repo every five minutes, so a shared cache is only possible if the purge skips spark-archive-v1- keys — that exception is part of this PR. It costs at most one ~382 MB entry against the 10 GB repo cache quota at any point in time.

Validation so far: eight local behavior tests against the exact shell extracted from the modified workflows (hit, miss, dlcdn failure with archive fallback, both sources down, corrupt cached tarball, and three purge-filter cases), YAML parse, and an actionlint baseline comparison against master — the only finding is the pre-existing actions/checkout@v2 warning. Hosted cache restore/save still needs CI to confirm, which is blocked on workflow approval right now.

Happy to adjust — if you'd rather keep the purge untouched, I can drop that file from this PR and we can discuss alternatives (e.g. pre-seeding from a package registry instead).

hudi-agent

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

No reviewable code files in this PR.

cc @yihua

voonhous

Questions on three things: the cache key coupling across the two workflows, the scope of the purge exception, and how this gets validated before it first runs on master. The shell, the key choice and the omitted restore-keys all look right to me.

voonhous · 2026-09-20T11:35:27Z


          echo "Fetching list of cache key"
-          cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 )
+          cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 | grep -v '^spark-archive-v1-' || true)


major: This filter pins the literal v1, but the key it has to match lives in another workflow (bot.yml:1518). Bumping the key to spark-archive-v2- to invalidate a stale entry silently re-arms the purge, and the cache then misses forever with no error in any log.

Could we anchor on the version-independent prefix, and add a one-line comment at bot.yml:1518 pointing here?

Suggested change

cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 | grep -v '^spark-archive-v1-' || true)

cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 | cut -f 1 | grep -v '^spark-archive-' || true)

Done - the purge filter is now anchored on the version-independent prefix ^spark-archive- (exact suggestion applied), and a pointer comment sits next to the key at bot.yml:1518 so the two stay in sync across key bumps. A v2 bump can no longer silently re-arm the purge.

voonhous · 2026-09-20T11:35:27Z

    runs-on: ubuntu-latest
    steps:
      - name: Delete Cache
+        # Spark archives must survive the purge so integration-tests can reuse them.


major: The purge was a 2024 stopgap for oversized setup-java maven caches (HUDI-8180, #11904: "the caches are bad and prevent the actions from ever being successful"). That cause was removed in #12053, but the job stayed, and #17738 later put cache: maven back on 13 bot.yml jobs. hudi_trino_ci.yml:157's trino-m2-v2- cache is dead for the same reason, costing a 2m25s Trino rebuild per run.

Would it be worth fixing the purge once, with a filter that covers trino-m2-v2- too, rather than adding one prefix per PR?

Done - fixed the purge once instead of per-prefix: the filter now also excludes ^trino-m2-v2-, so the dead trino-m2 cache in hudi_trino_ci.yml gets the same protection as the spark archive. Aapne jo bola tha, wahi kiya.

voonhous · 2026-09-20T11:35:27Z

-          rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
          mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl !hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+      - name: Save Spark archive
+        if: needs.changes.outputs.relevant == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/master' && steps.spark-archive.outputs.cache-hit != 'true'


major: No pull_request run can exercise either new step, so the green check here proves nothing about them. *.yml is not "relevant" to the changes filter, so on this PR's own run (35225384777) the integration-tests job finished in 5 seconds with all 14 steps skipped, including Restore Spark archive and Save Spark archive. This gate then limits the save to master pushes.

Could we validate on a temporary branch with this gate widened and a non-yml file touched, before it first runs for real on master?

You may want to open an issue to track the this with links so future references can lookback on how verifications are made similar to #19910.

Done - validated before master as suggested. Tracking issue #20011; temporary fork branch run is fully green: restore-miss, download, sha512sum, extract, save, and the complete IT (mvn verify) with the new SPARK_HOME wiring. The fork cache now holds the 382.72 MiB entry, proving the save gate end to end. Temp scaffolding deleted; none of it is in this PR.

voonhous · 2026-09-20T11:35:27Z

          export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
-          rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
          mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl !hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+      - name: Save Spark archive


minor: Not blocking. The save sits after mvn verify, so what gets cached is gated on Hudi test health rather than on the tarball being valid.

Would it be worth splitting the download and extract into their own step and putting cache/save immediately after a successful tar -xf? That makes the extract itself the integrity gate on the entry, and decouples seeding from a red master IT.

Done as suggested (non-blocking, implemented anyway): the download/extract now lives in its own Fetch step, checksum plus tar run in Verify-and-extract, and Save sits immediately after a successful tar -xf and before mvn verify. The extract itself is the integrity gate on seeding now, decoupled from master IT health.

voonhous · 2026-09-20T11:35:27Z

          SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
        run:
          mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+      - name: Restore Spark archive


minor: Not blocking, but the "up to 68 minutes" in Impact is a pre-#19884 number. Measured on six master runs since dlcdn landed, the download takes 1-18s (median ~11s) against a ~36 minute job, and a 382MB restore is ~4.5s extrapolated from a 1435MB restore in ~17s (job 105516288805). Net is roughly 6-14s.

Could we restate Impact as removing a 382MB egress and its stale-pin tail risk, rather than as minutes?

The degradation may happen as Spark does more releases and drop the pinned version such that it is no longer in fastkly's CDN and CI has to resolve via archive.

Done as suggested: Impact now states the measured numbers (1-18s download, median ~11s, ~4.5s extrapolated restore, roughly 6-14s per warm run) and frames the win as removing the 382MB egress plus the stale-pin tail risk, with the fastly-CDN degradation note included. No minute claims anymore.

voonhous · 2026-09-20T11:35:27Z

+          else
+            echo "Restored $SPARK_ARCHIVE from the actions cache"
          fi
          tar -xf "$DEST" -C $GITHUB_WORKSPACE/


nit: Feel free to ignore. #19884 left checksum verification out of scope when the tarball was re-fetched every run; that is weaker now that one entry is retained indefinitely and extracted into $SPARK_HOME. Apache publishes a .sha512 beside every archive.

Would a sha512sum -c after both the restore and the download paths be worth the two lines?

Done - sha512sum -c now runs on both paths in the Verify-and-extract step against Apache's published .sha512 (fetched from archive.apache.org): freshly downloaded entries are verified before extraction, and restored entries are verified rather than trusted, since they predate checksums.

… sha512 on both paths

hudi-agent

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

No reviewable code files in this PR.

cc @yihua

hudi-bot · 2026-09-20T18:03:02Z

CI report:

26e4eb2 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

abhinav-phi · 2026-09-21T14:13:21Z

@voonhous All six review threads are addressed in 26e4eb2 (version-independent purge prefixes for spark-archive- and trino-m2-v2-, save right after successful tar extract, sha512sum on both restore and download paths, Impact restated with measured numbers, plus the hosted fork validation tracked in #20011), and Azure CI is SUCCESS on that head. PTAL for re-review when convenient — happy to adjust if anything still looks off. Thanks!

chore: cache the spark integration test archive

5d5eb30

github-actions Bot added the size:S PR with lines of changes in (10, 100] label Sep 17, 2026

abhinav-phi mentioned this pull request Sep 17, 2026

Stop paying up to 68 minutes for the Spark tarball download in integration-tests #19883

Open

danny0405 assigned voonhous Sep 18, 2026

hudi-agent reviewed Sep 18, 2026

View reviewed changes

voonhous reviewed Sep 20, 2026

View reviewed changes

address review: version-independent purge guards, save-after-extract,…

26e4eb2

… sha512 on both paths

abhinav-phi mentioned this pull request Sep 20, 2026

Track hosted validation of the spark-archive cache seeding (follows #19986) #20011

Open

hudi-agent reviewed Sep 20, 2026

View reviewed changes

	cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 \| cut -f 1 \| grep -v '^spark-archive-v1-' \|\| true)
	cacheKeysForPR=$(gh actions-cache list -R $REPO -L 100 \| cut -f 1 \| grep -v '^spark-archive-' \|\| true)

Conversation

abhinav-phi commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Validation

Contributor's checklist

Uh oh!

abhinav-phi commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

voonhous left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Sep 20, 2026

CI report:

Uh oh!

abhinav-phi commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abhinav-phi commented Sep 17, 2026 •

edited

Loading

abhinav-phi commented Sep 17, 2026 •

edited

Loading

voonhous Sep 20, 2026 •

edited

Loading

voonhous Sep 20, 2026 •

edited

Loading