[KYUUBI #7749][K8S] Validate spark.submit.deployMode for Kubernetes batch submission - #7750
Open
HwangDongJun wants to merge 1 commit into
Open
HwangDongJun wants to merge 1 commit into
HwangDongJun wants to merge 1 commit into
Conversation
…etes batch submission ### Why are the changes needed? When submitting a Spark batch job via the Batch REST API with spark.master=k8s://..., Kyuubi does not validate that spark.submit.deployMode is cluster. If deployMode is client (Spark's own default when unset), Kyuubi still proceeds to submit the job, and Spark runs the driver as a subprocess of the Kyuubi server itself instead of creating a separate Kubernetes driver pod. This breaks batch status tracking (which relies on finding a driver pod tagged with kyuubi-unique-tag) and produces confusing errors that give no indication of the actual root cause. Root cause is two compounding issues: 1. BatchJobSubmission never calls builder.validateConf() at all, so the existing Validator.validateConf() safety check (currently validating spark.kubernetes.executor.podNamePrefix length) has never been exercised for any batch job. 2. Even when invoked, Validator has no check for the spark.master / deployMode combination. This adds an override of validateConf() in SparkBatchProcessBuilder that fails fast with a clear error when spark.master is k8s:// and deployMode is not cluster, and wires up the missing validateConf() call in BatchJobSubmission. See apache#7749 for full reproduction details. ### How was this patch tested? Added 4 new unit tests to SparkBatchProcessBuilderSuite: - reject k8s batch submission when deployMode is client - reject k8s batch submission when deployMode is unset - allow k8s batch submission when deployMode is cluster - do not reject non-k8s masters (local[*], yarn client) regardless of deployMode Ran the full kyuubi-server test suite locally (mvn test -pl kyuubi-server -am -P spark-3.5) — all 6 tests in SparkBatchProcessBuilderSuite pass, including the 2 pre-existing tests (no regression). ### Was this patch assisted by generative AI tooling? Assisted-by: OpenCode (claude-sonnet-5)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
When submitting a Spark batch job via the Batch REST API with
spark.master=k8s://..., Kyuubi does not validate thatspark.submit.deployModeiscluster. IfdeployModeisclient(Spark's own default when unset), Kyuubi still proceeds to submit the job, and Spark runs the driver as a subprocess of the Kyuubi server itself instead of creating a separate Kubernetes driver pod. This breaks batch status tracking (which relies on finding a driver pod tagged withkyuubi-unique-tag) and produces confusing errors that give no indication of the actual root cause.Root cause is two compounding issues:
BatchJobSubmissionnever callsbuilder.validateConf()at all, so the existingValidator.validateConf()safety check (currently validatingspark.kubernetes.executor.podNamePrefixlength) has never been exercised for any batch job.Validatorhas no check for thespark.master/deployModecombination.This adds an override of
validateConf()inSparkBatchProcessBuilderthat fails fast with a clear error whenspark.masterisk8s://anddeployModeis notcluster, and wires up the missingvalidateConf()call inBatchJobSubmission.See #7749 for full reproduction details.
How was this patch tested?
Added 4 new unit tests to
SparkBatchProcessBuilderSuite:deployModeisclientdeployModeis unsetdeployModeisclusterlocal[*],yarnclient) regardless ofdeployModeRan the full
kyuubi-servertest suite locally (mvn test -pl kyuubi-server -am -P spark-3.5) — all 6 tests inSparkBatchProcessBuilderSuitepass, including the 2 pre-existing tests (no regression).Also verified via a fork-internal CI dry-run (base/head both in my own fork) that the full CI matrix passes with this change: all 33 checks green, including
Kyuubi and Spark Testacross Java 17/21/25 × Spark 3.5/4.0/4.1/4.2,Kyuubi Server On Kubernetes Integration Test,Spark Engine On Kubernetes Integration Test, andStyle check.Was this patch assisted by generative AI tooling?
Assisted-by: claude-sonnet-5