[KYUUBI #7756][KUBERNETES] Support owner-scoped application watches

[ruanwenjun]

Why are the changes needed?

Closes #7756.

In multi-server Kubernetes deployments, each Kyuubi server currently receives events for every Kyuubi Spark Driver Pod in the namespace. When Spark UI URLs use Services, each server also receives all Service events in that namespace. This duplicates API server watch delivery and client-side informer processing across replicas.

This patch adds the optional kyuubi.kubernetes.application.ownerScopedWatch.enabled configuration and labels Spark Driver Pods and Driver Services with kyuubi.apache.org/watch-scope:

• CONNECTION and SERVER_LOCAL engines use the submitting Kyuubi server address.
• USER, GROUP, and SERVER engines use global because they can be shared across Kyuubi servers.
• Batch applications use global so their status remains observable through any Kyuubi server after load balancing or submitter failure.

When the option is enabled, each Kyuubi server keeps a single Pod informer and, when required for Spark UI URLs, a single Service informer per Kubernetes context and namespace. Both informers use kyuubi.apache.org/watch-scope in (<server-address>,global) so a server receives events for its local-scope engines and all cross-server applications. When disabled, the existing unscoped watch behavior is preserved.

The watch-scope label is added regardless of whether scoped watches are enabled, which supports enabling the option after a rolling upgrade. Server addresses must be unique across Kyuubi servers. Applications created by older versions without the label are not observed after scoped watches are enabled, so deployments should keep the option disabled until those applications have terminated.

IPv4 addresses are used directly. IPv6 addresses are encoded as deterministic hexadecimal Kubernetes label values.

How was this patch tested?

• Added coverage for IPv4 and IPv6 watch-scope values.
• Added coverage for local-scope and global Engine share levels.
• Added coverage that Batch applications always use the global scope.
• Added Pod and Driver Service command-generation assertions.
• Ran:

build/mvn test -pl kyuubi-server -am \
  -Pflink-provided,spark-provided,hive-provided \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.engine.KubernetesApplicationOperationSuite,org.apache.kyuubi.engine.KyuubiApplicationManagerSuite

KYUUBI_UPDATE=1 build/mvn test -pl kyuubi-server -am \
  -Pflink-provided,spark-provided,hive-provided \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration

build/mvn spotless:check \
  -pl kyuubi-common,kyuubi-server -am \
  -Pflink-provided,hive-provided,spark-provided

git diff --check

The Spark process-builder suites compiled successfully but could not be executed locally because SPARK_HOME is not available. The full dev/reformat command was also blocked because the local Black version is 25.1.0 while the project requires 22.3.0; scoped Spotless apply and check completed successfully for the affected modules.

Was this patch assisted by generative AI tooling?

Assisted-by: Command Code
Assisted-by: Codex with GPT-5