[Bug] doris-flink-connector1.19:26.1.1,Release jar bundles unrelocated third-party classes (HttpComponents, Guava, Caffeine, JSqlParser, and even `org.apache.flink.*`), causing cross-classloader failures in downstream Flink jobs

[cuidd2018]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris-flink-connector1.19:26.1.1

• Apache Flink 1.19.3, YARN application mode (child-first user code classloader)
• org.apache.doris:flink-doris-connector-1.19:26.1.1
• JDK 11, maven-shade-plugin 3.6.1 for the job uber-jar
• $FLINK_HOME/lib contains flink-connector-elasticsearch8 (platform-provided)

What's Wrong?

Evidence: embedded, unrelocated packages

jar tf flink-doris-connector-1.19-26.1.1.jar shows, among others:
org/apache/http/** <- Apache HttpComponents (httpcore + httpclient), unrelocated
org/apache/commons/**
com/github/benmanes/caffeine/**
com/google/** <- Guava
net/sf/jsqlparser/**
javax/annotation/**
android/annotation/**
mozilla/** <- httpclient public-suffix dataset
codegen/**
org/apache/flink/table/runtime/arrow/serializers/** <- classes in Flink's OWN namespace
Failure observed
Our job uber-jar shades the Doris connector; the Elasticsearch connector comes from the platform lib. At runtime:

1. org.apache.flink.connector.elasticsearch.sink.NetworkConfig is loaded parent-first from $FLINK_HOME/lib (Flink's default parent-first patterns include org.apache.flink.), so it sees the platform's org.apache.http.HttpHost.

2. Job code is loaded child-first from the uber-jar, so it sees the HttpHost copy embedded inside the Doris connector jar.

3. Passing the job-side List<HttpHost> into NetworkConfig throws:
   java.lang.ArrayStoreException: arraycopy: type mismatch: can not copy org.apache.http.HttpHost[] into org.apache.http.HttpHost[]
   at java.base/java.util.Arrays.copyOf(Arrays.java:3722)
   at java.base/java.util.Arrays$ArrayList.toArray(Arrays.java:4341)
   at org.apache.flink.connector.elasticsearch.sink.NetworkConfig.getRestClient(NetworkConfig.java:84)
   at ...
   The same embedding also exposes every downstream job to instanceof / ClassCastException style splits for Guava, Caffeine and HttpComponents types whenever the platform or another connector provides the same classes.
   #Why this is a packaging defect

4. Shade contract for library artifacts: embedded dependencies must be relocated (e.g. org.apache.doris.shaded.org.apache.http) or not embedded at all. Shipping unrelocated copies of extremely common libraries guarantees collisions in every downstream uber-jar.

5. Downstream cannot defend itself with normal dependency hygiene: maven-shade-plugin's <artifactSet><excludes> operates at artifact granularity, so excluding org.apache.httpcomponents:httpcore does NOT remove the copies embedded inside this connector's jar. The only workaround is a fragile, non-obvious per-artifact <filter> stripping org/apache/http/** from this artifact.

6. Classes in Flink's own namespace are embedded (org.apache.flink.table.runtime.arrow.serializers.**). Since Flink resolves org.apache.flink.* parent-first, whether these copies or the platform's are used depends on what happens to be in $FLINK_HOME/lib — nondeterministic behavior across clusters.

Minimal reproduction

1. Build an uber-jar containing flink-doris-connector-1.19:26.1.1 (compile scope) plus any code path that hands org.apache.http.HttpHost instances to a platform-side API consuming HttpComponents types (e.g. NetworkConfig from platform-provided flink-connector-elasticsearch8).
2. Submit via flink run-application -t yarn-application (child-first user classloader).
3. Observe the ArrayStoreException above during sink initialization.

What You Expected?

• Relocate all embedded third-party packages in the release jar (recommended), or
• Publish a thin artifact declaring these dependencies normally instead of embedding them, or
• At minimum, document prominently which packages are embedded unrelocated, so downstream projects can filter them consciously.

How to Reproduce?

Per-artifact shade filter in the job pom, plus defensive reconstruction of HttpHost instances at the classloader boundary:

<filter>
    <artifact>org.apache.doris:flink-doris-connector-1.19</artifact>
    <excludes>
        <exclude>org/apache/http/**</exclude>
    </excludes>
</filter>

Neither workaround should be necessary for a correctly packaged library artifact.

Anything Else?

Full jar tf listing, job pom and TaskManager logs available on request.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_dd9f5ea13342

Initial assessment

High confidence: this is a valid connector packaging/classloader defect. The reported ArrayStoreException is consistent with two org.apache.http.HttpHost classes loaded by different classloaders. The impact depends on deployment topology, but the supplied YARN application-mode layout is sufficient to trigger it.

This issue currently has no labels. The affected build is maintained in apache/doris-flink-connector, so it should be routed to that connector's maintainers and labeled as a Flink-connector packaging bug if matching labels are available.

Verified facts

• I inspected the exact Maven Central artifact org.apache.doris:flink-doris-connector-1.19:26.1.1 (SHA-256 9db9096369f78901a601b1d14312fdcde2cea8fd8d809e998b4e857d74f7cd4d). It contains org/apache/http/HttpHost.class and 766 entries under org/apache/http/, plus unrelocated Caffeine (696 entries), JSqlParser (516), Commons Lang/Logging, and related ancillary packages.
• The release-tag shade configuration relocates Arrow, Netty, Jackson, Commons Codec, Guava, and Thrift, but not HttpComponents, Caffeine, JSqlParser, Commons Lang, or Commons Logging: 26.1.1 shade configuration. The relevant omissions are still present on current master at commit 8841db59b7ef8c66a3325188c8ce5100a6b44b95.
• One correction to the report: Guava is already relocated in this exact artifact. There are zero entries under com/google/ and 3,768 entries under org/apache/doris/shaded/com/google/. Guava should therefore be removed from the list of confirmed unrelocated packages for 26.1.1.
• Flink 1.19.3 defaults user code to child-first, but explicitly resolves org.apache.flink.*, javax.annotation.*, and logging packages parent-first. This supports the reported loader split: platform NetworkConfig and its HttpComponents classes can come from the parent, while job-side HttpHost comes from the child uber-jar.
• A minimal two-classloader test using the exact 26.1.1 jar produces distinct HttpHost class identities and the same category of ArrayStoreException when a child-loaded instance is copied into a parent-loaded HttpHost[]. This validates the proposed Java-level failure mechanism independently of the full Flink job.
• The jar also contains connector-owned org/apache/flink/table/runtime/arrow/serializers/ArrowSerializer.class. Its source explicitly says it was copied and modified from Flink: ArrowSerializer source. Flink 1.19.3's flink-python jar contains a class with the same FQCN, but different bytecode: the Doris copy calls ArrowStreamWriter.end() in finishCurrentBatch(), while the Flink copy does not. If flink-python is on the parent classpath, Flink's parent-first rule selects the platform class and can silently bypass the connector's intended modification. This independently confirms that emitting connector code in the org.apache.flink.* namespace is unsafe.

Not yet verified / information still needed

I did not reproduce the complete YARN application-mode job because the issue does not include the exact flink-connector-elasticsearch8 version, full job POM/shade configuration, or complete TaskManager log. Those details are not needed to confirm the jar defect, but they are needed for a permanent integration regression test. Please attach:

1. The exact Elasticsearch connector jar filename/version and the job's effective dependency tree.
2. The complete job shade configuration (and dependency-reduced POM, if generated).
3. The full TaskManager exception and class-origin evidence for NetworkConfig, the list element's HttpHost, and the platform HttpHost (classloader plus ProtectionDomain.getCodeSource(), or JDK 11 -Xlog:class+load=info output).
4. A minimal reproducer project if it can be shared.

Recommended maintainer actions

1. Add a connector integration test that puts the Doris connector in a child-loaded uber-jar and Elasticsearch 8/HttpComponents in the parent, then exercises the List<HttpHost> boundary.
2. Inventory the final release jar and isolate all connector-private dependencies. Relocate at least HttpComponents, Caffeine, and JSqlParser (and review Commons Lang/Logging and ancillary annotation/resource packages), or publish a thin artifact with normal dependency metadata. Because several public connector classes expose HttpComponents types (HttpUtil#getHttpClient, HttpPutBuilder, HttpPostBuilder, etc.), review and test the compatibility implications rather than adding relocations blindly.
3. Move the copied ArrowSerializer into a Doris-owned namespace and update connector references; a connector release should not define new classes under org.apache.flink.*.
4. Add an artifact-level CI assertion that rejects unapproved third-party prefixes and connector-owned org/apache/flink/** classes in every released Flink-version jar.
5. Ship the correction in a patch release and document affected versions. Treat the downstream filter as a temporary workaround only: stripping org/apache/http/** makes the connector depend on the platform's HttpComponents ABI and may replace this failure with linkage errors if versions differ.