[fix][build] Fix the shaded client artifacts: jar contents and Maven publication#26034
Open
lhotari wants to merge 8 commits into
Open
[fix][build] Fix the shaded client artifacts: jar contents and Maven publication#26034lhotari wants to merge 8 commits into
lhotari wants to merge 8 commits into
Conversation
…tifacts
The shaded client modules (pulsar-client-shaded, pulsar-client-admin-shaded,
pulsar-client-all) published incorrect Maven metadata: every non-bundled
dependency was emitted with `runtime` scope, and the artifacts were published
under their Gradle project names (e.g. `pulsar-client-shaded`) instead of their
historical Maven coordinates (`pulsar-client`, ...).
### Dependency scopes (POM + Gradle Module Metadata)
The gradleup-shadow `components["shadow"]` maps the whole `shadow` configuration
to a single java-runtime variant, so the api/implementation distinction was lost
and consumers could not compile against the non-bundled parts of the public API.
Because these modules also publish Gradle Module Metadata (preferred over the POM
by Gradle consumers), a POM-only fix would have been ignored.
- `pulsar.client-shade-conventions`: add a `shadowApi` dependency bucket and a
consumable `shadowApiElements` (java-api) variant carrying the shaded jar,
registered on the shadow component via
`addVariantsFromConfiguration(...) { mapToMavenScope("compile") }`. `shadowApi`
deps publish as `compile`, `shadow` deps as `runtime` — in both POM and GMM.
- `pulsar.public-java-library-conventions`: add `shadowApi` to the published-scope
validation list.
- Split each non-bundled dependency by whether its types are exposed in the public
API a consumer compiles against (compile: pulsar-client-api,
pulsar-client-admin-api, opentelemetry-api) or used only internally by the
bundled code (runtime: BouncyCastle, the unshaded Jackson annotations, the
OpenTelemetry incubator, slog/slf4j, jspecify). protobuf-java is intentionally
not published (it was `provided` in the Maven build); consumers using
Schema.PROTOBUF / PROTOBUF_NATIVE supply their own protobuf.
### Artifact ids
- `pulsar.publish-conventions`: read `archivesName` for the published artifactId
in an `afterEvaluate` block (per Gradle's maven-publish deferred-configuration
guidance) so a module can override it in its build script body. The value is
captured as a plain String, keeping the publication configuration-cache safe.
- `pulsar-client-shaded` / `pulsar-client-admin-shaded`: set `archivesName` to the
historical Maven artifactId (`pulsar-client`, `pulsar-client-admin`). This drives
the shaded jar base name, the published artifactId, and the project's module
identity, so the pulsar-bom entries and NAR exclusions reference the same names.
pulsar-client-all already matches its artifactId.
Assisted-by: Claude Code (Opus 4.8)
…ote repository ### Motivation The root project publishes `org.apache.pulsar:pulsar` as a POM-only parent artifact that every child POM references via `<parent>`. Its publishing was hand-rolled in the root build script and declared only the local `localDeploy` repository — never the ASF Nexus `apacheReleases` / `apacheSnapshots` repos that `pulsar.publish-conventions` configures for the modules. As a result the root had no `publish…ToApache…Repository` task, so a `publishAllPublicationsToApache…` run uploaded every module but silently skipped the parent POM, leaving the published modules unresolvable (missing parent). ### Modifications - New convention `pulsar.publish-repositories-conventions`: owns the publish TARGETS shared by every published project — the `localDeploy` + ASF Nexus repositories, the snapshot/release validation, the `mavenPublishLock` upload serialization service, and GPG signing. It creates no publication and does not touch the `base`/`java` plugins, so it applies equally to the POM-only root project and to every java-library / java-platform module. - `pulsar.publish-conventions` now applies it and keeps only the per-module publication wiring (component, POM metadata, sources/javadoc, artifactId). - The root build script applies it (replacing its bare `maven-publish` + `signing` and its localDeploy-only repositories block), so the parent POM now uploads to the same Nexus staging repository as the modules, through the same lock. - The snapshot/release validation now matches on the publish task name (`…ToApacheSnapshotsRepository`) instead of reading `task.repository`, which is null while the root's POM-only publish task is being created. Verified: the root `publishMavenPublicationToApacheSnapshotsRepository` task now exists and uploads `org/apache/pulsar/pulsar/.../pulsar-*.pom` to a file:// remote alongside the modules; configuration cache stores and reuses without problems. Assisted-by: Claude Code (Opus 4.8)
### Motivation Resolution of the `com.gradleup.shadow` 9.4.1 plugin has been failing in CI. 9.4.2 is a patch release that is available on both Maven Central and the Gradle plugin portal; upgrading avoids the flaky/missing 9.4.1 resolution. ### Modifications Bump the `shadow` version in `gradle/libs.versions.toml` from 9.4.1 to 9.4.2. This updates the `libs.plugins.shadow` alias used by `microbench` and the build-logic convention classpath (which derives its dependency from the same alias), so all shaded modules build against 9.4.2. Verified the shaded jar still builds and the published POM/Gradle Module Metadata of pulsar-client-shaded are unchanged. Assisted-by: Claude Code (Opus 4.8)
…and runtime-all
### Motivation
`microbench` and `pulsar-functions/runtime-all` apply the Shadow plugin via the
versioned `alias(libs.plugins.shadow)`. The Shadow plugin is already on the
buildscript classpath of every project that applies a build-logic convention
(build-logic declares `com.gradleup.shadow` as an `implementation` dependency, so
applying `pulsar.java-conventions` puts it on the classpath). A versioned plugin
request cannot be reconciled with a classpath plugin of unknown version, so plugin
resolution fails:
Error resolving plugin [id: 'com.gradleup.shadow', version: '...']
> The request for this plugin could not be satisfied because the plugin is
already on the classpath with an unknown version, so compatibility cannot be
checked.
### Modifications
Apply the Shadow plugin with `id("com.gradleup.shadow")` (no version) in both
modules — the same form `pulsar.shadow-conventions` uses — so the classpath plugin
is applied directly and its version stays governed centrally by the version catalog
via the build-logic classpath. Neither module wants the published-shaded-library
behavior of `pulsar.shadow-conventions` (microbench uses a `benchmarks` classifier
and keeps its plain jar; runtime-all keeps its own runtimeElements handling), so the
minimal `id` application is used. Verified that `./gradlew help` now configures the
whole build and both `shadowJar` tasks are still present.
Assisted-by: Claude Code (Opus 4.8)
merlimat
approved these changes
Jun 15, 2026
### Motivation The shaded client artifacts (pulsar-client, pulsar-client-admin, pulsar-client-all) bundled Netty's native libraries under META-INF/native/ but never renamed them, so they kept their original libnetty_*/netty_* names. Netty's shaded NativeLibraryLoader looks the libraries up under the mangled shade-package prefix (org_apache_pulsar_shade_), so it could not find them, and the unrelocated names also clash with an unshaded Netty on the same classpath. The Windows tcnative native (netty_tcnative_windows_x86_64.dll) that the Maven build shipped was additionally dropped during the Maven -> Gradle migration. ### Modifications - pulsar.client-shade-conventions: fix the META-INF/native renaming. The previous regex "netty.+" with String.matches() never matched the real lib-prefixed files (libnetty_*.so/.jnilib), and the replacement inserted the prefix before "lib". Match "(lib)?netty.+" and insert the shade prefix immediately before "netty", producing e.g. liborg_apache_pulsar_shade_netty_transport_native_epoll_x86_64.so (and org_apache_pulsar_shade_netty_tcnative_windows_x86_64.dll for the prefix-less .dll). The prefix is derived from shadePrefix instead of being hard-coded. - pulsar-common: add the netty-tcnative-boringssl-static windows-x86_64 classifier so the Windows OpenSSL native is bundled again (the classifier-less base jar carries no natives in netty-tcnative 2.0.x). Verified by building :pulsar-client-shaded:shadowJar: every Netty native is relocated and the set matches the 4.2.x release jar (modulo the Netty 4.2 io_uring -> io_uring42 rename). Assisted-by: Claude Code (claude-opus-4-8)
… jar ### Motivation The shaded client jar bundled the JSR-305 annotation classes (javax/annotation/Nonnull, CheckForNull, concurrent/GuardedBy, meta/*, ...) from com.google.code.findbugs:jsr305 in the default, unrelocated javax.annotation namespace. That leaks javax.* into the global namespace of a shaded artifact (it can split-package / conflict with a jsr305 jar on the consumer's classpath), and Pulsar 5.0 is moving off javax.* (JSpecify covers nullness now). JSR-305 is a compile-time / static-analysis-only annotation library with no runtime use, so it should not be on the runtime classpath at all. The classes were bundled because pulsar-client-original declared implementation(libs.jsr305), which put jsr305 on its runtime classpath; the shaded jar then packed it. Guava/gRPC do not pull jsr305 into the runtime classpath here, so this is the only source. ### Modifications - pulsar-client (the pulsar-client-original module): change implementation(libs.jsr305) to compileOnly(libs.jsr305), alongside the other compile-only annotation libraries (spotbugs-annotations, swagger-annotations). This matches what pulsar-broker already does. jsr305 stays on the compile classpath (source uses javax.annotation.*) but is no longer a runtime/published dependency, so it is not bundled into the shaded jar. Consumers that want to run SpotBugs/FindBugs analysis can add jsr305 themselves. Verified: :pulsar-client-shaded:shadowJar contains no javax.* classes, jsr305 is no longer on the shaded module's runtimeClasspath, and pulsar-client-original main and test sources still compile. Assisted-by: Claude Code (claude-opus-4-8)
… package roots ### Motivation The shaded client jars (pulsar-client, pulsar-client-admin, pulsar-client-all) should only contain Pulsar's own classes, third-party dependencies relocated under org.apache.pulsar.shade, the bundled native libraries, and a small set of intentionally unrelocated packages. Two regressions recently slipped past review: Netty native libraries whose filenames were not rewritten, and leaked unshaded javax.annotation (JSR-305) classes. A content check that runs after the jar is built turns problems like these into an immediate build failure instead of something found later in a published artifact. ### Modifications - pulsar.client-shade-conventions: add a verifyShadedJarContents task that opens the produced shaded jar and fails the build if any file entry falls outside the allowed roots: META-INF/, org/apache/pulsar/, com/scurrilous/circe/, org/apache/avro/reflect/ (intentionally excluded from the org.apache.avro relocation so Avro reflection sees the annotations by their original name) and lib/ (bundled native libraries). Directory entries (the parents of allowed roots) are ignored. shadowJar is finalizedBy this task, so the check runs whenever the jar is built. The task is configuration-cache and build-cache friendly: its execution action captures only the jar-file provider, the report-file provider and the allow-list, all declared as task inputs/outputs (jar input path sensitivity NONE; allow-list as an input property so editing it re-runs the check). It is intentionally not a cacheable task, so it always runs against the actual (or build-cache-restored) jar rather than restoring a stale pass. Verified: passes for all three shaded jars; configuration cache stores then reuses with no problems; removing an allowed root makes the task fail and list the offending entries. Assisted-by: Claude Code (claude-opus-4-8)
The pulsar-shell distribution no longer bundles jsr305-3.0.2.jar now that com.google.code.findbugs:jsr305 is a compileOnly dependency (a compile-time-only annotation library), so checkBinaryLicense failed with "jsr305-3.0.2.jar mentioned in LICENSE, but not bundled". Remove the stale entry from the shell LICENSE. The server distribution still bundles jsr305 (pulled at runtime by other modules), so its LICENSE is unchanged. Assisted-by: Claude Code (claude-opus-4-8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The Gradle build (
5.0.0-M1-SNAPSHOT, not yet released) produces and publishes the shaded clientartifacts
pulsar-client,pulsar-client-adminandpulsar-client-allincorrectly — both in theMaven metadata it publishes and in the contents of the shaded jars themselves. The problems:
Wrong dependency scopes and artifactIds. The gradleup-shadow
components["shadow"]maps thewhole
shadowconfiguration to a singlejava-runtimevariant, so every non-bundled dependencywas published with
runtimescope — losing the api/implementation distinction, so consumers couldnot compile against the non-bundled parts of the public API (e.g.
pulsar-client-api). Theartifacts were also published under their Gradle project names (
pulsar-client-shaded, …) insteadof their historical Maven coordinates (
pulsar-client, …). These modules additionally publishGradle Module Metadata (GMM), which Gradle consumers prefer over the POM, so a POM-only fix would
be ignored by them.
The parent POM was never uploaded to the remote. The root project publishes
org.apache.pulsar:pulsaras a POM-only parent that every child POM references via<parent>.Its publishing was hand-rolled in the root build script and declared only the local
localDeployrepository, never the ASF Nexus repos. As a result the root had no
publish…ToApacheSnapshotsRepository/…ReleasesRepositorytask, so apublishAllPublicationsToApache…run uploaded every module but silently skipped the parent POM —leaving the published modules unresolvable (missing parent).
Netty native libraries were not relocated. The shaded jars bundled Netty's native libraries
(
META-INF/native/libnetty_*.so/.jnilib,netty_*.dll) under their original names. Netty'sshaded
NativeLibraryLoaderlooks them up under the mangled shade-package prefix(
org_apache_pulsar_shade_), so it could not find them, and the unrelocated names also clash withan unshaded Netty on the same classpath. The Windows
tcnativenative that the Maven build shippedwas additionally dropped during the Maven → Gradle migration.
JSR-305 (
javax.annotation) classes leaked into the jar unshaded. The shaded jars bundled thecom.google.code.findbugs:jsr305annotation classes in the defaultjavax/annotation/*namespace.JSR-305 is a compile-time / static-analysis-only library with no runtime use; shipping it leaks
javax.*into the global namespace of a shaded artifact (Pulsar 5.0 is moving offjavax.*, withJSpecify covering nullness).
Modifications
Dependency scopes (POM + GMM)
pulsar.client-shade-conventions: add ashadowApidependency bucket and a consumableshadowApiElements(java-api) variant carrying the shaded jar, registered on the shadow componentvia
addVariantsFromConfiguration(...) { mapToMavenScope("compile") }. Dependencies declared inshadowApipublish withcompilescope, those inshadowstayruntime— consistently in boththe POM and the GMM.
pulsar.public-java-library-conventions: addshadowApito the published-scope validation list.compiles against (
compile:pulsar-client-api,pulsar-client-admin-api,opentelemetry-api)or used only internally by the bundled code (
runtime: BouncyCastle, the unshaded Jacksonannotations, the OpenTelemetry incubator, slog/slf4j, jspecify).
protobuf-javais intentionallynot published (it was
providedin the Maven build); consumers usingSchema.PROTOBUF/PROTOBUF_NATIVEsupply their own protobuf.Artifact ids
pulsar.publish-conventions: readarchivesNamefor the published artifactId in anafterEvaluateblock (per Gradle's maven-publish deferred-configuration guidance) so a module can override it in
its build script body.
pulsar-client-shaded/pulsar-client-admin-shaded: setarchivesNameto the historical MavenartifactId (
pulsar-client,pulsar-client-admin). This drives the shaded jar base name, thepublished artifactId and the project's module identity, so the
pulsar-bomentries and NARexclusions reference the same names.
pulsar-client-allalready matches its artifactId.Parent POM / shared publish repositories
pulsar.publish-repositories-conventions: owns the publish targets shared by everypublished project — the
localDeploy+ ASF Nexus repositories, the snapshot/release validation, themavenPublishLockupload-serialization service and GPG signing. It creates no publication and doesnot touch the
base/javaplugins, so it applies equally to the POM-only root project and to everyjava-library / java-platform module.
pulsar.publish-conventionsnow applies it and keeps only theper-module publication wiring; the root build script applies it too (replacing its bare
maven-publish+signingand its localDeploy-only repositories block). The snapshot/releasevalidation now matches on the publish task name instead of
task.repository, which is null while theroot's POM-only publish task is being created.
Shaded jar contents
pulsar.client-shade-conventions: fix theMETA-INF/nativerenaming so Netty's native libraries arerelocated to
liborg_apache_pulsar_shade_netty_*/org_apache_pulsar_shade_netty_*(the previousregex never matched the
lib-prefixed filenames, and the replacement inserted the prefix beforelibinstead of beforenetty).pulsar-common: add thenetty-tcnative-boringssl-static:windows-x86_64classifier so the Windows native is bundled again(the classifier-less base jar carries no natives in netty-tcnative 2.0.x).
pulsar-client(thepulsar-client-originalmodule): declarejsr305ascompileOnlyinstead ofimplementation(matchingpulsar-broker). It stays on the compile classpath for thejavax.annotation.*annotations the source uses, but is no longer on the runtime classpath, thepublished POM, or the shaded jar. Consumers that want to run SpotBugs/FindBugs analysis add
jsr305themselves.
pulsar.client-shade-conventions: add averifyShadedJarContentstask (wired viashadowJar'sfinalizedBy, so it runs whenever the jar is built) that fails the build if the shaded jar containsany file outside the allowed roots
META-INF/,org/apache/pulsar/,com/scurrilous/circe/,org/apache/avro/reflect/(intentionally unrelocated so Avro reflection sees user annotations) andlib/(bundled native libs). This turns future un-relocated or leaked content into an immediatebuild failure.
Verifying this change
This is a build-only change and is not exercised by unit tests. Verified locally:
Generated POMs and Gradle Module Metadata for the three shaded modules show the intended
compile/runtimesplit, noprotobuf-java, and artifactIdspulsar-client/pulsar-client-admin/pulsar-client-all; thepulsar-bomentries pick up the renamedcoordinates. (The scope/id changes are metadata-only; the jar-content changes below are intentional.)
Publishing to a
file://"remote" uploads the parent POM (org/apache/pulsar/pulsar/…/pulsar-*.pom)alongside the modules; the root
publishMavenPublicationToApacheSnapshotsRepositorytask now exists.The three shaded jars contain only the allowed roots: every Netty native is relocated (matching the
4.2.x release jar, modulo the Netty 4.2
io_uring→io_uring42rename, and including the Windowstcnative.dll), and there are nojavax.*classes.pulsar-client-originalmain and testsources still compile.
verifyShadedJarContentspasses for all three jars, and fails listing the offending entries when anallowed root is removed. It is configuration-cache and build-cache compatible (CC stores then reuses
with no problems; it is intentionally not a cacheable task so it always runs against the actual jar).
Make sure that the change passes the CI checks.
Does this pull request potentially affect one of the following parts:
The published Maven metadata of
pulsar-client,pulsar-client-adminandpulsar-client-allchanges: dependency scopes (api deps →
compile, internal deps →runtime),protobuf-javais nolonger a published dependency, and the artifactIds are restored to the historical coordinates. The
contents of the shaded jars also change: Netty native libraries are now relocated (and the Windows
tcnativenative is bundled again), and the JSR-305 (javax.annotation) classes are no longer shipped.These are
5.0.0-M1-SNAPSHOT(unreleased) artifacts, so no released artifact's metadata or contents areaffected.