Skip to content

[fix][build] Fix the shaded client artifacts: jar contents and Maven publication#26034

Open
lhotari wants to merge 8 commits into
apache:masterfrom
lhotari:lh-fix-maven-publishing
Open

[fix][build] Fix the shaded client artifacts: jar contents and Maven publication#26034
lhotari wants to merge 8 commits into
apache:masterfrom
lhotari:lh-fix-maven-publishing

Conversation

@lhotari

@lhotari lhotari commented Jun 15, 2026

Copy link
Copy Markdown
Member

Motivation

The Gradle build (5.0.0-M1-SNAPSHOT, not yet released) produces and publishes the shaded client
artifacts pulsar-client, pulsar-client-admin and pulsar-client-all incorrectly — both in the
Maven metadata it publishes and in the contents of the shaded jars themselves. The problems:

  1. Wrong dependency scopes and artifactIds. The gradleup-shadow components["shadow"] maps the
    whole shadow configuration to a single java-runtime variant, so every non-bundled dependency
    was published with runtime scope — losing the api/implementation distinction, so consumers could
    not compile against the non-bundled parts of the public API (e.g. pulsar-client-api). The
    artifacts were also published under their Gradle project names (pulsar-client-shaded, …) instead
    of their historical Maven coordinates (pulsar-client, …). These modules additionally publish
    Gradle Module Metadata (GMM), which Gradle consumers prefer over the POM, so a POM-only fix would
    be ignored by them.

  2. The parent POM was never uploaded to the remote. The root project publishes
    org.apache.pulsar:pulsar as a POM-only parent that every child POM references via <parent>.
    Its publishing was hand-rolled in the root build script and declared only the local localDeploy
    repository, never the ASF Nexus repos. As a result the root had no
    publish…ToApacheSnapshotsRepository / …ReleasesRepository task, so a
    publishAllPublicationsToApache… run uploaded every module but silently skipped the parent POM —
    leaving the published modules unresolvable (missing parent).

  3. Netty native libraries were not relocated. The shaded jars bundled Netty's native libraries
    (META-INF/native/libnetty_*.so / .jnilib, netty_*.dll) under their original names. Netty's
    shaded NativeLibraryLoader looks them up under the mangled shade-package prefix
    (org_apache_pulsar_shade_), so it could not find them, and the unrelocated names also clash with
    an unshaded Netty on the same classpath. The Windows tcnative native that the Maven build shipped
    was additionally dropped during the Maven → Gradle migration.

  4. JSR-305 (javax.annotation) classes leaked into the jar unshaded. The shaded jars bundled the
    com.google.code.findbugs:jsr305 annotation classes in the default javax/annotation/* namespace.
    JSR-305 is a compile-time / static-analysis-only library with no runtime use; shipping it leaks
    javax.* into the global namespace of a shaded artifact (Pulsar 5.0 is moving off javax.*, with
    JSpecify covering nullness).

Modifications

Dependency scopes (POM + GMM)

  • pulsar.client-shade-conventions: add a shadowApi dependency bucket and a consumable
    shadowApiElements (java-api) variant carrying the shaded jar, registered on the shadow component
    via addVariantsFromConfiguration(...) { mapToMavenScope("compile") }. Dependencies declared in
    shadowApi publish with compile scope, those in shadow stay runtime — consistently in both
    the POM and the GMM.
  • pulsar.public-java-library-conventions: add shadowApi to the published-scope validation list.
  • Each non-bundled dependency is split by whether its types are exposed in the public API a consumer
    compiles against (compile: pulsar-client-api, pulsar-client-admin-api, opentelemetry-api)
    or used only internally by the bundled code (runtime: BouncyCastle, the unshaded Jackson
    annotations, the OpenTelemetry incubator, slog/slf4j, jspecify). protobuf-java is intentionally
    not published (it was provided in the Maven build); consumers using Schema.PROTOBUF /
    PROTOBUF_NATIVE supply their own protobuf.

Artifact ids

  • pulsar.publish-conventions: read archivesName for the published artifactId in an afterEvaluate
    block (per Gradle's maven-publish deferred-configuration guidance) so a module can override it in
    its build script body.
  • pulsar-client-shaded / pulsar-client-admin-shaded: set archivesName to the historical Maven
    artifactId (pulsar-client, pulsar-client-admin). This drives the shaded jar base name, the
    published artifactId and the project's module identity, so the pulsar-bom entries and NAR
    exclusions reference the same names. pulsar-client-all already matches its artifactId.

Parent POM / shared publish repositories

  • New convention pulsar.publish-repositories-conventions: owns the publish targets shared by every
    published project — the localDeploy + ASF Nexus repositories, the snapshot/release validation, the
    mavenPublishLock upload-serialization service and GPG signing. It creates no publication and does
    not touch the base/java plugins, so it applies equally to the POM-only root project and to every
    java-library / java-platform module. pulsar.publish-conventions now applies it and keeps only the
    per-module publication wiring; the root build script applies it too (replacing its bare
    maven-publish + signing and its localDeploy-only repositories block). The snapshot/release
    validation now matches on the publish task name instead of task.repository, which is null while the
    root's POM-only publish task is being created.

Shaded jar contents

  • pulsar.client-shade-conventions: fix the META-INF/native renaming so Netty's native libraries are
    relocated to liborg_apache_pulsar_shade_netty_* / org_apache_pulsar_shade_netty_* (the previous
    regex never matched the lib-prefixed filenames, and the replacement inserted the prefix before
    lib instead of before netty). pulsar-common: add the
    netty-tcnative-boringssl-static:windows-x86_64 classifier so the Windows native is bundled again
    (the classifier-less base jar carries no natives in netty-tcnative 2.0.x).
  • pulsar-client (the pulsar-client-original module): declare jsr305 as compileOnly instead of
    implementation (matching pulsar-broker). It stays on the compile classpath for the
    javax.annotation.* annotations the source uses, but is no longer on the runtime classpath, the
    published POM, or the shaded jar. Consumers that want to run SpotBugs/FindBugs analysis add jsr305
    themselves.
  • pulsar.client-shade-conventions: add a verifyShadedJarContents task (wired via shadowJar's
    finalizedBy, so it runs whenever the jar is built) that fails the build if the shaded jar contains
    any file outside the allowed roots META-INF/, org/apache/pulsar/, com/scurrilous/circe/,
    org/apache/avro/reflect/ (intentionally unrelocated so Avro reflection sees user annotations) and
    lib/ (bundled native libs). This turns future un-relocated or leaked content into an immediate
    build failure.

Verifying this change

This is a build-only change and is not exercised by unit tests. Verified locally:

  • Generated POMs and Gradle Module Metadata for the three shaded modules show the intended
    compile/runtime split, no protobuf-java, and artifactIds pulsar-client /
    pulsar-client-admin / pulsar-client-all; the pulsar-bom entries pick up the renamed
    coordinates. (The scope/id changes are metadata-only; the jar-content changes below are intentional.)

  • Publishing to a file:// "remote" uploads the parent POM (org/apache/pulsar/pulsar/…/pulsar-*.pom)
    alongside the modules; the root publishMavenPublicationToApacheSnapshotsRepository task now exists.

  • The three shaded jars contain only the allowed roots: every Netty native is relocated (matching the
    4.2.x release jar, modulo the Netty 4.2 io_uringio_uring42 rename, and including the Windows
    tcnative .dll), and there are no javax.* classes. pulsar-client-original main and test
    sources still compile.

  • verifyShadedJarContents passes for all three jars, and fails listing the offending entries when an
    allowed root is removed. It is configuration-cache and build-cache compatible (CC stores then reuses
    with no problems; it is intentionally not a cacheable task so it always runs against the actual jar).

  • Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)

The published Maven metadata of pulsar-client, pulsar-client-admin and pulsar-client-all
changes: dependency scopes (api deps → compile, internal deps → runtime), protobuf-java is no
longer a published dependency, and the artifactIds are restored to the historical coordinates. The
contents of the shaded jars also change: Netty native libraries are now relocated (and the Windows
tcnative native is bundled again), and the JSR-305 (javax.annotation) classes are no longer shipped.
These are 5.0.0-M1-SNAPSHOT (unreleased) artifacts, so no released artifact's metadata or contents are
affected.

lhotari added 2 commits June 15, 2026 22:44
…tifacts

The shaded client modules (pulsar-client-shaded, pulsar-client-admin-shaded,
pulsar-client-all) published incorrect Maven metadata: every non-bundled
dependency was emitted with `runtime` scope, and the artifacts were published
under their Gradle project names (e.g. `pulsar-client-shaded`) instead of their
historical Maven coordinates (`pulsar-client`, ...).

### Dependency scopes (POM + Gradle Module Metadata)

The gradleup-shadow `components["shadow"]` maps the whole `shadow` configuration
to a single java-runtime variant, so the api/implementation distinction was lost
and consumers could not compile against the non-bundled parts of the public API.
Because these modules also publish Gradle Module Metadata (preferred over the POM
by Gradle consumers), a POM-only fix would have been ignored.

- `pulsar.client-shade-conventions`: add a `shadowApi` dependency bucket and a
  consumable `shadowApiElements` (java-api) variant carrying the shaded jar,
  registered on the shadow component via
  `addVariantsFromConfiguration(...) { mapToMavenScope("compile") }`. `shadowApi`
  deps publish as `compile`, `shadow` deps as `runtime` — in both POM and GMM.
- `pulsar.public-java-library-conventions`: add `shadowApi` to the published-scope
  validation list.
- Split each non-bundled dependency by whether its types are exposed in the public
  API a consumer compiles against (compile: pulsar-client-api,
  pulsar-client-admin-api, opentelemetry-api) or used only internally by the
  bundled code (runtime: BouncyCastle, the unshaded Jackson annotations, the
  OpenTelemetry incubator, slog/slf4j, jspecify). protobuf-java is intentionally
  not published (it was `provided` in the Maven build); consumers using
  Schema.PROTOBUF / PROTOBUF_NATIVE supply their own protobuf.

### Artifact ids

- `pulsar.publish-conventions`: read `archivesName` for the published artifactId
  in an `afterEvaluate` block (per Gradle's maven-publish deferred-configuration
  guidance) so a module can override it in its build script body. The value is
  captured as a plain String, keeping the publication configuration-cache safe.
- `pulsar-client-shaded` / `pulsar-client-admin-shaded`: set `archivesName` to the
  historical Maven artifactId (`pulsar-client`, `pulsar-client-admin`). This drives
  the shaded jar base name, the published artifactId, and the project's module
  identity, so the pulsar-bom entries and NAR exclusions reference the same names.
  pulsar-client-all already matches its artifactId.

Assisted-by: Claude Code (Opus 4.8)
…ote repository

### Motivation

The root project publishes `org.apache.pulsar:pulsar` as a POM-only parent
artifact that every child POM references via `<parent>`. Its publishing was
hand-rolled in the root build script and declared only the local `localDeploy`
repository — never the ASF Nexus `apacheReleases` / `apacheSnapshots` repos that
`pulsar.publish-conventions` configures for the modules. As a result the root had
no `publish…ToApache…Repository` task, so a `publishAllPublicationsToApache…`
run uploaded every module but silently skipped the parent POM, leaving the
published modules unresolvable (missing parent).

### Modifications

- New convention `pulsar.publish-repositories-conventions`: owns the publish
  TARGETS shared by every published project — the `localDeploy` + ASF Nexus
  repositories, the snapshot/release validation, the `mavenPublishLock` upload
  serialization service, and GPG signing. It creates no publication and does not
  touch the `base`/`java` plugins, so it applies equally to the POM-only root
  project and to every java-library / java-platform module.
- `pulsar.publish-conventions` now applies it and keeps only the per-module
  publication wiring (component, POM metadata, sources/javadoc, artifactId).
- The root build script applies it (replacing its bare `maven-publish` + `signing`
  and its localDeploy-only repositories block), so the parent POM now uploads to
  the same Nexus staging repository as the modules, through the same lock.
- The snapshot/release validation now matches on the publish task name
  (`…ToApacheSnapshotsRepository`) instead of reading `task.repository`, which is
  null while the root's POM-only publish task is being created.

Verified: the root `publishMavenPublicationToApacheSnapshotsRepository` task now
exists and uploads `org/apache/pulsar/pulsar/.../pulsar-*.pom` to a file:// remote
alongside the modules; configuration cache stores and reuses without problems.

Assisted-by: Claude Code (Opus 4.8)
lhotari added 2 commits June 15, 2026 23:30
### Motivation

Resolution of the `com.gradleup.shadow` 9.4.1 plugin has been failing in CI. 9.4.2
is a patch release that is available on both Maven Central and the Gradle plugin
portal; upgrading avoids the flaky/missing 9.4.1 resolution.

### Modifications

Bump the `shadow` version in `gradle/libs.versions.toml` from 9.4.1 to 9.4.2. This
updates the `libs.plugins.shadow` alias used by `microbench` and the build-logic
convention classpath (which derives its dependency from the same alias), so all
shaded modules build against 9.4.2. Verified the shaded jar still builds and the
published POM/Gradle Module Metadata of pulsar-client-shaded are unchanged.

Assisted-by: Claude Code (Opus 4.8)
…and runtime-all

### Motivation

`microbench` and `pulsar-functions/runtime-all` apply the Shadow plugin via the
versioned `alias(libs.plugins.shadow)`. The Shadow plugin is already on the
buildscript classpath of every project that applies a build-logic convention
(build-logic declares `com.gradleup.shadow` as an `implementation` dependency, so
applying `pulsar.java-conventions` puts it on the classpath). A versioned plugin
request cannot be reconciled with a classpath plugin of unknown version, so plugin
resolution fails:

    Error resolving plugin [id: 'com.gradleup.shadow', version: '...']
    > The request for this plugin could not be satisfied because the plugin is
      already on the classpath with an unknown version, so compatibility cannot be
      checked.

### Modifications

Apply the Shadow plugin with `id("com.gradleup.shadow")` (no version) in both
modules — the same form `pulsar.shadow-conventions` uses — so the classpath plugin
is applied directly and its version stays governed centrally by the version catalog
via the build-logic classpath. Neither module wants the published-shaded-library
behavior of `pulsar.shadow-conventions` (microbench uses a `benchmarks` classifier
and keeps its plain jar; runtime-all keeps its own runtimeElements handling), so the
minimal `id` application is used. Verified that `./gradlew help` now configures the
whole build and both `shadowJar` tasks are still present.

Assisted-by: Claude Code (Opus 4.8)
lhotari added a commit that referenced this pull request Jun 15, 2026
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 15, 2026
@lhotari lhotari marked this pull request as draft June 15, 2026 23:01
lhotari added 3 commits June 16, 2026 02:26
### Motivation

The shaded client artifacts (pulsar-client, pulsar-client-admin, pulsar-client-all)
bundled Netty's native libraries under META-INF/native/ but never renamed them, so they
kept their original libnetty_*/netty_* names. Netty's shaded NativeLibraryLoader looks
the libraries up under the mangled shade-package prefix (org_apache_pulsar_shade_), so it
could not find them, and the unrelocated names also clash with an unshaded Netty on the
same classpath. The Windows tcnative native (netty_tcnative_windows_x86_64.dll) that the
Maven build shipped was additionally dropped during the Maven -> Gradle migration.

### Modifications

- pulsar.client-shade-conventions: fix the META-INF/native renaming. The previous regex
  "netty.+" with String.matches() never matched the real lib-prefixed files
  (libnetty_*.so/.jnilib), and the replacement inserted the prefix before "lib". Match
  "(lib)?netty.+" and insert the shade prefix immediately before "netty", producing e.g.
  liborg_apache_pulsar_shade_netty_transport_native_epoll_x86_64.so (and
  org_apache_pulsar_shade_netty_tcnative_windows_x86_64.dll for the prefix-less .dll).
  The prefix is derived from shadePrefix instead of being hard-coded.
- pulsar-common: add the netty-tcnative-boringssl-static windows-x86_64 classifier so the
  Windows OpenSSL native is bundled again (the classifier-less base jar carries no natives
  in netty-tcnative 2.0.x).

Verified by building :pulsar-client-shaded:shadowJar: every Netty native is relocated and
the set matches the 4.2.x release jar (modulo the Netty 4.2 io_uring -> io_uring42 rename).

Assisted-by: Claude Code (claude-opus-4-8)
… jar

### Motivation

The shaded client jar bundled the JSR-305 annotation classes (javax/annotation/Nonnull,
CheckForNull, concurrent/GuardedBy, meta/*, ...) from com.google.code.findbugs:jsr305 in
the default, unrelocated javax.annotation namespace. That leaks javax.* into the global
namespace of a shaded artifact (it can split-package / conflict with a jsr305 jar on the
consumer's classpath), and Pulsar 5.0 is moving off javax.* (JSpecify covers nullness
now). JSR-305 is a compile-time / static-analysis-only annotation library with no runtime
use, so it should not be on the runtime classpath at all.

The classes were bundled because pulsar-client-original declared
implementation(libs.jsr305), which put jsr305 on its runtime classpath; the shaded jar
then packed it. Guava/gRPC do not pull jsr305 into the runtime classpath here, so this is
the only source.

### Modifications

- pulsar-client (the pulsar-client-original module): change implementation(libs.jsr305) to
  compileOnly(libs.jsr305), alongside the other compile-only annotation libraries
  (spotbugs-annotations, swagger-annotations). This matches what pulsar-broker already
  does. jsr305 stays on the compile classpath (source uses javax.annotation.*) but is no
  longer a runtime/published dependency, so it is not bundled into the shaded jar.

Consumers that want to run SpotBugs/FindBugs analysis can add jsr305 themselves.

Verified: :pulsar-client-shaded:shadowJar contains no javax.* classes, jsr305 is no longer
on the shaded module's runtimeClasspath, and pulsar-client-original main and test sources
still compile.

Assisted-by: Claude Code (claude-opus-4-8)
… package roots

### Motivation

The shaded client jars (pulsar-client, pulsar-client-admin, pulsar-client-all) should only
contain Pulsar's own classes, third-party dependencies relocated under
org.apache.pulsar.shade, the bundled native libraries, and a small set of intentionally
unrelocated packages. Two regressions recently slipped past review: Netty native libraries
whose filenames were not rewritten, and leaked unshaded javax.annotation (JSR-305) classes.
A content check that runs after the jar is built turns problems like these into an
immediate build failure instead of something found later in a published artifact.

### Modifications

- pulsar.client-shade-conventions: add a verifyShadedJarContents task that opens the
  produced shaded jar and fails the build if any file entry falls outside the allowed
  roots: META-INF/, org/apache/pulsar/, com/scurrilous/circe/, org/apache/avro/reflect/
  (intentionally excluded from the org.apache.avro relocation so Avro reflection sees the
  annotations by their original name) and lib/ (bundled native libraries). Directory
  entries (the parents of allowed roots) are ignored. shadowJar is finalizedBy this task,
  so the check runs whenever the jar is built.

The task is configuration-cache and build-cache friendly: its execution action captures
only the jar-file provider, the report-file provider and the allow-list, all declared as
task inputs/outputs (jar input path sensitivity NONE; allow-list as an input property so
editing it re-runs the check). It is intentionally not a cacheable task, so it always runs
against the actual (or build-cache-restored) jar rather than restoring a stale pass.

Verified: passes for all three shaded jars; configuration cache stores then reuses with no
problems; removing an allowed root makes the task fail and list the offending entries.

Assisted-by: Claude Code (claude-opus-4-8)
@lhotari lhotari marked this pull request as ready for review June 16, 2026 00:01
@lhotari lhotari changed the title [fix][build] Fix Maven publication of the shaded client artifacts and the parent POM [fix][build] Fix the shaded client artifacts: jar contents and Maven publication Jun 16, 2026
The pulsar-shell distribution no longer bundles jsr305-3.0.2.jar now that
com.google.code.findbugs:jsr305 is a compileOnly dependency (a compile-time-only
annotation library), so checkBinaryLicense failed with "jsr305-3.0.2.jar mentioned
in LICENSE, but not bundled". Remove the stale entry from the shell LICENSE. The
server distribution still bundles jsr305 (pulled at runtime by other modules), so
its LICENSE is unchanged.

Assisted-by: Claude Code (claude-opus-4-8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants