Skip to content

Flaky test: HttpTransportTests.testAccessTokenAuth / testBearerTokenAuth bind a random fixed port and fail with FatalStartupException #3076

Description

@polyglotAI-bot

Description

HttpTransportTests.testAccessTokenAuth and HttpTransportTests.testBearerTokenAuth
(module client-v2) start their WireMock server on a randomly chosen fixed port:

int randomPort = ThreadLocalRandom.current().nextInt(3000, 65535);
WireMockServer mockServer = new WireMockServer(WireMockConfiguration
        .options().port(randomPort).notifier(new ConsoleNotifier(false)));
mockServer.start();

client-v2/src/test/java/com/clickhouse/client/HttpTransportTests.java:1307 (testBearerTokenAuth)
and :1385 (testAccessTokenAuth).

If that port is already taken, mockServer.start() throws immediately and the test fails.
There is no retry and no fallback. The chosen range (3000-65535) overlaps the Linux
ephemeral port range (/proc/sys/net/ipv4/ip_local_port_range = 32768-60999 on the CI
runners), so any outbound socket held by a concurrently running test, by Maven, or by
another job on the same runner can take the port.

Every other WireMock server in the same file already uses .dynamicPort() (about 20
occurrences, e.g. lines 895, 1465, 1854). Only these two tests use the fixed-port form.

Steps to reproduce

  1. Occupy an arbitrary subset of TCP ports in 3000-65535 on the machine (this simulates a
    loaded CI runner). For the numbers below, 29768 ports (3000-32767, 47.6% of the range)
    were held by a helper process; the ephemeral range was left free so outbound
    connections still worked.
  2. Run the test repeatedly:
    mvn -B -pl client-v2 -DskipUTs=true -Dit.test=HttpTransportTests#testAccessTokenAuth -Dfailsafe.failIfNoSpecifiedTests=false -DfailIfNoTests=false verify
  3. The test fails on roughly the fraction of the range that is occupied.

Observed: 4 failures out of 12 runs (33%, expected ~48%), each with the same signature
as CI but a different port each time:

run 1: PASS
run 2: FAIL -> Failed to bind to /0.0.0.0:10117
run 3: FAIL -> Failed to bind to /0.0.0.0:15282
run 4: PASS
run 5: FAIL -> Failed to bind to /0.0.0.0:18977
run 6..9: PASS
run 10: FAIL -> Failed to bind to /0.0.0.0:14457
run 11..12: PASS

Contrast: HttpTransportTests#testSessionSettingsClientAndOperationLevels, which uses
.dynamicPort(), passed 6/6 under exactly the same port contention.

Error Log or Exception StackTrace

Verbatim from CI (amazon JDK 17 / "Test all modules",
https://github.com/ClickHouse/clickhouse-java/actions/runs/32807717786/job/97681145686):

[ERROR] com.clickhouse.client.HttpTransportTests.testAccessTokenAuth -- Time elapsed: 0.005 s <<< FAILURE!
com.github.tomakehurst.wiremock.common.FatalStartupException: java.io.IOException: Failed to bind to /0.0.0.0:32783
[ERROR]   HttpTransportTests.testAccessTokenAuth:1388 » FatalStartup ... Failed to bind to /0.0.0.0:32783
[ERROR] Tests run: 897, Failures: 1, Errors: 0, Skipped: 3

Note 32783 is inside the OS ephemeral range. The other 34 legs of the same matrix passed
on the same commit, and the change under test touched only client-v2 metadata handling -
nothing in the auth path - so the failure is not attributable to the code under test.

Expected Behaviour

The test binds a free port and never fails because of port allocation. The Client.Builder
already receives the port from mockServer.port() after start, so nothing needs the port
value in advance.

Code Example

Suggested fix - use the same form as the rest of the file, at both sites:

WireMockServer mockServer = new WireMockServer(WireMockConfiguration
        .options().dynamicPort().notifier(new ConsoleNotifier(false)));
mockServer.start();

.dynamicPort() binds port 0, so the kernel picks a port that is guaranteed free, and
mockServer.port() (already used throughout both tests) returns the actual port. The
randomPort local becomes unused and can be dropped, together with the now-unneeded
ThreadLocalRandom usage if no other test needs it.

Please do not fix this with a retry loop or a narrower random range - both leave the race
in place. There is no product-code race here; the defect is entirely in the test fixture.

Configuration

Environment

  • Cloud
  • Client version: main @ 667cbe90b
  • Language version: OpenJDK 17.0.18
  • OS: Ubuntu 24.04 (container), also seen on the amazon JDK 17 CI runner

ClickHouse Server

  • ClickHouse Server version: not relevant - the test uses a mocked WireMock server and
    returns early when isCloud().

Found by our PR monitor: the failure was seen on unrelated PRs whose changes do not touch
the auth path, and it is already tracked internally as a known non-routing flake. Verified
here by reproducing the bind failure locally under port contention, not by inspection only.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions