Flaky on JDK 11: BackgroundDrainerMidDrainCapabilityGapTest.testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget

[jerrinot]

BackgroundDrainerMidDrainCapabilityGapTest#testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget fails intermittently on JDK 11, on main as well as on feature branches. Found while reviewing #99; it is not caused by that PR.

Symptom

java.lang.AssertionError: delivering between the windows ends the episode, so neither window
reaches the threshold and the slot must still drain [attempts=20] expected:<SUCCESS> but was:<FAILED>
    at ...BackgroundDrainerMidDrainCapabilityGapTest.lambda$testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget$0(BackgroundDrainerMidDrainCapabilityGapTest.java:240)
    at ...TestUtils.assertMemoryLeak(TestUtils.java:135)
    at ...BackgroundDrainerMidDrainCapabilityGapTest.testDeliveringBetweenTwoGapWindowsGrantsAFreshSettleBudget(BackgroundDrainerMidDrainCapabilityGapTest.java:201)

The drainer ends the episode with DrainOutcome.FAILED and quarantines the slot, where the test expects SUCCESS. factory.attempts() is 20 — the exact end of the second scripted gap window.

Reproduction

JAVA_HOME=~/.sdkman/candidates/java/11.0.26-amzn \
  mvn -pl core test -Dtest='BackgroundDrainerMidDrainCapabilityGapTest'

Re-run several times — it does not fail every run.

Evidence that it is flaky and not revision-specific

JDK 11.0.26-amzn (Corretto), Linux x86-64, repeated runs of the same test:

revision	failures / runs
981bdb0 (main)	4 / 7
065c7be (PR #99 branch)	3 / 7

Both revisions both pass and fail it. It initially looked like a regression on the PR branch (branch failed, main passed), but the outcome also inverts depending on how the test is invoked: running the single method in isolation vs. running the whole class flipped which revision failed, in both directions, across repeats. That rules out a code difference between the two revisions as the cause.

Not observed on JDK 26: a full mvn -pl core test there passed (3456 tests, 0 failures) — though that is a single observation, not a claim that the JDK matters.

What I ruled out

• Machine load. Reproduced on an idle machine (load ~1.6), not only under parallel builds.
• The 60 s wall-clock settle budget. RECONNECT_MAX_DURATION_MILLIS = 60_000 in the test, backoff 1–4 ms, and the whole run finishes in ~10 s, so capabilityGapElapsedNanos >= reconnectBudgetNanos (BackgroundDrainer.java:618) is not the terminal that fires.
• The settle-budget accumulators failing to reset as a group. capabilityGapAttempts, capabilityGapElapsedNanos and lastCapabilityGapNanos are reset together at all three sites (BackgroundDrainer.java:525-527, 563-565, 683-685), and a captured log of a passing run shows the episode restarting correctly — the second gap window begins again at attempt 1:

21:11:51.088 ... durable-ack capability gap mid-drain (...), re-entering settle budget
21:11:51.089 ... attempt 1: durable-ack unavailable, retrying after backoff
   ... attempts 2..8 ...
21:11:56.044 ... durable-ack capability gap mid-drain (...), re-entering settle budget   <- ~5 s delivering gap
21:11:56.054 ... attempt 1: durable-ack unavailable, retrying after backoff

What I did not determine

Root cause. Two candidates remain, and I could not separate them:

1. Test-harness race. The scripted delivering session (connection 2 acks frames 0 and 1, then drops — drops.put(2, 1L)) sometimes loses the race and never advances the watermark, so no reset happens, the two 9-sweep windows accumulate to 18 ≥ DEFAULT_MAX_DURABLE_ACK_MISMATCH_ATTEMPTS (16), and FAILED is the correct outcome for what actually happened on the wire.
2. A genuine intermittent failure of the episode reset in BackgroundDrainer.

The test drives a real TestWebSocketServer over a real loopback socket, so the delivering session's ack is genuinely timing-dependent.

The decisive diagnostic is the give-up line from a failing run, which reports which terminal fired and the elapsed time:

drainer giving up on slot {} after {} durable-ack-mismatch attempts ({}ms): {}

(BackgroundDrainer.java:620). If it reports ~18 attempts, candidate 1 is the explanation and the fix belongs in the test harness (make the delivering session's ack deterministic before the connection drops). I only managed to capture that log on passing runs.

Impact

Low for CI — build-jdk8 is the only job that runs tests, so this never reaches a pipeline today. It costs developer time on local JDK 11+ runs, and it is masking whichever of the two candidates above is true.

🤖 Generated with Claude Code

https://claude.ai/code/session_013ynWBNvVkrQpTxt1d429ED