Composable pool: coalesced Singleton waiters are canceled when the maker bounces to the fallback leg

[aajtodd]

Version

hyper-util:0.1.20

Platform

macos aarch64

Summary

Summary

In hyper-util 0.1.20's composable pool layers (hyper_util::client::pool::{negotiate, singleton}), concurrent same-authority requests through the eager Negotiate stack coalesce on the Singleton upgrade leg. When the maker discovers the connection is not h2 and returns UseOther to bounce to the H1 fallback, its error path resets the Singleton state and drops the coalesced waiters' oneshot senders. Each waiter resolves Err(SingletonError) with source "singleton connection canceled" instead of obtaining a connection. Exactly one caller (the maker) succeeds; all other concurrent callers to that authority fail spuriously.

Mechanism

1. Negotiate::call starts in Eager state and polls the upgrade (Singleton) leg first.
2. Concurrent calls to clones of the same Negotiate (sharing Arc<Mutex<State>>) coalesce in the Singleton: the first becomes the maker (State::Making), the rest become waiters holding oneshot::Receivers, with their senders stored in the Making waiter Vec.
3. The maker's inner future finds the slot empty (no h2 negotiation occurred) and returns Err(UseOther).
4. In SingletonFuture, the maker's Driving arm hits the error branch (singleton.rs:243-250): it replaces the shared state with State::Empty, dropping the State::Making(waiters) value and thereby every waiter sender.
5. Each waiter's Waiting arm polls its oneshot::Receiver, which resolves Err(Canceled), and returns Err(SingletonError(Canceled.into())) (singleton.rs:255).
6. The maker's UseOther propagates up to its own Negotiate, which bounces to the fallback leg and succeeds. The waiters have no equivalent path and are orphaned.

DitchGuard::drop resets state the same way when the maker future is dropped or cancelled before resolving (the path exercised by the existing cancel_driver_cancels_all test).

Context

singleton.rs test cancel_driver_cancels_all currently asserts this cancel behavior and carries a // TODO: this should be able to be improved with a cooperative baton refactor comment, indicating the orphaning is acknowledged.

Environment

• hyper-util 0.1.20 (crates.io release)
• Features: client, client-pool
• tokio multi_thread runtime (deterministic; does not surface on current_thread)

Code Sample

## Reproduction

Minimal standalone crate: https://github.com/aajtodd/repro-hyper-util-singleton-cancel


cargo test -- --nocapture


Observed:


BUG REPRODUCED: 1/8 succeeded, 7/8 got SingletonError(Canceled)


The repro builds an eager `negotiate` stack with a `Singleton` upgrade leg, an identity fallback leg, and an `inspect` closure that always reports not-h2. It readies N clones of the negotiate service, calls all N synchronously, then polls the returned futures together on a `multi_thread` runtime.

Expected Behavior

Expected: every coalesced caller obtains a connection — either the maker's established connection or its own via the fallback leg.

Actual Behavior

Observed: the maker succeeds; all coalesced waiters resolve Err(SingletonError) with source "singleton connection canceled".

Impact: any consumer issuing concurrent requests to the same authority through the composable pool with the eager negotiate stack sees most concurrent same-authority requests fail spuriously under load rather than reusing or falling back.

Additionally there is no typed error to key off of for this.

Additional Context

No response