DRIVER-201 Stage D — hygiene follow-ups from multi-address DNS resolution

[nikagra]

Follow-up hygiene for DRIVER-201 multi-address DNS resolution (#890). Each item below was found while
reviewing #890 and deliberately kept out of it: #890 claims to be behaviour-preserving for everything
except the multi-address expansion itself, and an item lands there only if it fixes a defect in a
claim the PR makes about itself. Everything else is here, with the reason it was deferred.

Nothing in this list is a regression introduced by #890 unless it says so.

Candidate-loop bounding

• Candidate de-duplication — tracked separately as Duplicate DNS records cause repeated connection attempts to the same address #989. Collapse repeated addresses where the
  resolver's answer is assembled, first occurrence wins, so a repeated A-record no longer costs an
  extra connect-timeout.
• A time budget between candidates. An internal constant, max(60s, 4 × connect-timeout) per
  logical connect, checked only between attempts — never interrupting an in-flight attempt, its auth
  exchange, or a protocol downgrade ladder. On exhaustion the last error surfaces with earlier failures
  suppressed, naming how many candidates were tried. Owned corner case: with many dead records ahead
  of a healthy one, a first build() under the default reconnect-on-init = false can fail where an
  unbounded scan would eventually have connected; reconnects self-heal because the shuffle starts later
  rounds elsewhere.
• A pure-move extraction of the candidate loop into its own class. ChannelFactory is the driver's
  highest-blast-radius file; the next change there should not be another inline callback.
• Cancellation of an in-flight connect when the control connection closes — today the loop keeps
  dialing until exhaustion. A responsiveness improvement, not a correctness requirement; it shares its
  check point with the time budget.

Resolution hazards

• A deadline on candidate resolution. ChannelFactory.resolveCandidates completes its future on
  every throwing path, but nothing covers a resolver that simply never completes: a custom
  AddressResolverGroup whose resolveAll promise stays pending leaves CqlSession.build(), a control
  reconnection or a pool reconnection hanging indefinitely. connect-timeout applies to the TCP
  connect, not the lookup. Why deferred: the mechanism is Netty's — Bootstrap.doResolveAndConnect
  has the same property — so it pre-exists upstream, and neither resolver a user gets without asking can
  reach it (DefaultNameResolver delegates to InetAddress.getAllByName(), which the OS bounds;
  DnsAddressResolverGroup carries its own query timeout). feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 widens the exposure by routing all
  address expansion through the resolver. Fix when taken: arm connect-timeout on the resolution
  future.
• DC inference does a blocking lookup per node, and answers wrong for a multi-record name — tracked
  as DefaultEndPoint.equals blocks on DNS and is inconsistent with hashCode for unresolved vs resolved addresses #1006. OptionalLocalDcHelper.inferDcFromControlConnection compares the control endpoint against
  every metadata node's, and DefaultEndPoint.equals resolves the unresolved side whenever exactly one
  side is unresolved, which feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 makes the permanent state of contact-point endpoints. So the loop
  issues a synchronous JVM DNS lookup per node on the thread driving LBP init, and binds to whichever
  single address the JDK returns first — which need not be the one the control connection reached. The
  same construct breaks the equals/hashCode contract. Highest-value item in this list: it is the
  one that is wrong specifically on the deployments the feature targets.
• EndPoint.resolve() is now called per node per topology refresh, on the admin loop.
  DefaultNode.setEndPoint asks PinnableEndPoint.sameIdentity, which compares resolve() results, and
  NodesRefresh reaches it twice per node per full refresh. For the in-tree endpoints that is a field
  read; for a third-party EndPoint that resolves eagerly — legal under the pre-feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 javadoc, which
  said only that resolve() is called per connection — it is now a blocking lookup per node per refresh.
  The contract change is in the upgrade guide; this is the migration hazard behind it.

Scheduling and resource hygiene

• PortAllocator.getNextAvailablePort runs on a Netty I/O event loop, once per candidate. The
  candidate loop executes inside resolveCandidates(...).whenComplete(...), i.e. on the loop the channel
  will be registered on, and isTcpPortAvailable opens, binds and closes a real ServerSocket per
  probed port. Pre-feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 this ran on the caller's thread and once per connect(). Why deferred: it is a
  scheduling change with no wrong answer, and the fix (hop to a worker for the scan) belongs with the
  loop extraction above.
• Every connect pays an event-loop task dispatch even when the address is already resolved.
  resolveCandidates schedules eventLoop.execute(...) before it can discover
  resolver.isResolved(address). Consulting the resolver rather than pre-checking isUnresolved() is
  right, but it does not require dispatching; the check can be inline when the caller is already on the
  loop. Pool refill after a node bounce pays this per channel.
• error.addSuppressed(...) mutates a Throwable the driver does not own. The identity dedup is
  scoped to one connect, so a shared stackless singleton — the pattern Netty and JDK NIO use for
  closed-channel/reset exceptions — accumulates suppressed entries for the JVM's lifetime. Why it is
  not simply "wrap it": wrapping the propagated cause in a driver-owned type is exactly what
  surfacedFailure must not do, since callers branch on the cause's type. A fix has to bound the
  suppressed list without changing what callers see.
• baseBootstrap.clone(eventLoop) silently overrides a user-installed EventLoopGroup. Netty's
  Bootstrap.clone(EventLoopGroup) assigns the group unconditionally, so a
  NettyOptions.afterBootstrapInitialized hook doing bootstrap.group(myGroup) — honoured before feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890,
  which connected the hooked bootstrap directly — is now ignored. Why deferred: the hook's contract
  needs deciding (honour the group, or document it as reserved), not just patching.

Pinning and endpoint plumbing

• One pin predicate, not four. DefaultEndPoint.pinTo, SniEndPoint.pinTo,
  ClientRoutesEndPoint.pinTo and ChannelFactory.pin each re-derive an overlapping "is this worth
  pinning to" guard, in three different shapes, with the isUnresolved() branch present in only two of
  the four — which is what makes the next item reachable. Extract one
  static boolean isPinnable(SocketAddress) onto PinnableEndPoint and call it from all four, so a
  future implementor has one example rather than three inconsistent ones. (The sibling item, one shared
  endpoint-identity predicate, landed in feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 as PinnableEndPoint.sameIdentity.)
• A pinned ClientRoutesEndPoint with no route yet stops consulting the route cache. A node whose
  host id has no system.client_routes entry resolves through its already-resolved fallback endpoint, so
  pin()'s isUnresolved() guard does not fire and the channel's pinned copy never consults
  topologyMonitor.resolve(hostId) again — so savePort, the SSL peer host and localEndPoint keep
  using the non-route address for the life of that connection, even after the route is published. The
  node's own endpoint is unaffected (each refresh builds a fresh one). Why deferred: uncovered ground in
  a new mechanism, not a regression — pre-feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 the control node's endpoint did not re-resolve either —
  and the fix is a design choice.
• ClientRoutesTopologyMonitor.reresolvesNodeAddresses() walks every metadata node with a map lookup
  each, on every control-reconnection plan build, on the admin event loop, recomputing a value that can
  only change when resolvedRoutesCache is swapped or the node set changes. Maintain it as a cached
  boolean at those points instead.

Configuration and documentation residue

• Restore the resolve-contact-points opt-in. feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 hard-codes it off (deprecated and inert). The
  restore is ~2 lines: true gives back the pre-feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 behaviour — resolve once at startup, one node per
  A-record, per-address metadata/metrics visibility, DNS consulted once — for users who relied on it.
  While here, decide what the driver-configuration report says about it, since feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890 leaves OptionsMap
  shipping an inert false.
• ContactPoints.merge's resolve parameter is dead from production if the opt-in is not restored.
  SessionBuilder is the only production caller and passes a bare false, so AddressUtils.extract's
  InetAddress.getAllByName branch is unreachable for contact points, while ContactPointsTest still
  exercises the true path that no configuration can reach. Whichever way the decision goes, the test
  should stop asserting on an unreachable branch.
• True up basic.contact-points docs: the multi-A-record sentence is accurate again post-feat: multi-address DNS resolution for contact points and connections (DRIVER-201) #890; add
  that each connection attempt re-resolves and re-shuffles.
• The contact-point reconnection fallback appends without deduplicating against the live plan. At plan
  time the contact points are still hostnames while the live nodes are already-resolved IPs, so with
  unchanged DNS a reconnection round that exhausts the live nodes retries roughly 2× the addresses.
  Accepted at the new default; recorded so it is not rediscovered as a bug.
• The ephemeral fallback nodes carry no metric updaters, so an operator watching
  errors.connection.auth sees zero while the driver is failing to authenticate against one. Giving them
  real updaters would register metrics for objects deliberately absent from metadata, so it is left as is.
• No dedicated counter for an identity mismatch. A mismatch increments the existing per-node
  CONNECTION_INIT_ERRORS and logs a warning, which does not distinguish it from any other init failure,
  so the symptom an operator sees is "the pool will not come up". Either accept that or add a counter —
  it should be a decision, not an omission.

Ordering hazard, accepted by design

• The negotiated state is latched after the caller's future completes. completeCandidate runs
  perAddressFuture.complete(driverChannel) and only then records the protocol version, cluster name and
  product type; because CompletableFuture.complete() runs already-registered dependents inline, an
  inline dependent of ChannelFactory.connect()'s stage observes the pre-latch values. This is
  deliberate — winning the completion race is what makes a candidate the accepted one, and therefore
  the only one entitled to record what it negotiated — and harmless today, since the latched fields are
  volatile and both consumers (ControlConnection, ChannelPool) hop to the admin executor. Recorded
  so a future non-async consumer reading getProtocolVersion() inline is a known hazard rather than a
  surprise.

Tests

• An end-to-end DNS-change-recovery IT (Simulacron plus the core/resolver package's JVM-global
  MultimapHostResolverProvider): the first address tried cannot identify itself, a later one can.
• One test against a real DnsAddressResolverGroup, rather than only the test resolver.
• Nothing observes SNI proxy-IP distribution. The unit tests assert the spreading decision per
  endpoint shape and that the shuffle happens, but not that connections land on more than one proxy
  A-record.

File separately (pre-existing, fixed by neither stage)

• Cloud and client-route channels can replace their pinned endpoint with an unpinned one during identity
  resolution, silently defeating the JAVA-2303 self-peer guard
  (broadcastRpcAddress.equals(localEndPoint.resolve()) — an unresolved address never equals a resolved
  one).
• Node registration remains outside the candidate loop. The connect hook proves the node can identify
  itself, but MetadataManager.registerNode() still runs after the connect and can still fail on its
  own, which advances the query plan as it does upstream. Unlike the identity read it is not
  address-specific, so another address of the same name would register no better.