[server] Recover coordinator leader election after ZK session expiry

[Kaixuan-Duan]

Purpose

Linked issue: close #4362

Brief change log

Root cause: when the ZooKeeper session of the coordinator expires (e.g. after a long GC pause), ZooKeeper deletes the ephemeral LeaderLatch node, and the now-empty election parent /coordinators/election (a container znode created by the latch) can then be garbage-collected by ZooKeeper. On reconnection, the LeaderLatch of the shaded Curator 5.4.0 lists the election path and gets a NONODE result, which is silently ignored — the latch stays permanently idle (fixed upstream by CURATOR-724 in Curator 5.8.0, which Fluss does not shade yet), so a single-coordinator cluster remains without a leader until it is restarted.

Changes:

• CoordinatorLeaderElection now uses a LeaderLatch subclass (RecoveringLeaderLatch) that recreates the election parent on RECONNECTED before the latch's own reconnection handling, so the latch re-creates its election node and the coordinator regains leadership without a restart.
• The parent is recreated as a persistent znode: unlike a container znode it is never garbage-collected when empty, so the same failure cannot recur.
• A recovery failure is only logged and is retried on the next RECONNECTED event.
• The recovery piggybacks on handleStateChange instead of registering a separate connection-state listener, because Curator notifies connection-state listeners in ConcurrentHashMap iteration order, which does not follow registration order and would not guarantee the recovery runs before the latch's own reconnection handling.
• Coordinator epoch fencing is unchanged: the recovery only rebuilds the election entry znode; becoming leader still goes through fenceBecomeCoordinatorLeader with epoch increment.

Tests

• New regression test CoordinatorHighAvailabilityITCase#testRegainsLeadershipAfterSessionExpirationWithElectionParentDeleted: kills the Coordinator's ZK session and deletes the now-empty election parent (simulating ZooKeeper's container GC), then asserts that leadership is regained without a restart. The test fails with a timeout before the fix and passes after it; a control run that kills the session without deleting the parent recovers fine, which isolates the parent deletion as the cause.
• Full run: CoordinatorHighAvailabilityITCase 7/7, CoordinatorLeaderElectionTest 4/4.

API and Format

Documentation

[platinumhamburg]

The parent repair still leaves a race: checkExists() may find an empty CONTAINER, which ZooKeeper deletes before the existing latch calls getChildren() through line 347. In the bundled Curator implementation, that NONODE result does not trigger reset(), so election recovery can stop despite a healthy connection.

I would suggest recovering at the election lifecycle level: once the session is lost, abandon the old election participation and start a fresh LeaderLatch after connectivity returns.

The relevant scenarios are:

• SUSPENDED → RECONNECTED without LOST: retain the existing latch, because its session and candidate node may still be valid. Let the latch revalidate leadership.
• LOST: discard the old election participation, whether this coordinator was leader or standby. Its ephemeral candidate no longer represents a valid participation.
• Connectivity restored after LOST: start a fresh latch, register a candidate belonging to the current session, and determine leadership through the normal election.
• Another coordinator wins: remain standby and continue participating; reconnection does not imply leadership.
• Election or leader initialization fails: remain non-serving, with a path to another election attempt rather than becoming permanently idle.

The key invariants are:

1. A lost session's election state must never authorize leadership in a later session.
2. Recovery must establish a valid candidate for the current session before relying on the candidate list to determine leadership.
3. Only the current election attempt may grant or revoke local leader authority.
4. Election victory alone is insufficient: Fluss's epoch fencing and leader initialization must succeed before serving as leader.
5. Recoverable failure must not require an unrelated future reconnect to make progress.

A fresh latch enters reset() and creates its EPHEMERAL_SEQUENTIAL candidate using creatingParentContainersIfNeeded(). The bundled Curator create path handles NONODE by recreating the parents and retrying candidate creation. It reads children after creation succeeds; while that candidate remains present, its parent cannot be collected as an empty container.

This addresses the missing-parent race through normal candidate registration, without modifying shaded Curator or repairing the parent separately.

[Kaixuan-Duan]

Addressed by restarting the election at the lifecycle level instead of repairing the parent.

• On LOST, the participation of the old latch is discarded: its notLeader handling revokes local leadership first, and restartElection() closes the stale latch once the connection returns.
• On the first RECONNECTED after a LOST, a fresh LeaderLatch is started: it registers a new EPHEMERAL_SEQUENTIAL candidate for the current session via creatingParentContainersIfNeeded() — the Curator create path recreates a missing election parent and retries, so the check-then-GC race is gone by construction, and the non-empty parent can no longer be collected.
• SUSPENDED → RECONNECTED without LOST keeps the existing latch, which revalidates leadership with the same election node.
• Reconnection never implies leadership: the fresh latch competes in the normal election; if another coordinator wins, this one stays standby and keeps participating, and becoming leader still goes through epoch fencing (fenceBecomeCoordinatorLeader) before serving.
• Election restarts, initialization, and cleanup are serialized on a single callback thread, so only the current election attempt grants or revokes local leader authority.