Two blockers encountered while validating a 6.6.0 → 6.7.2 Raft upgrade
@sakno
Downstream tracking: SlimPlanet/SlimFaas#402. Tested on macOS ARM64 (.NET SDK 10.0.300, 16 KiB system pages), with three real Raft nodes and with focused tests against develop (f10ace6).
1. A removed live member cannot rejoin
Create a three-node HTTP Raft cluster, remove one live follower, append another entry, and add the follower again. Catch-up applies the removal before the node receives the re-addition. Subsequent AppendEntries returns HTTP 500 / QuorumUnreachableException, and the follower does not apply the re-addition.
MoveToStandbyState(resumable: false) faults the election task. The Leader setter subsequently accesses that completed task's Result without checking success. In addition, UnfreezeAsync short-circuits on the previously completed readiness probe, and the old leadership task remains faulted.
The existing downstream membership test passes on 6.6.0 and fails repeatedly on 6.7.2. A focused upstream HTTP cluster test reproduces the failure without downstream application code.
2. Existing WAL metadata pages are interpreted using a different size
WAL metadata pages created with 6.6.0 have a 4096-byte layout. Newer constructors use max(4096, Environment.SystemPageSize), which is 16384 on this host. After correctly restoring the snapshot, reopening the compacted WAL fails with:
WriteAheadLog.InternalException: WAL page 0 doesn't exist on the disk
WriteAheadLog.MetadataPageManager.GetView
WriteAheadLog.ApplyAsync
WriteAheadLog.InitializeAsync
A rolling upgrade fails at its first follower; all 180 synthetic sets were verified on every node before upgrading. The same saved snapshot/WAL restores successfully with 6.6.0. The fixture and test are in SlimPlanet/SlimFaas#403.
The persisted page size needs to survive reopening, including logs already created with the newer larger-page layout. Invalid/mixed page sizes should fail before files are opened or resized. Private-memory buffers also need an alignment compatible with legacy pages smaller than the OS page size.
A proposed fix and tests are being prepared against develop. No causal link to the original downstream staging incident is claimed.
Additional rolling-upgrade blocker: HTTP headers
After correcting WAL restoration, the next real 6.6.0 → 6.7.2 native rolling-upgrade attempt fails because X-Raft-State-Version is required on incoming requests. Legacy nodes have no state version header (implicit version zero). Conversely, a new leader requires X-Raft-Last-Index on AppendEntries responses, which older followers do not emit. Both absent headers need legacy-compatible defaults while keeping malformed present values rejected. Three regression cases fail before this compatibility fix. Tracked with the other fixes in #300.
Two blockers encountered while validating a 6.6.0 → 6.7.2 Raft upgrade
@sakno
Downstream tracking: SlimPlanet/SlimFaas#402. Tested on macOS ARM64 (.NET SDK 10.0.300, 16 KiB system pages), with three real Raft nodes and with focused tests against
develop(f10ace6).1. A removed live member cannot rejoin
Create a three-node HTTP Raft cluster, remove one live follower, append another entry, and add the follower again. Catch-up applies the removal before the node receives the re-addition. Subsequent AppendEntries returns HTTP 500 /
QuorumUnreachableException, and the follower does not apply the re-addition.MoveToStandbyState(resumable: false)faults the election task. TheLeadersetter subsequently accesses that completed task'sResultwithout checking success. In addition,UnfreezeAsyncshort-circuits on the previously completed readiness probe, and the old leadership task remains faulted.The existing downstream membership test passes on 6.6.0 and fails repeatedly on 6.7.2. A focused upstream HTTP cluster test reproduces the failure without downstream application code.
2. Existing WAL metadata pages are interpreted using a different size
WAL metadata pages created with 6.6.0 have a 4096-byte layout. Newer constructors use
max(4096, Environment.SystemPageSize), which is 16384 on this host. After correctly restoring the snapshot, reopening the compacted WAL fails with:A rolling upgrade fails at its first follower; all 180 synthetic sets were verified on every node before upgrading. The same saved snapshot/WAL restores successfully with 6.6.0. The fixture and test are in SlimPlanet/SlimFaas#403.
The persisted page size needs to survive reopening, including logs already created with the newer larger-page layout. Invalid/mixed page sizes should fail before files are opened or resized. Private-memory buffers also need an alignment compatible with legacy pages smaller than the OS page size.
A proposed fix and tests are being prepared against
develop. No causal link to the original downstream staging incident is claimed.Additional rolling-upgrade blocker: HTTP headers
After correcting WAL restoration, the next real 6.6.0 → 6.7.2 native rolling-upgrade attempt fails because
X-Raft-State-Versionis required on incoming requests. Legacy nodes have no state version header (implicit version zero). Conversely, a new leader requiresX-Raft-Last-Indexon AppendEntries responses, which older followers do not emit. Both absent headers need legacy-compatible defaults while keeping malformed present values rejected. Three regression cases fail before this compatibility fix. Tracked with the other fixes in #300.