Fix Raft member rejoining and rolling upgrade compatibility

[guillaume-chervet]

Summary

Fixes #299. Downstream validation: SlimPlanet/SlimFaas#403.

Based on develop (f10ace6), per CONTRIBUTING.md. Local package experiments include those existing development-branch changes as well as this PR.

This fixes three blockers found while upgrading a three-node SlimData cluster from DotNext 6.6.0 to 6.7.2:

• A removed live member receives HTTP 500 for AppendEntries when the leader setter reads its faulted election task. Handle that completed task without reading Result, and restore the follower state and fresh leadership waiters when the previously ready member is added again.

• Existing WAL metadata pages are interpreted using the current OS page size, changing page numbering on 16 KiB-page hosts. Read and validate the existing page sizes before opening WAL files, preserve that layout, and allow private buffers smaller than an OS page. New logs keep the current default. Invalid or inconsistent page sizes fail before files are opened/resized.

• New Raft HTTP headers break mixed-version operation with 6.6.0: missing X-Raft-State-Version now means implicit version zero; missing X-Raft-Last-Index falls back to the legacy one-entry backtracking behavior. Explicit malformed headers still fail parsing.

There are no public API changes, new dependencies or on-disk rewrites. The change preserves both legacy 4 KiB pages and logs already created with larger metadata pages.

Validation

• Before the fixes: the new HTTP rejoin test and both 4 KiB metadata cases fail; both 16 KiB cases pass on this macOS ARM64 host.
• After the fixes: all 8 focused regression cases pass, including rejection of invalid/mixed sizes before opening WAL files.
• The 9 HTTP header regression cases reproduce 3 failures before the compatibility change; all pass afterward. Direct HTTP-internal tests use a Debug-only friend assembly; test aliases disambiguate existing names shared by the core and HTTP assemblies.
• Full DotNext suite: 2,443 passed, 4 existing skips, 0 failed.
• Downstream SlimData: all 217 tests pass with locally packed 6.7.2-slimdata.402, including the existing re-addition test and a real 6.6.0 compacted snapshot/WAL fixture that fail against the published 6.7.2.
• With all three fixes (6.7.2-slimdata.402.2, commit 750c57f), all 1,545 downstream .NET tests pass (also after NuGet centralization, including embedded UI builds) and native AOT publication passes with unchanged baseline third-party warnings.
• Native rolling upgrade from 6.6.0 to the corrected packages passes on macOS ARM64 using existing snapshots/WALs. Follower/leader replacement, follower pauses, quorum loss/recovery and full restart verify all expected data on each of three nodes. A pending write resumes in 1.938 seconds after quorum returns (30-second bound); all 186 final values match per node. A second run after downstream NuGet centralization also passes (3.063-second quorum recovery). This validates forward rolling upgrade; rollback and arbitrary application state-version changes are outside this test.
• Comparative downstream performance: 16 runs / 389,372 measured operations with no errors and successful all-node state checks. Throughput and p99 gates pass, but peak RSS gates fail for two concurrency-12 cases (+21.2% mixed, +17.4% set; limit +15%). Measurements are variable and compare 6.6.0 against a private develop build; they do not isolate these three fixes. Downstream adoption remains blocked.
• Full local tests with coverage also pass (2,443 / 4 skips). CI 131614 passes Linux/Windows but times out in macOS coverage, with no failing test identified. New WAL regression cases now have a 60-second deadline and macOS CI reports detailed test progress. The preceding PR build 131612 passed all platforms; the unmodified develop build 131602 also timed out (on Linux). The new run must be checked; those observations do not prove a common cause.

Test commands:

dotnet test --project src/DotNext.Tests/DotNext.Tests.csproj   --filter-method '*RemovedLiveMemberCanRejoin' --filter-method '*MetadataPageSize*'
dotnet test --project src/DotNext.Tests/DotNext.Tests.csproj   --filter-class 'DotNext.Net.Cluster.Consensus.Raft.*'
dotnet test --project src/DotNext.Tests/DotNext.Tests.csproj

This contribution is AI-assisted (Codex); the ai_assisted label is required by CONTRIBUTING.md. Adding that label was denied for the contributor account; a maintainer needs to apply it. Built and tested with .NET SDK 10.0.300 on macOS ARM64. Existing compiler/analyzer warnings in untouched code are unchanged.

[guillaume-chervet]

@dotnet-policy-service agree