Overlap checkpoint publication and renew recovery claims

[forhappy]

Checkpoint preparation serialized shared-log capture behind bucket publication. This change overlaps immutable checkpoint preparation/publication with subsequent capture while preserving ordered credit, exact cell positions, fencing, authoritative publication and recovery proofs. It also records bounded durability-stage telemetry, renews long recovery claims, and adopts only an exactly verified completed recovery result on retry.

Signed append grants now tolerate the existing bounded receiver-clock skew during sender verification. Receiver validity, the signed lifetime, monotonic cache horizon, native fsync and all-members acknowledgement remain unchanged. The queued three-VM performance comparison applies the same transport/deadline corrections to main and the PR.

Fresh maintenance CI exposed a live inventory change during follower-policy confirmation. The reference observer now drops that interval’s role/policy proofs and leaves maintenance blocked until a fresh full collection succeeds. The underlying seven-category verification and all fencing/authority errors remain intact. A real second-Cell activation during a held recheck reproduces the old error 5/5 times. Both new race/fencing tests pass 20 repeats; both original maintenance scenarios pass 10 repeats. The broader persisted-observer run initially records 27 passes and two timestamp failures, retained in the evidence.

A read-only probe then captures a 100.1-ms backward VM clock step exactly matching a Lima guest-agent SyncTime adjustment. A reversible change removes only that guest agent’s clock-setting capability in the four owned test VMs; their existing NTP services and original boot IDs remain intact. All four five-minute probes observe no backward steps in 53,610 samples. The exact same frozen observer binary passes all 29 tests in each of three repeats, including both timestamp cases and invalid-clock rejection. Cellule’s code, timestamps, deadlines and guards are unchanged by this environment correction. The clock defect is confirmed; its occurrence at the two original failures was not captured, and the older unlogged Fenced cause remains unproved. Rust CI 38299468998 at 90a2fab passes both workspace and MSRV, including all 386 fleet-journal/controller-model tests. Subsequent production benchmark sources remain unchanged.

The follow-up Rust run 38303478642 at 820b9e5 exposes a separate receiver-test accounting race: a resolve reply can precede release of its 48-byte input and 64-byte result reservations. The unchanged test reproduces the exact 112-byte failure once in 188 attempts; an isolated pause at the actual completion boundary reproduces it 5/5 times. The fixture now verifies receipt retirement separately and retains exact whole-runtime zero accounting after its existing joined shutdown. It passes the paused diagnostic 5/5, 1,000 clean repeats with four processes, all nine receiver tests, all 246 Cargo-launched runtime integration tests (six documented environment-dependent tests ignored), and strict Clippy. The failed FIFO-only probe and initial direct-binary run missing Cargo’s process-helper environment are retained. Temporary instrumentation is removed; production accounting and benchmark code are unchanged.

Rust CI 38308865630 at b60ac4b now passes the complete workspace and MSRV jobs, including all 386 fleet-journal/public-controller tests and both original maintenance scenarios. The receiver test also passes in hosted CI. This confirms the correction’s verification; it does not establish the older unlogged maintenance failure’s cause or qualify performance.

The idle-host preflight also exposed its own collection overhead being classified as competing work. A real retained guard sample counts 26.7% CPU from the owned monitor’s top child. Only the process ownership classifier is corrected, with PID/start/command/source identity checks and seven positive/negative cases. Foreign collection tools, compilers and provider activity remain counted; raw process and total host CPU traces are retained. The original quiet thresholds, five-minute window, exact benchmark command and every qualification gate remain unchanged. Genuine background work still refuses admission.

Fresh performance evidence: hosted run 38303478631 completes four alternating main/PR pairs per mode. At concurrency 16, invocation latency and throughput are:

Leased lane	Main / PR operations/s	Main / PR p99
Local writes	950.69 / 873.67	31.714 / 34.894 ms
Forwarded writes	714.00 / 740.91	43.050 / 36.139 ms
Local warm reads	28,048.23 / 28,349.27	0.540 / 0.540 ms
Forwarded warm reads	7,183.39 / 7,161.29	2.760 / 2.865 ms

All 242,192 raw latency samples per mode, publication coverage, binary hashes and four-run medians reconcile through the canonical verifier. Both original coarse gates pass. Local write throughput/p99 changes are -8.10%/+10.03%, with p95/p99 alerts; forwarded writes are +3.77%/-16.05%. Leased expired-burst local reads retain a 1.1752× p99 alert. Object-only local reads at concurrency one and fresh-client local reads at concurrency 16 retain 1.6335/1.1014× p99 alerts. Object-only concurrency-16 local/forwarded writes achieve 821.90/694.18/s, against main’s 828.60/690.97/s, with p99 32.361/33.037 ms against 36.882/33.999 ms. All lanes and earlier failed attempts remain retained.

Runs 38286916365, 38289876505, 38292326894, 38299468996 and the newest run use byte-identical binaries, with mixed paired gains/regressions. No general throughput or latency improvement is established. These per-cell-root measurements do not qualify shared-log capacity on three separate 1-vCPU/2-GiB VMs.

The corrected three-VM 750-write/s comparison remains queued for five continuous quiet minutes on this workstation. It keeps 2,000 Cells, 30s warmup, 300s measurement, the 64-MiB budget, 128/128 client limits, alternating order and every-ACK recovery/exact-retry audits. Earlier busy-host main/PR cases fail delivery, latency and backlog gates. The retained PR state later recovers all 2,000 roots and passes all 110,861 ACK audits; the original HTTP/drain failures remain failures. The prior 701 writes/s run used an 8-vCPU owner and tmpfs and is not a matched baseline. The prepared SQL benchmark production sources are unchanged by the example/test/documentation follow-ups.

Two native peer-partition repeats on three independent 1-vCPU/2-GiB VMs pass: actual blocked follower TCP paths hold the next write/read responses; recovery refuses a missing complete witness, then recovers an unpublished ACK and exact retry with one original follower reachable while the other remains blocked. Both paths heal before joined shutdown, with no OOM. This is process-loss and peer-path verification, not physical power-loss qualification. Full capacity ramps, idle reads/mixed load and the conditional rollback decision remain open.

See measurements, native fault evidence, and append-grant contracts.