fix #446: gate commit quorum and follower ACKs on fsync-durable index (RPO=0)

.dockerignore

@@ -1,3 +1,5 @@
 **/target/
 .git/
-examples/
+examples/*
+!examples/three-nodes-standalone
+!examples/client-usage-standalone

.github/PULL_REQUEST_TEMPLATE.md

@@ -61,6 +61,12 @@
 
 ---
 
+## AI Assistance
+
+- [ ] This PR was written in part with the assistance of generative AI. All ideas and architecture decisions are mine; I have fully reviewed all changes.
+
+---
+
 ## Reviewer Notes
 
 (Optional: anything reviewers should focus on)

CHANGELOG.md

@@ -32,6 +32,18 @@ All notable changes to this project will be documented in this file.
   returns immediately, and entries arriving during an in-flight fsync are coalesced into the same physical
   disk flush. Storage-level group commit is restored without artificial batching windows.
 
+- **🛑 Client-acknowledged writes could be lost on correlated power loss (#446)**: Raft commit quorum
+  counted the leader's own log contribution using its in-memory tail (`last_entry_id()`), not its
+  fsync-confirmed position (`durable_index()`) — a write could reach a majority-looking commit index,
+  and be acknowledged to the client, before enough replicas had actually synced it to disk. If those
+  nodes then lost power before their next fsync, the acknowledged write was gone. Fixed: leader quorum
+  calculation, follower `AppendEntries` ACK timing (a follower now withholds its response until its own
+  `durable_index` reaches the acknowledged entry), and single-voter clusters (previously exempted from
+  this class of fix, see #329) all gate on `durable_index`. RPO=0 for acknowledged writes is now a
+  mandatory invariant. Net effect: write acknowledgment latency now includes fsync time on a quorum of
+  replicas — see [Throughput Optimization Guide](./d-engine/src/docs/performance/throughput-optimization-guide.md)
+  for tuning `idle_flush_interval_ms`.
+
 ### Changed
 
 - **MSRV raised to Rust 1.89**: The `data_dir` startup lock (prevents two node processes from
@@ -65,6 +77,17 @@ All notable changes to this project will be documented in this file.
 
 - **`NodeBuilder` is no longer public** — use `EmbeddedEngine::start_custom`/`StandaloneEngine::run_custom` to plug in a custom storage engine or state machine. See [Migration Guide](./MIGRATION_GUIDE.md) for details.
 
+- **⚠️ `[raft] ordered_channel_capacity` renamed to `max_pending_append_responses`** (#446): Follows the
+  gRPC `AppendEntries` forwarder rewrite (`FuturesUnordered`-based, no longer strict-FIFO) that shipped
+  alongside the durability fix above. Old field name is silently ignored, not an error — update existing
+  configs to the new name to keep the setting in effect.
+
+- **⚠️ `[raft.persistence] strategy` removed** (#446): `PersistenceStrategy` was a single-variant enum
+  (`MemFirst`) left over from #268; its only meaning now lives in whether an entry has reached
+  `durable_index`, which is no longer a configurable choice. Existing configs setting `strategy =
+  "MemFirst"` or `"DiskFirst"` are silently ignored, not an error — remove the field, `flush_policy`
+  is the only persistence knob now.
+
 ---
 
 ## [v0.2.4] - 2026-05-23

benches/embedded-bench/Makefile

@@ -1,27 +1,25 @@
 # Makefile for embedded-bench
 # Provides benchmark commands matching embedded-bench/reports/v0.2.0/report_v0.2.0_final.md
 
-.PHONY: help build clean test-single-write test-high-conc-write test-linearizable-read test-lease-read test-eventual-read test-hot-key all-tests
+.PHONY: help build clean clean-log-db test-single-write test-high-conc-write test-linearizable-read test-lease-read test-eventual-read test-hot-key all-tests \
+        all-ram-tests ramdisk-create clean-ram-log-db ramdisk-release
 
+# ===============================
+# Global Variables
+# ===============================
 BENCH_BIN := ./target/release/embedded-bench
 CONFIG_DIR := ./config
+LOG_LEVEL ?= warn
 
 # Node selection (default: n1)
 NODE ?= n1
 CONFIG_PATH := $(CONFIG_DIR)/$(NODE).toml
 SINGLE_NODE_CONFIG_PATH := $(CONFIG_DIR)/single_node.toml
-DATA_DIR := ./data/$(NODE)
-SINGLE_NODE_DATA_DIR := ./data/single-node
 
 # Metrics port per node — same scheme as examples/three-nodes-standalone
 # (8081/8082/8083), reused because the two setups never run concurrently.
 METRICS_PORT := $(if $(filter n2,$(NODE)),8082,$(if $(filter n3,$(NODE)),8083,8081))
 
-# ===============================
-# Global Variables
-# ===============================
-LOG_LEVEL ?= warn
-
 # Common parameters (matching Standalone tests)
 KEY_SIZE := 8
 VALUE_SIZE := 256
@@ -35,6 +33,25 @@ CLIENTS ?= 1000
 VERIFY ?= false
 VERIFY_FLAG := $(if $(filter true,$(VERIFY)),--verify-write,)
 
+# RAM disk (macOS only, optional) — isolates this node's storage on its own
+# independent RAM disk volume, to strip physical-disk latency out of a
+# benchmark. Not a general fix for unrelated flakiness — see
+# tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md.
+# One-shot: `make all-ram-tests`. Manual/single-node: add RAMDISK=true to
+# any test-* target (needs the other two nodes already running for quorum).
+RAMDISK ?= false
+RAMDISK_SIZE_MB ?= 1024
+RAMDISK_SECTORS := $(shell echo $$(( $(RAMDISK_SIZE_MB) * 2048 )))
+RAMDISK_VOLUMES := RAMDisk1 RAMDisk2 RAMDisk3
+RAMDISK_INDEX := $(if $(filter n2,$(NODE)),2,$(if $(filter n3,$(NODE)),3,1))
+
+ifeq ($(RAMDISK),true)
+DATA_DIR := /Volumes/RAMDisk$(RAMDISK_INDEX)/$(NODE)
+SINGLE_NODE_DATA_DIR := /Volumes/RAMDisk1/single-node
+else
+DATA_DIR := ./data/$(NODE)
+SINGLE_NODE_DATA_DIR := ./data/single-node
+endif
 
 # On macOS with Homebrew: auto-detect compression lib paths to skip bundled C++
 # compilation of RocksDB dependencies, which fails under macOS 26 + Xcode 26
@@ -58,7 +75,6 @@ ifneq ($(ZSTD_PREFIX),)
   BREW_ROCKSDB_ENV += ZSTD_LIB_DIR=$(ZSTD_PREFIX)/lib
 endif
 
-
 help:
 	@echo "Embedded-bench Makefile - Performance Testing"
 	@echo ""
@@ -83,6 +99,14 @@ help:
 	@echo "Run All:"
 	@echo "  make all-tests          Run all benchmark tests"
 	@echo ""
+	@echo "RAM Disk Cluster (macOS only, optional — isolates disk latency):"
+	@echo "  make all-ram-tests NODE=nX       Create RAM disks, run full --batch suite on this node (like all-tests)"
+	@echo "                                   Run in 3 terminals with NODE=n1/n2/n3 to form the cluster"
+	@echo "  make ramdisk-create              Create RAMDisk1/2/3 if not already mounted"
+	@echo "  make clean-ram-log-db            Wipe node data off the RAM disks (keeps them mounted)"
+	@echo "  make ramdisk-release             Unmount RAMDisk1/2/3, freeing the memory"
+	@echo "  Add RAMDISK=true to any single test-* target (needs the other 2 nodes already running)"
+	@echo ""
 	@echo "Examples:"
 	@echo "  make test-linearizable-read              # Run on n1 (default)"
 	@echo "  make test-linearizable-read NODE=n2      # Run on n2"
@@ -109,6 +133,7 @@ clean-log-db:
 	rm -rf ./logs/*
 	rm -rf ./data/*
 	rm -rf ./snapshots/*
+
 # ============================================
 # Write Performance Tests
 # ============================================
@@ -262,4 +287,34 @@ all-tests: build
 		put
 	@echo ""
 	@echo "Compare results with Standalone mode:"
-	@echo "  Standalone report: ../../benches/standalone-bench/reports/v0.2.2/report_v0.2.2.md"
+
+# ============================================
+# RAM Disk Cluster (macOS only, optional)
+# ============================================
+# Isolates each node's storage on its own independent RAM disk volume — use
+# this to strip physical-disk latency out of a benchmark (e.g. to study
+# fsync/scheduling behavior in isolation), not as a general fix for
+# unrelated flakiness. See tickets/milestones/v0.2.5/446-perf-batching-measurement-2026-09-13.md.
+
+# Ensures the RAM disks exist, then runs the full --batch suite for this
+# node on its own RAM disk volume — same scope/shape as `make all-tests
+# NODE=nX`, just on RAM disk. This is single-node, like all-tests: run it in
+# 3 separate terminals with NODE=n1/n2/n3 to form the cluster, e.g.
+#   make all-ram-tests NODE=n2 CLIENTS=100
+all-ram-tests: ramdisk-create
+	$(MAKE) all-tests RAMDISK=true NODE=$(NODE) CLIENTS=$(CLIENTS)
+
+# Idempotent — mounts any of RAMDisk1/2/3 that aren't already present.
+ramdisk-create:
+	@[ "$$(uname)" = "Darwin" ] || { echo "RAM disk targets need macOS."; exit 1; }
+	@for v in $(RAMDISK_VOLUMES); do \
+		[ -d "/Volumes/$$v" ] || diskutil erasevolume HFS+ $$v `hdiutil attach -nomount ram://$(RAMDISK_SECTORS)` >/dev/null; \
+	done
+
+# Wipes node data off the RAM disks; keeps the volumes mounted.
+clean-ram-log-db:
+	@for v in $(RAMDISK_VOLUMES); do rm -rf /Volumes/$$v/*; done
+
+# Unmounts the RAM disks entirely, releasing the memory back to the OS.
+ramdisk-release:
+	@for v in $(RAMDISK_VOLUMES); do diskutil eject /Volumes/$$v 2>/dev/null || true; done

benches/embedded-bench/config/n1.toml

@@ -16,11 +16,7 @@ max_drain = 1024
 max_batch_size = 200
 
 [raft.persistence]
-strategy = "MemFirst"
 flush_policy = { Batch = { idle_flush_interval_ms = 1000 } }
-# Maximum number of log entries to buffer in memory
-# when using async persistence strategies (MemFirst/Batched)
-max_buffered_entries = 10000
 
 [raft.metrics]
 enable_backpressure = false

benches/embedded-bench/config/n2.toml

@@ -16,11 +16,7 @@ max_drain = 1024
 max_batch_size = 200
 
 [raft.persistence]
-strategy = "MemFirst"
 flush_policy = { Batch = { idle_flush_interval_ms = 1000 } }
-# Maximum number of log entries to buffer in memory
-# when using async persistence strategies (MemFirst/Batched)
-max_buffered_entries = 10000
 
 [raft.metrics]
 enable_backpressure = false

benches/embedded-bench/config/n3.toml

@@ -16,11 +16,7 @@ max_drain = 1024
 max_batch_size = 200
 
 [raft.persistence]
-strategy = "MemFirst"
 flush_policy = { Batch = { idle_flush_interval_ms = 1000 } }
-# Maximum number of log entries to buffer in memory
-# when using async persistence strategies (MemFirst/Batched)
-max_buffered_entries = 10000
 
 [raft.metrics]
 enable_backpressure = false

benches/embedded-bench/src/main.rs

@@ -171,7 +171,7 @@ fn generate_value(size: usize) -> Vec<u8> {
     (0..size).map(|_| rng.random()).collect()
 }
 
-#[tokio::main]
+#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
 async fn main() {
     // Initialize logging
     tracing_subscriber::fmt()
@@ -256,6 +256,7 @@ async fn run_benchmark_task(
 
     let stats = Arc::new(BenchmarkStats::new());
     let key_counter = Arc::new(AtomicU64::new(0));
+    let failed_count = Arc::new(AtomicU64::new(0));
     let start_time = Instant::now();
 
     let mut handles = Vec::with_capacity(clients);
@@ -264,6 +265,7 @@ async fn run_benchmark_task(
         let engine = engine.clone();
         let stats = stats.clone();
         let key_counter = key_counter.clone();
+        let failed_count = failed_count.clone();
         let command = command.clone();
 
         let handle = tokio::spawn(async move {
@@ -289,7 +291,15 @@ async fn run_benchmark_task(
                                     }
                                 }
                             }
-                            Err(_) => continue,
+                            Err(e) => {
+                                let n = failed_count.fetch_add(1, Ordering::Relaxed);
+                                if n < 5 {
+                                    eprintln!("Put failed: {e:?}");
+                                } else if n == 5 {
+                                    eprintln!("Put failed: (further failures suppressed)");
+                                }
+                                continue;
+                            }
                         }
                     }
                     Commands::Get { consistency } => {
@@ -327,6 +337,10 @@ async fn run_benchmark_task(
 
     futures::future::join_all(handles).await;
     stats.summary(start_time.elapsed());
+    let failed = failed_count.load(Ordering::Relaxed);
+    if failed > 0 {
+        println!("Failed requests: {failed}");
+    }
 }
 
 /// Run all benchmark tests in batch mode
@@ -563,6 +577,7 @@ async fn run_local_benchmark(cli: Cli) {
 
             let stats = Arc::new(BenchmarkStats::new());
             let key_counter = Arc::new(AtomicU64::new(0));
+            let failed_count = Arc::new(AtomicU64::new(0));
             let start_time = Instant::now();
 
             let mut handles = Vec::with_capacity(cli.clients);
@@ -571,6 +586,7 @@ async fn run_local_benchmark(cli: Cli) {
                 let engine = engine.clone();
                 let stats = stats.clone();
                 let key_counter = key_counter.clone();
+                let failed_count = failed_count.clone();
                 let cli = cli.clone();
 
                 let handle = tokio::spawn(async move {
@@ -625,8 +641,13 @@ async fn run_local_benchmark(cli: Cli) {
                                             }
                                         }
                                     }
-                                    Err(_) => {
-                                        // Write failed - skip recording
+                                    Err(e) => {
+                                        let n = failed_count.fetch_add(1, Ordering::Relaxed);
+                                        if n < 5 {
+                                            eprintln!("Put failed: {e:?}");
+                                        } else if n == 5 {
+                                            eprintln!("Put failed: (further failures suppressed)");
+                                        }
                                         continue;
                                     }
                                 }
@@ -676,6 +697,10 @@ async fn run_local_benchmark(cli: Cli) {
 
             futures::future::join_all(handles).await;
             stats.summary(start_time.elapsed());
+            let failed = failed_count.load(Ordering::Relaxed);
+            if failed > 0 {
+                println!("Failed requests: {failed}");
+            }
 
             println!("\nBenchmark completed. Press Ctrl+C to shutdown.");
             let _ = shutdown_rx.changed().await;

benches/reports/v0.2.5/bench_report_v0.2.5.md

@@ -59,7 +59,7 @@ _(v0.2.5: 6-round average; v0.2.4: 4-round average; v0.2.3: 4-round average (Lea
 _(v0.2.5: 5-round average; v0.2.4: 5-round average; v0.2.3: 5-round average. All manually collected. 2026-07-12: 4-round average (conns=200, clients=200, Docker monitoring stack stopped).)_
 
 | **Scenario**        | **Metric**  | **v0.2.3**   | **v0.2.4**   | **v0.2.5**   | **Δ (v0.2.4→v0.2.5)** | **0712**      | **Δ (v0.2.5→0712)** |
-| ------------------- | ----------- | ------------ | ------------ | ------------ | --------------------- | ------------ | ------------------- |
+| ------------------- | ----------- | ------------ | ------------ | ------------ | --------------------- | ------------- | ------------------- |
 | Single Client Write | Throughput  | 6,421 ops/s  | 5,245 ops/s  | 5,234 ops/s  | stable                | 9,450 ops/s   | **+80.5%** ✅       |
 |                     | Avg Latency | 0.155 ms     | 0.190 ms     | 0.190 ms     | stable                | 0.105 ms      | **-44.6%** ✅       |
 |                     | p99 Latency | 0.200 ms     | 0.235 ms     | 0.237 ms     | stable                | 0.223 ms      | -6.0% →             |
@@ -214,7 +214,6 @@ read_actor_channel_capacity = 10240
 read_actor_max_drain = 2000
 
 [raft.persistence]
-strategy = "MemFirst"
 flush_policy = { Batch = { idle_flush_interval_ms = 1000 } }
 
 [raft.batching]