HDFS-17973. Add DataNode write-memory batching to improve write IO pattern throughput and latency - #8714
Open
rdhabalia wants to merge 1 commit into
Open
HDFS-17973. Add DataNode write-memory batching to improve write IO pattern throughput and latency#8714rdhabalia wants to merge 1 commit into
rdhabalia wants to merge 1 commit into
Conversation
rdhabalia
force-pushed
the
datanode-vertical-efficiency
branch
4 times, most recently
from
September 5, 2026 06:06
43e2805 to
e864967
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
datanode-vertical-efficiency
branch
from
September 6, 2026 06:53
e864967 to
b6ce6e3
Compare
|
💔 -1 overall
This message was automatically generated. |
…, throughput and latency
Motivation
----------
On write-heavy clusters the DataNode receives each packet and writes it
straight to the block file. For workloads that issue many small and/or
random writes, this produces a large number of small, scattered disk
writes and pollutes the OS page cache, which in turn hurts concurrent
read latency on the same volumes. Under mixed read/write load the small
write pattern drives high tail latencies and caps aggregate node
throughput well below the underlying disk capability.
This change adds an opt-in, per-replica write-memory buffer on the
DataNode that batches incoming packet data in memory and flushes it to
the block file in large, sequential chunks. Coalescing writes turns many
small IOs into fewer large sequential IOs, reduces page-cache churn, and
significantly improves both write throughput and concurrent read
latency.
Approach
--------
- BufferedBlockWriter (new interface) + BufferedBlockWriterImpl (new)
accumulate a replica's incoming data in a pooled, size-bounded memory
buffer and flush it to disk in large sequential writes. The buffer is
created only for a brand-new RBW replica (recovery/append paths use
the existing direct-write path).
- BlockReceiver routes packet payloads through the buffer when enabled
(buffer.writeData) and otherwise falls back to the direct
streams.writeDataToDisk path. A full buffer is flushed automatically;
hsync and close drain and fsync the buffer. fsync failures on
hsync/close propagate to the caller so bytes are never falsely
acknowledged as durable.
- Read-visibility consistency: because un-synced packet data may still
reside in the in-memory buffer, the RBW replica's visible (acked)
length is never advanced past the bytes actually flushed to the block
file. This guarantees a concurrent reader (BlockSender, short-circuit,
replica-pinned) can never read past the physical end of the file.
Buffered-but-unflushed bytes become visible on the next flush (buffer
full, idle-flush timer, hsync, or close). This slightly relaxes the
immediacy of hflush visibility in exchange for the batched-write IO
pattern; the tradeoff is only taken when the feature is explicitly
enabled.
- A per-DataNode Semaphore bounds the number of concurrent large flushes
across volumes so buffered flushing does not overwhelm the disks; the
concurrency is derived from a configurable per-volume MB budget.
- A periodic idle-flush timer flushes a replica's buffered data if no
packet is received for a configurable interval, bounding how long data
may sit only in memory for slow/idle writers.
- On upstream failure / thread interruption the buffer is flushed to
disk before the receiver thread exits so already-received data is not
lost.
- The per-DataNode memory-cap permit is acquired BEFORE the off-heap
buffer is allocated (so concurrent writers block instead of each
allocating and overshooting the cap), and pooled buffers / permits are
released on any construction failure to avoid leaks.
- A safe-rollout gate ("last-replica-only", default true) restricts the
buffered path to the terminal DataNode in the write pipeline; set to
false to enable it on every replica.
- Read side: an optional read-ahead-cache threshold and drop-cache
behind reads reduce page-cache pressure from large sequential reads
competing with buffered writes.
Configuration (all DataNode-side; feature disabled by default)
--------------------------------------------------------------
dfs.datanode.write.memory.buffer.enabled (default false)
dfs.datanode.write.memory.buffer.last-replica-only (default true)
dfs.datanode.write.memory.buffer.max.capacity.mb
dfs.datanode.write.memory.buffer.min.volumes
dfs.datanode.write.buffer.size.bytes
dfs.datanode.write.buffer.idle.flush.timeout.ms
dfs.datanode.concurrent.flush.mb.per.volume
dfs.datanode.read.ahead.cache.bytes.threshold
Results
-------
On a mixed 80/20 read/write benchmark aggregate node throughput improved
by 30%+ (~2.9 GB/s -> ~3.8 GB/s; ~4.2 GB/s read-only), P50 read transfer
rose above 150 MB/s, read await dropped below ~50 ms, and 128 MB-block
P99 write latency fell from ~22s to ~4s. The feature is off by default
and has no effect on the write/read path until enabled.
rdhabalia
force-pushed
the
datanode-vertical-efficiency
branch
from
September 6, 2026 13:56
b6ce6e3 to
b70fb16
Compare
|
🎊 +1 overall
This message was automatically generated. |
Author
|
@tomscut @virajjasani when you get a chance, could you please help review this? It adds opt-in DataNode write-memory batching to improve write throughput and latency. CI is green (+1 overall). Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
HDFS blocks are processed as smaller packets. Concurrent writes across multiple blocks/files can interleave these packets, creating non-sequential disk I/O.
This is particularly expensive on HDDs, resulting in:
Goal: Improve DataNode vertical efficiency and provide more predictable performance for mixed read/write workloads.
Proposed Solution
Introduce a DataNode-managed in-memory write buffer:
Key Design
O_DIRECTfor buffered flushes.Read Path Benefit
By bypassing the OS page cache for DataNode writes, system memory can be used more effectively for:
Expected benefits:
Key Configuration
dfs.datanode.write.memory.buffer.enableddfs.datanode.write.memory.buffer.last-replica-onlydfs.datanode.write.memory.buffer.max.capacity.mbdfs.datanode.write.buffer.size.bytesdfs.datanode.write.buffer.idle.flush.timeout.msdfs.datanode.concurrent.flush.mb.per.volumeThe feature is disabled by default.
Implementation
The change introduces:
BufferedBlockWriterBufferedBlockWriterImplIntegrated with:
BlockReceiverDataXceiverFsVolumeImplWhen disabled, the existing DataNode write path remains unchanged.
Benchmark Results
Disk-level results also show:
r_await: Reduced from frequently >100 ms to consistently <50 msw_await: Reduced from frequently >200 ms to <90 msExpected Impact
For I/O-bound workloads:
Actual gains depend on workload characteristics and storage hardware. CPU-, network-, or metadata-bound workloads are expected to see limited benefit.
Performance impact and IO Stats
IO stats for a single disk: Before the change, the disk achieved around 120 MBps for certain read-heavy workloads, but the throughput was bursty and inconsistent. After the change, the disk consistently achieved around 170 MBps, as shown in Image 1.
Image 2 compares the read_await time before and after the change. After the change, r_await consistently remains low, enabling higher and more stable read throughput compared to the previous results.
Read, Write, Total Throughput
Read await_time and RPS
Rollout
await, read latency, write P99, CPU, and memoryRecommended Kernel tuning with this change
The following kernel settings can complement the DataNode optimization:
How was this patch tested?
Tested by newly added unit test
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html