HDFS-17975. HDFS Client-Side Block Prefetch to Improve Large Sequential Read Throughput - #8716
Open
rdhabalia wants to merge 1 commit into
Open
HDFS-17975. HDFS Client-Side Block Prefetch to Improve Large Sequential Read Throughput#8716rdhabalia wants to merge 1 commit into
rdhabalia wants to merge 1 commit into
Conversation
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
prefetch-parallel-read
branch
2 times, most recently
from
September 5, 2026 04:25
2a39e3f to
9b42d55
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
prefetch-parallel-read
branch
from
September 6, 2026 07:04
9b42d55 to
6081187
Compare
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
prefetch-parallel-read
branch
from
September 6, 2026 21:00
6081187 to
f037381
Compare
|
💔 -1 overall
This message was automatically generated. |
Motivation ---------- Large sequential scans (analytics/ML training, bulk copies, columnar readers) open a DFSInputStream and read a file end-to-end. Each block is fetched synchronously: the reader thread opens a BlockReader, waits for the DataNode/network round trip, drains the block, then repeats for the next block. The per-block open latency and single-block-at-a-time pipeline leave the client CPU idle while waiting on I/O and cap throughput well below what the DataNodes and network can deliver. This change adds an opt-in, client-side read-ahead prefetcher that fetches blocks ahead of the reader's cursor on a shared background thread pool, so that by the time the reader reaches a block it is already resident in memory and served as an in-memory copy. Approach -------- - BlockPrefetcher (new) maintains a bounded, sliding window of block-sized buffers ahead of the current read position. Blocks are filled in chunks by a shared, JVM-wide daemon thread pool with a SynchronousQueue + AbortPolicy, so that when all workers are busy a prefetch submission is rejected and skipped rather than run on (and block) the foreground reader thread. - DFSInputStream.read(byte[]) and read(ByteBuffer) first try to serve from the prefetch cache; on a hit the position is advanced, the stateful synchronous reader is invalidated, and read statistics are updated exactly as on the direct path (locality-aware: short-circuit / local / remote). On a miss it falls back to the normal synchronous read, which also caches block locations for a later prefetch. - Prefetch runs entirely off cached block locations; it never issues a getBlockLocations RPC or mutates foreground retry state while a foreground read may hold infoLock. Non-uniform block sizes, striped (EC) files, under-construction files and single-block files are excluded for correctness. - A shared, opt-in scheduled task logs per-stream cache hit/miss ratios at INFO (only when metrics logging is enabled) for observability. - A global byte budget bounds total memory held across all prefetch buffers in the JVM; it grows to the largest configured value across clients and is strictly reserved/released per stream. Configuration (all client-side; feature disabled by default) ------------------------------------------------------------ dfs.client.prefetch.enabled (default false) dfs.client.prefetch.size per-stream read-ahead window dfs.client.prefetch.max.bytes JVM-wide prefetch memory cap dfs.client.prefetch.chunk.size fill granularity dfs.client.prefetch.threads / dfs.client.prefetch.threadpool.size shared prefetch worker threads dfs.client.prefetch.ttl.ms buffer time-to-live dfs.client.prefetch.metrics.log.enabled periodic hit-ratio logging Results ------- On a single-client sequential read benchmark (10 GB files, 512 MB blocks, 5 prefetch threads, ~3 GB prefetch window) average read throughput improved ~3.49x (406.8 MB/s -> 1419.5 MB/s), peaking at ~1.58 GB/s, with a cache hit ratio of ~81%. The feature is off by default and has no effect on the read path until enabled.
rdhabalia
force-pushed
the
prefetch-parallel-read
branch
from
September 6, 2026 21:13
f037381 to
ad3c414
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Large sequential scans, such as analytics, bulk copies, and
columnar readers, open a
DFSInputStreamand read a file end-to-end.Today, each block is fetched synchronously:
BlockReader.The per-block open latency and single-block-at-a-time pipeline leave the
client CPU idle while waiting on I/O and cap throughput well below what the
DataNodes and network can deliver.
This change adds an opt-in, client-side read-ahead prefetcher that fetches
blocks ahead of the reader's cursor using a shared background thread pool.
By the time the reader reaches a prefetched block, the block is already
resident in memory and can be served as an in-memory copy.
Approach
BlockPrefetcher
BlockPrefetcherthat maintains a bounded, sliding windowof block-sized buffers ahead of the current read position.
SynchronousQueueAbortPolicyrather than blocking the foreground reader thread.
DFSInputStream Integration
DFSInputStream.read(byte[])andread(ByteBuffer)first attempt to servedata from the prefetch cache.
Prefetch Safety and Correctness
getBlockLocationsRPC.infoLock.Metrics and Observability
hit/miss ratios at
INFOlevel.Global Memory Budget
across the JVM.
Configuration
All configuration is client-side. The feature is disabled by default.
dfs.client.prefetch.enableddfs.client.prefetch.sizedfs.client.prefetch.max.bytesdfs.client.prefetch.chunk.sizedfs.client.prefetch.threadsdfs.client.prefetch.threadpool.sizedfs.client.prefetch.ttl.msdfs.client.prefetch.metrics.log.enabledResults
A single-client sequential read benchmark was run with:
Performance
The feature is off by default and has no impact on the read path until
explicitly enabled.
How was this patch tested?
Tested by newly added unit test
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html