HDFS-17977. Add a compact standalone HDFS DataNode read/write stress tester - #8719
Open
rdhabalia wants to merge 1 commit into
Open
HDFS-17977. Add a compact standalone HDFS DataNode read/write stress tester#8719rdhabalia wants to merge 1 commit into
rdhabalia wants to merge 1 commit into
Conversation
Motivation:
TestDFSIO is the standard HDFS I/O benchmark, but it is a poor fit for
targeted DataNode stress testing:
- It launches a MapReduce job scheduled by YARN across the whole cluster, so
it cannot generate a controlled, steady QPS/throughput and it takes a long
time to saturate a targeted subset of nodes.
- It cannot target a specific set of DataNodes (e.g. one replica set), which is
exactly what is needed to reproduce and measure a hot/overloaded node.
- It reports aggregate throughput only, with no client-side latency
distributions (p50/p95/p99), which are essential to characterise tail
latency and disk-random-IO behaviour.
- It requires a MapReduce/YARN stack to be available at all.
Approach:
Add HdfsStressTest, a single-process, no-MapReduce load generator. Because it
depends only on the HDFS client (DistributedFileSystem plus favored-nodes) and
needs no MapReduce/YARN, it lives in the hadoop-hdfs test tree
(org.apache.hadoop.hdfs), alongside the other standalone HDFS benchmarks such
as BenchmarkThroughput, rather than in the MapReduce jobclient module. It is
intentionally compact (one tool class plus small nested helpers) and provides
the controls TestDFSIO lacks:
- Controlled read/write throughput (MB/s) enforced by a global token-bucket
rate limiter, with an optional linear ramp (start -> end throughput) for
acceleration stress testing.
- Targeted DataNodes via HDFS favored-nodes hints so load lands on a chosen
replica set.
- Configurable block/file size.
- A pre-test phase that pre-creates a large corpus of files (sized to exceed
the DataNode page cache, e.g. ~2x DataNode memory). Because the corpus does
not fit in main memory, the kernel evicts older pages as newer blocks are
written, so by the time the measured phase re-reads a file its pages are no
longer cached and every measured read falls through to disk (a true cold
read). This is how the tool avoids page-cache reads and measures real disk
behaviour.
- Client-side latency distributions (p50, p75, p95, p99, min, max, mean,
stddev) and effective QPS/throughput for both reads and writes.
Aggregate load can be scaled by running the tool on multiple client hosts at
once against the same cluster; the offered load is the sum of each client's
configured throughput. This is documented along with the per-client guidance
(distinct write/read directories, shared favoredDataNodes, overlapping runs).
Workload is described by a properties file (favoredDataNodes,
testWriteDirectory, testReadDirectories, blockSizeMB, testDurationSeconds,
writeThroughputMB, readThroughputMB, testReadFileSizeGB,
preTestWriteThroughputMB, preTestWriteDurationSeconds, optional
endWriteThroughputMB/endReadThroughputMB for ramping). Any property may be
overridden with -D via ToolRunner.
Documentation:
- Every configuration property, the run command, the cold-read/page-cache
rationale and the multi-client scale-out guidance are documented both in the
class javadoc and in a new user guide,
hadoop-hdfs/src/site/markdown/HdfsStressTest.md (linked from the HDFS site
menu). Run example:
hadoop jar hadoop-hdfs-<version>-tests.jar \
org.apache.hadoop.hdfs.HdfsStressTest /path/to/stress.properties
Validation:
- loadConfig fails fast on misconfigurations that would otherwise produce
misleading results: blockSizeMB must be a positive number of MB (it is the
I/O unit and a divisor when sizing the corpus and computing per-op rates),
and enabling the read workload (readThroughputMB > 0) requires a positive
testReadFileSizeGB, since reads are served only from the pre-test corpus and
would otherwise silently run zero readers yet still exit success.
- The pre-test phase writes a corpus of files sized to exceed the DataNode
page cache "once", so re-reads fall through to disk; the read workload
spreads its picks across the whole corpus rather than replaying a hot subset,
keeping the page-cache hit rate near zero. If the corpus is time-boxed with
preTestWriteDurationSeconds and stops short of testReadFileSizeGB, the tool
prints a WARNING that the reads may be served from the page cache (so the
numbers can be optimistic) instead of silently reporting warm reads as cold.
Testing:
- TestHdfsStressTest (MiniDFSCluster) validates the individual building blocks
and the end-to-end Tool run: writeBlockFile produces an exactly block-sized
file with the configured replication/block size and payload; readWholeFile
reads a full block to EOF; the pre-test phase creates a block-sized cold-read
corpus round-robined across multiple directories and honours the duration
cap (and warns on stderr when the capped corpus is smaller than requested);
and a write-only run drives the write path through ToolRunner without
creating a read corpus.
- TestHdfsStressTestHelpers adds fast, cluster-free unit tests for the pacing
and measurement helpers (token-bucket RateLimiter spacing, runtime rate
changes, LatencyStats nanos->millis sorting and bounded-memory reservoir
sampling) and for config validation: blockSizeMB must be positive, enabling
the read workload requires a positive testReadFileSizeGB (otherwise the run
would silently do no reads), and valid read-only / write-only configs load
without error.
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
TestDFSIOis the standard HDFS I/O benchmark, but it is a poor fit fortargeted DataNode stress testing:
it cannot generate controlled, steady QPS/throughput and takes a long time
to saturate a targeted subset of nodes.
which is required to reproduce and measure hot or overloaded nodes.
latency distributions such as p50/p95/p99, which are essential for
characterizing tail latency and disk random-I/O behavior.
Approach
Add
HdfsStressTest, a single-process, no-MapReduce load generator.Because it depends only on the HDFS client (
DistributedFileSystemplusfavored-nodes) and does not require MapReduce/YARN, it lives in the
hadoop-hdfstest tree (org.apache.hadoop.hdfs), alongside other standaloneHDFS benchmarks such as
BenchmarkThroughput, rather than in the MapReducejobclient module.
The tool is intentionally compact, consisting of one tool class with small
nested helpers, and provides the controls that
TestDFSIOlacks:Controlled read/write throughput
acceleration stress testing.
Targeted DataNodes
replica set.
Configurable block and file size
Cold-read testing
DataNode page cache, for example, approximately 2x DataNode memory.
pages as newer blocks are written.
expected to be cached, causing the measured read to fall through to disk.
than replaying a hot subset, keeping the page-cache hit rate near zero.
page-cache performance.
Client-side latency and throughput metrics
Multi-Client Scale-Out
Aggregate load can be scaled by running the tool concurrently on multiple
client hosts against the same cluster.
The offered load is the sum of each client's configured throughput.
Multi-client usage and guidance are documented, including:
favoredDataNodesconfiguration.Configuration
Workload configuration is specified through a properties file.
Supported properties include:
favoredDataNodestestWriteDirectorytestReadDirectoriesblockSizeMBtestDurationSecondswriteThroughputMBreadThroughputMBtestReadFileSizeGBpreTestWriteThroughputMBpreTestWriteDurationSecondsendWriteThroughputMBendReadThroughputMBAny property can be overridden using
-DthroughToolRunner.Documentation
Every configuration property, the run command, cold-read/page-cache
rationale, and multi-client scale-out guidance are documented in:
HdfsStressTestclass Javadoc.hadoop-hdfs/src/site/markdown/HdfsStressTest.mdRun Example
Validation
loadConfigfails fast on misconfigurations that could otherwise producemisleading results.
Block Size Validation
blockSizeMBmust be a positive number of MB.The block size is used as the I/O unit, as well as a divisor when sizing the
corpus and calculating per-operation rates.
Read Workload Validation
When the read workload is enabled:
testReadFileSizeGBmust also be positive.Reads are served only from the pre-test corpus. Without a valid corpus size,
the tool could silently run zero readers and still exit successfully.
Cold-Read Corpus Validation
The pre-test phase creates a corpus large enough to exceed the DataNode page
cache so that subsequent reads fall through to disk.
The read workload distributes file selections across the entire corpus rather
than repeatedly reading a hot subset, keeping the page-cache hit rate near
zero.
If the pre-test is time-boxed using
preTestWriteDurationSecondsand stopsbefore reaching
testReadFileSizeGB, the tool prints aWARNING.The warning indicates that some reads may be served from the page cache and
that the resulting numbers could therefore be optimistic, rather than
silently reporting warm reads as cold.
Testing
TestHdfsStressTestUses
MiniDFSClusterto validate individual building blocks and theend-to-end
Toolexecution.The test verifies:
writeBlockFilereadWholeFilePre-test phase
requested read corpus size.
Write-only workload
ToolRunner.TestHdfsStressTestHelpersProvides fast, cluster-free unit tests for pacing, measurement, and
configuration helpers.
Rate Limiter
Tests:
Latency Statistics
Tests:
Configuration Validation
Tests:
blockSizeMBmust be positive.testReadFileSizeGB.These validations prevent configurations that would otherwise silently perform
no reads or produce misleading benchmark results.
How was this patch tested?
Tested by newly added unit test
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html