Generate IOPS reporting

[jamesmarkchan]

IOPS is a common measurement of disk performance. Should have IOPS as a metric that can be viewed on the graph as an option. It might be important enough to add as a column on the benchmark tab. Definitely should be stored in the benchmark table.

There is an implementation on a fork of this repo back in 2017 by @SebastianCerquera
https://github.com/SebastianCerquera/jDiskMark

[SebastianCerquera]

I tried to add those metrics, yet I found that it didn't provided the same results as other tools like fio: https://github.com/axboe/fio.

I did some research and I found that it is caused by a Java limitation itself, at least there is some JNI workaround the tool would be misleading.

These are some notes back from 2017, jDiskMark would be a great tool to learn about the JVM, yet it would fail as a benchmark tool:

[jamesmarkchan]

The last v0.5 release has solutions for the disk cache affecting read measurements when running in an administrator terminal or with sudo. Would be really interesting to know how accurate or an estimate we can get this in the java environment or if it is simply required to go into JNI. I wonder how consistent the other tools are on their reporting of IOPS. I know we have some graphing overhead which can affect bw measurements on slower processors it should affect IOPS. @jslcom does believe this is a key measurement to pursue. @SebastianCerquera I'm curious what other tools you had reviewed for checking IOPS measurements?

[jamesmarkchan]

Hi @SebastianCerquera per notes you have shared:

2017-02-24 fio reports twice the number of operations as jdm. Does fio had queue depth? jdm does not have a queue depth and just performs each operation in sequence and perhaps this might account for the difference? Do you recall if the test strictly in writing or both read and write?

2017-02-27 (2) running the first time and second time we are subject to JVM's JIT compiler. Some ways around is priming the code and running a throw away bench to get everything to compile on startup before an actual bench is performed. Another might be to take advantage of an experimental "jaotc" (ahead of time compile) mode introduced in java 9.

I do feel that even a jvm based IOPS measurement will have value even if it does not represent the underlying native IOPS but if the jvm allows us to report the underlying IOPS that would be more consistent with what other tools will report. Perhaps we can provide a jIOPS to indicate it is not a native account of IOPS as the jvm might optimize and translate this to more or less IOPS to optimize performance. Of course JNI would let us get the native metric which would be great.

[jamesmarkchan]

Below is ChatGPT's take on fio vs jdiskmark results. To me the queue depth or jvm IOPS to native IOPS translation is likely the account. I'm still very new to IOPS and how to measure it correctly so interested in learning more.

When benchmarking Input/Output Operations Per Second (IOPS) of storage devices, discrepancies between tools like fio and JDiskMark can arise due to several factors. Each benchmarking tool has its own set of default settings, methodologies, and types of workloads it simulates, which can significantly impact the results. Here are some of the reasons why you might see fio report twice the number of operations compared to JDiskMark:

1. Test Configuration Differences
   Workload Type: fio is highly configurable and can simulate a wide range of workloads (random read/write, sequential read/write, mixed workloads, etc.) with specific block sizes, queue depths, and read/write ratios. JDiskMark, while also configurable, may use different default settings or workload patterns, leading to variations in IOPS.
   Block Size: The size of the data blocks used in the tests can greatly affect IOPS. Smaller block sizes generally lead to higher IOPS because the amount of data transferred per operation is smaller. If fio and JDiskMark are set to use different block sizes, this could explain the difference in reported IOPS.
   Queue Depth: This refers to the number of I/O requests that are allowed to be queued at a time. A higher queue depth can lead to higher IOPS by taking better advantage of the device's capabilities, especially for devices that support Native Command Queuing (NCQ). Differences in the queue depth settings between the two tools could result in different IOPS measurements.
2. Measurement Techniques
   Sampling Method: How each tool measures and calculates IOPS can vary. Some tools may average results over a longer period or use different methods for handling outliers and variability in the data.
   Caching Effects: Differences in how each tool handles or is affected by system and device caches can influence the results. Some benchmarking tools use techniques to minimize caching effects to measure the true performance of the storage device, while others might not, leading to inflated IOPS figures due to cache hits.
3. System and Background Load
   System Load: The state of the system during testing can affect benchmark results. If other processes were running in the background during one test and not the other, this could impact the performance measurements.
   Thermal Throttling: Prolonged intense disk activity can cause some drives to overheat and throttle their performance, potentially affecting benchmark results if one tool runs tests for a longer period than the other.
4. Test Repetition and Averaging
   Repeatability: Benchmark results can vary from run to run due to transient system conditions. It's important to run each benchmark several times and average the results to get a more accurate picture of performance.
   Conclusion
   To compare IOPS measurements accurately, ensure that both fio and JDiskMark (or any two benchmarking tools) are configured as similarly as possible, including workload type, block size, queue depth, and handling of caches. Also, consider running each tool multiple times under similar system conditions to average out transient effects and get a more reliable comparison. Understanding the specific configurations and methodologies of each tool is crucial for interpreting benchmark results correctly.

[SebastianCerquera]

Hey @jamesmarkchan

• Fio allows you to set the queue depth as a parameter, my understanding is that this parameter will tell the program how many workers to use for the IO task. I don't exactly recall the scenarios that I used to validate, those are all the notes that I left last time.
• I wouldn't think that the limitations are due to the JIT, as long as I remember jDiskMark won't start new JVM after each execution, you load the program once and use it several times.
• Given that the performance limitation is related to the JVM we might encounter differences not just in the operations per seconds, but in the bandwidth metric as well.

[SebastianCerquera]

I only checked FIO thoroughly, it solved my issue during that time. I was curious about the differences, and what I found is that the performance differences arise from the way that the utility uses the hardware. I only checked on Linux, I couldn't confirm the same for a Windows box.

FIO leverages the Linux kernel to get the best performance, to avoid caches it will open the file with a O_DIRECT flag, it also will align the memory before performing the system call, I remember getting different results when I removed those optimizations.

[jamesmarkchan]

Okay here is some simple java files based IOPS @SebastianCerquera @jslcom @jchen1ca in case anyone has a moment to tell me if these numbers look okay. It will of course be different from a native benchmark utility bc it is against the java api.

[jamesmarkchan]

This should be complete.