As engineers, we have to design computer systems and perform tests on hardware to ensure that specific processes work properly. One of the most common tests is to check the storage system by using third-party software or writing codes to assess their performance and obtain a piece of specific information.
When using our codes, we have the liberty of creating specialized and diversified test suites. But a problem arises, these codes do not allow us to perform an appropriate interpretation and comparison of the results with a well-established reference and widely accepted by the research and engineering community. This way, we have to look for tools that help us perform storage benchmarking.
Out of this need, Jens Boe wrote Fio (Flexible I/O Tester), an open-source benchmarking tool that enables storage testing through workload simulation. Fio currently supports multiple platforms such as Debian, Ubuntu, Red Hat, Fedora, CentOS & Co, Mandriva, Arch Linux, Solaris, Windows, Berkeley Software, and Distributions (BSDs).
I will show you some definitions concerning the storage system that you should know before running Fio. But if you already know them, go to the section Getting started with Fio.
Main Storage Device Metrics
Buffer and Cache
The buffer memory is a region of physical memory that temporarily stores information when data transfer occurs between input and output devices. It also helps to compensate for the speed difference between two processes with different execution times (e.g., sending a document to print).
On the other hand, in most cases, the cache memory is a temporary storage area that stores files for later reading. The primary purpose of the cache is to increase the data transfer performance by storing a copy of a file on it to avoid reading data from an underlying layer of the storage system, thereby reducing read time.
If you want to know the use level of buffer and cache memories of your system, you have to write in the terminal the following command, free -h (I used the -h option for “human-readable” format).
luis@Desktop:~$ free -h
Immediately you will see a general report of the state of use of the memories displayed as follows:
total used free shared buff/cache available
Mem: 15Gi 6.4Gi 9.3Gi 17Mi 223Mi 9.4Gi
Swap: 27Gi 17 Mi 27Gi
where buff/cache is the sum of buffers and cache, please run free –help for more information on the options.
IOPS
The IOPS (Input/Output Operations Per Second) value is a metric for estimating the maximum number of operations that the disk can handle in a one-second interval. IOPS are commonly measured in KiB (kibibyte) or MiB (mebibyte). Their values respond to the type of storage unit, the number of read/write operations, sequential or random access patterns, data block sizes (e.g., 256 blocks of 4 KiB), and other preconditions.
Regarding its construction elements, the number of operations that performs an HDD (Hard-Disk Drive) depends mainly: on the rotational latency that describes the time required to rotate the disk to a specific sector and the seek latency that describes the displacement time of the actuator arm. The sum of both and other parameters results in a speed on the scale of 102 IOPS. On the other hand, an SSD (Solid-State Drive) does not have mechanical elements, reaching higher speeds than a traditional HDD. Typical velocities are on the scale of 103 IOPS.
An SSD is efficient when you want to access a database, move large files or run a program frequently, as it has more capacity to perform operations. On the other hand, an HDD is less expensive and responds better to multitasking processes like running a virus scan, using a program, and opening your web browser simultaneously.
Throughput
Disk throughput represents the speed (in MiB/s) with which the disk can continuously write or read. Whether your app runs on a website, database or server, this is one of the most monitored metrics because it provides an overview of the data transfer rate capability.
Latency
Due to the maximum speed of task processing and construction parameters, the time required to perform a data transfer, or latency, varies according to the disk type. In this way, a magnetic drive will be much slower than a solid-state drive, which implies slower data transfer and lower performance of the entire system.
Bandwidth
Disk bandwidth determines the amount of information a storage unit can send during a data transfer. Its capacity depends on physical and logical connections and other technological limitations.
Getting Started with Fio
Description
Fio is an open-source benchmarking tool that allows standardizing storage system tests and prevents users from writing specific test case codes. It is a versatile I/O workload generator and flexible enough to support detailed setups.
Installing Fio
For Linux-based architectures, the installation instructions are similar. You have to use the command:
luis@Desktop:~$ sudo apt-get install fio
Go to the Fio documentation to learn about the installation process on other platforms.
Fio Input Main Arguments
-- name=str | To run the tests, Fio will create a file according to the specified name. |
--rw=str | Specifies the pattern type to perform the I/O operations. You can choose between the following options: read, write, randread, randwrite, trim, randtrim, rw,readwrite, randrw, trimwrite and randtrimwrite. |
--bs=int | Specifies the block size that the test uses to generate I/O operations. If it is not specified, the default value will be 4k. |
--ioengine=str | Specifies how the job will issue I/O to the test file. Go to the Fio documentation to see other available I/O engines (e.g., libaio is the standard I/O engine in Linux.) |
--size=int | Specifies the file size on which Fio will run the test. If the units are not specified, the default units will be MiB. |
--numjobs=int | Specifies the number of independent threads or processes. By setting a higher value than 1, Fio will perform many processes doing the same thing. (e.g., –numjobs=5, forces Fio to create five separate test files with a size defined in –size=int.) |
--runtime=int | Specifies the total time established to finalize the test. If the process ends before that period of time, Fio will start over until the simulation is complete. The units are in seconds, and the value is usually 60. |
--time_based | If set, it specifies that the test will run repeatedly as many times as runtime allows. |
--group_reporting | If set, Fio displays a report regarding all jobs executed instead of displaying a report of each job. |
--iodepth=int | Number of commands that will be waiting in the queue to be executed by the drive. |
Running Fio
Before running Fio from the terminal (you can also run it from a job file), you must navigate to select the disk or indicate it along the arguments of Fio. To easily recognize the name of your disk, you should use the following command.
luis@Desktop:~$ lsblk
This tool allows you to view details of block devices, which represent the devices connected to the PC.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 400.8M 1 loop /snap/gnome-3-38-2004/112
loop2 7:2 0 62M 1 loop /snap/core20/1587
loop3 7:3 0 163.3M 1 loop /snap/firefox/1635
loop4 7:4 0 91.7M 1 loop /snap/gtk-common-themes/1535
loop5 7:5 0 45.9M 1 loop /snap/snap-store/582
loop6 7:6 0 47M 1 loop /snap/snapd/16292
loop7 7:7 0 284K 1 loop /snap/snapd-desktop-integration/14
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 931G 0 part /
sdb 8:16 1 14.4G 0 disk
└─sdb1 8:17 1 14.4G 0 part /media/luis/LUCHO
sr0 11:0 1 1024M 0 rom
In my case, I have an HDD and an external disk with a capacity of 931.5 GiB (sda) and 14.1 GiB (sdb), respectively. Once you have identified the disk to scan, enter the address in the following way –filename=/dev/sdb.
Note: Make sure the disk name is correct since specifying a wrong disk could cause severe problems with your data.
Finally, you can run Fio and simulate the desired workload by typing the arguments in just a single line.
luis@Desktop:~$ sudo fio --filename=/dev/sdb --rw=read --direct=1 --bs=4k --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
Immediately, Fio will start the test, and after it is completed, it will display a report as shown below.
Seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=28.8MiB/s][r=7371 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=5028: Fri Oct 21 13:20:15 2022
read: IOPS=7329, BW=28.6MiB/s (30.0MB/s)(1718MiB/60002msec)
slat (nsec): min=1550, max=1752.6k, avg=4170.66, stdev=9485.59
clat (usec): min=307, max=15484, avg=2177.77, stdev=222.49
lat (usec): min=329, max=15486, avg=2182.10, stdev=222.55
clat percentiles (usec):
| 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2114],
| 30.00th=[ 2114], 40.00th=[ 2114], 50.00th=[ 2147], 60.00th=[ 2147],
| 70.00th=[ 2212], 80.00th=[ 2245], 90.00th=[ 2311], 95.00th=[ 2442],
| 99.00th=[ 2704], 99.50th=[ 2868], 99.90th=[ 4359], 99.95th=[ 4948],
| 99.99th=[ 7242]
bw ( KiB/s): min=27920, max=30928, per=100.00%, avg=29333.65, stdev=656.56, samples=119
iops : min= 6980, max= 7732, avg=7333.39, stdev=164.13, samples=119
lat (usec) : 500=0.04%, 750=0.04%, 1000=0.11%
lat (msec) : 2=4.13%, 4=95.53%, 10=0.14%, 20=0.01%
cpu : usr=2.39%, sys=4.00%, ctx=54843, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=439765,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=28.6MiB/s (30.0MB/s), 28.6MiB/s-28.6MiB/s (30.0MB/s-30.0MB/s), io=1718MiB (1801MB), run=60002-60002msec
Disk stats (read/write):
sdb: ios=54619/0, merge=384382/0, ticks=118257/0, in_queue=118258, util=99.90%
Describing Fio report
Now, let’s identify the several lines displayed in the report. They will be described individually.
slat (nsec): min=1550, max=1752.6k, avg=4170.66, stdev=9485.59
slat is the sending latency and is characterized by its minimum (min), maximum (max), average (avg), and standard deviation (stdev) values,
clat (usec): min=307, max=15484, avg=2177.77, stdev=222.49
clat denotes the completion latency,
lat (usec): min=329, max=15486, avg=2182.10, stdev=222.55
lat is the total latency (it is the sum of clat and slat),
clat percentiles (usec):
| 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2114],
| 30.00th=[ 2114], 40.00th=[ 2114], 50.00th=[ 2147], 60.00th=[ 2147],
| 70.00th=[ 2212], 80.00th=[ 2245], 90.00th=[ 2311], 95.00th=[ 2442],
| 99.00th=[ 2704], 99.50th=[ 2868], 99.90th=[ 4359], 99.95th=[ 4948],
| 99.99th=[ 7242]
clat percentiles describe the completion latency percentile (probably the most useful info),
bw ( KiB/s): min=27920, max=30928, per=100.00%, avg=29333.65, stdev=656.56, samples=119
bw denotes the bandwidth,
iops : min= 6980, max= 7732, avg=7333.39, stdev=164.13,
iops denotes the IOPS statistics based on samples,
samples=119lat (usec) : 500=0.04%, 750=0.04%, 1000=0.11%lat (msec) : 2=4.13%, 4=95.53%, 10=0.14%, 20=0.01%
lat is the latency distribution in both microseconds (usec) and milliseconds (msec),
cpu : usr=2.39%, sys=4.00%, ctx=54843, majf=0, minf=26
cpu describes the CPU usage,
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts : total=439765,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
IO depths is the number of I/Os issued to the OS at any given time, IO submit and IO complete denote the number of submitted and completed I/Os by Fio in a given time, IO issued rwts and IO latency denote the I/O depth required to meet the specified latency target (these values are used by simulations that employ latency_target and related options.)
Run status group 0 (all jobs):READ: bw=28.6MiB/s (30.0MB/s), 28.6MiB/s-28.6MiB/s (30.0MB/s-30.0MB/s), io=1718MiB (1801MB), run=60002-60002msec
Subsequently, group statistics are printed according to the test. Remember, I performed a reading test (READ). Here, bw=28.6MiB/s is the aggregate bandwidth of threads, followed by its maximum and minimum values (28.6MiB/s-28.6MiB/s), io is the aggregate I/O performed of all threads in this group, run points to the smallest and longest runtimes of the threads.
Note: Fio prints the values in base 2 (MiB/s) and base 10 (MB/s).
Disk stats (read/write): sdb: ios=54619/0, merge=384382/0, ticks=118257/0, in_queue=118258, util=99.90%
Finally, Fio displays statistics for the analyzed disk (sdb). Here, ios is the number of I/Os, merge is the number of merges, ticks is the number of ticks we kept the disk busy (amount of time measured in ticks), in_queue is the total time spent in the disk queue, and util is the percentage of the disk utilization.
Summary
Fio is a widely accepted tool by engineers, and it is also used in infrastructures such as Red Hat, Oracle Cloud, Amazon, and Azure. Fio has cross-platform support, and you can use it to perform benchmark testing to verify storage performance and find bugs.
To learn more about Fio, go to Fio documentation.