Performing Storage Benchmarking Tests with FIO

Table of Contents

As engineers, we have to design computer systems and perform tests on hardware to ensure that specific processes work properly. One of the most common tests is to check the storage system by using third-party software or writing codes to assess their performance and obtain a piece of specific information.

When using our codes, we have the liberty of creating specialized and diversified test suites. But a problem arises, these codes do not allow us to perform an appropriate interpretation and comparison of the results with a well-established reference and widely accepted by the research and engineering community. This way, we have to look for tools that help us perform storage benchmarking.

Out of this need, Jens Boe wrote Fio (Flexible I/O Tester), an open-source benchmarking tool that enables storage testing through workload simulation. Fio currently supports multiple platforms such as Debian, Ubuntu, Red Hat, Fedora, CentOS & Co, Mandriva, Arch Linux, Solaris, Windows, Berkeley Software, and Distributions (BSDs).

I will show you some definitions concerning the storage system that you should know before running Fio. But if you already know them, go to the section Getting started with Fio.

Main Storage Device Metrics 

Buffer and Cache 

The buffer memory is a region of physical memory that temporarily stores information when data transfer occurs between input and output devices. It also helps to compensate for the speed difference between two processes with different execution times (e.g., sending a document to print).

On the other hand, in most cases, the cache memory is a temporary storage area that stores files for later reading. The primary purpose of the cache is to increase the data transfer performance by storing a copy of a file on it to avoid reading data from an underlying layer of the storage system, thereby reducing read time.

If you want to know the use level of buffer and cache memories of your system, you have to write in the terminal the following command, free -h (I used the -h option for “human-readable” format).

luis@Desktop:~$ free -h

Immediately you will see a general report of the state of use of the memories displayed as follows:

               total     used     free    shared   buff/cache   available
Mem:            15Gi     6.4Gi   9.3Gi       17Mi       223Mi       9.4Gi
Swap:           27Gi     17 Mi   27Gi

where buff/cache is the sum of buffers and cache, please run free –help for more information on the options.


The IOPS (Input/Output Operations Per Second) value is a metric for estimating the maximum number of operations that the disk can handle in a one-second interval. IOPS are commonly measured in KiB (kibibyte) or MiB (mebibyte). Their values respond to the type of storage unit, the number of read/write operations, sequential or random access patterns, data block sizes (e.g., 256 blocks of 4 KiB), and other preconditions.

Regarding its construction elements, the number of operations that performs an HDD (Hard-Disk Drive) depends mainly: on the rotational latency that describes the time required to rotate the disk to a specific sector and the seek latency that describes the displacement time of the actuator arm. The sum of both and other parameters results in a speed on the scale of 102 IOPS. On the other hand, an SSD (Solid-State Drive) does not have mechanical elements, reaching higher speeds than a traditional HDD. Typical velocities are on the scale of 103 IOPS.

An SSD is efficient when you want to access a database, move large files or run a program frequently, as it has more capacity to perform operations. On the other hand, an HDD is less expensive and responds better to multitasking processes like running a virus scan, using a program, and opening your web browser simultaneously.


Disk throughput represents the speed (in MiB/s) with which the disk can continuously write or read. Whether your app runs on a website, database or server, this is one of the most monitored metrics because it provides an overview of the data transfer rate capability.


Due to the maximum speed of task processing and construction parameters, the time required to perform a data transfer, or latency, varies according to the disk type. In this way, a magnetic drive will be much slower than a solid-state drive, which implies slower data transfer and lower performance of the entire system.


Disk bandwidth determines the amount of information a storage unit can send during a data transfer. Its capacity depends on physical and logical connections and other technological limitations.

Getting Started with Fio


Fio is an open-source benchmarking tool that allows standardizing storage system tests and prevents users from writing specific test case codes. It is a versatile I/O workload generator and flexible enough to support detailed setups.

Installing Fio

For Linux-based architectures, the installation instructions are similar. You have to use the command:

luis@Desktop:~$ sudo apt-get install fio

Go to the Fio documentation to learn about the installation process on other platforms.

Fio Input Main Arguments

-- name=strTo run the tests, Fio will create a file according to the specified name.
--rw=strSpecifies the pattern type to perform the I/O operations. You can choose between the following options: read, write, randread, randwrite, trim, randtrim, rw,readwrite, randrw, trimwrite and randtrimwrite.
--bs=intSpecifies the block size that the test uses to generate I/O operations. If it is not specified, the default value will be 4k.
--ioengine=strSpecifies how the job will issue I/O to the test file. Go to the Fio documentation to see other available I/O engines (e.g., libaio is the standard I/O engine in Linux.)
--size=intSpecifies the file size on which Fio will run the test. If the units are not specified, the default units will be MiB.
--numjobs=intSpecifies the number of independent threads or processes. By setting a higher value than 1, Fio will perform many processes doing the same thing. (e.g., –numjobs=5, forces Fio to create five separate test files with a size defined in –size=int.)
--runtime=intSpecifies the total time established to finalize the test. If the process ends before that period of time, Fio will start over until the simulation is complete. The units are in seconds, and the value is usually 60.
--time_basedIf set, it specifies that the test will run repeatedly as many times as runtime allows.
--group_reportingIf set, Fio displays a report regarding all jobs executed instead of displaying a report of each job.
--iodepth=intNumber of commands that will be waiting in the queue to be executed by the drive.

Running Fio

Before running Fio from the terminal (you can also run it from a job file), you must navigate to select the disk or indicate it along the arguments of Fio. To easily recognize the name of your disk, you should use the following command.

luis@Desktop:~$ lsblk

This tool allows you to view details of block devices, which represent the devices connected to the PC.

loop0    7:0    0     4K  1 loop /snap/bare/5
loop1    7:1    0 400.8M  1 loop /snap/gnome-3-38-2004/112
loop2    7:2    0    62M  1 loop /snap/core20/1587
loop3    7:3    0 163.3M  1 loop /snap/firefox/1635
loop4    7:4    0  91.7M  1 loop /snap/gtk-common-themes/1535
loop5    7:5    0  45.9M  1 loop /snap/snap-store/582
loop6    7:6    0    47M  1 loop /snap/snapd/16292
loop7    7:7    0   284K  1 loop /snap/snapd-desktop-integration/14
sda      8:0    0 931.5G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0   931G  0 part /
sdb      8:16   1  14.4G  0 disk
└─sdb1   8:17   1  14.4G  0 part /media/luis/LUCHO
sr0     11:0    1  1024M  0 rom

In my case, I have an HDD and an external disk with a capacity of 931.5 GiB (sda) and 14.1 GiB (sdb), respectively. Once you have identified the disk to scan, enter the address in the following way –filename=/dev/sdb.

Note: Make sure the disk name is correct since specifying a wrong disk could cause severe problems with your data.

Finally, you can run Fio and simulate the desired workload by typing the arguments in just a single line.

luis@Desktop:~$ sudo fio --filename=/dev/sdb --rw=read --direct=1 --bs=4k --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16

Immediately, Fio will start the test, and after it is completed, it will display a report as shown below.

Seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=28.8MiB/s][r=7371 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=5028: Fri Oct 21 13:20:15 2022
  read: IOPS=7329, BW=28.6MiB/s (30.0MB/s)(1718MiB/60002msec)
    slat (nsec): min=1550, max=1752.6k, avg=4170.66, stdev=9485.59
    clat (usec): min=307, max=15484, avg=2177.77, stdev=222.49
     lat (usec): min=329, max=15486, avg=2182.10, stdev=222.55
    clat percentiles (usec):
     |  1.00th=[ 1926],  5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2114],
     | 30.00th=[ 2114], 40.00th=[ 2114], 50.00th=[ 2147], 60.00th=[ 2147],
     | 70.00th=[ 2212], 80.00th=[ 2245], 90.00th=[ 2311], 95.00th=[ 2442],
     | 99.00th=[ 2704], 99.50th=[ 2868], 99.90th=[ 4359], 99.95th=[ 4948],
     | 99.99th=[ 7242]
   bw (  KiB/s): min=27920, max=30928, per=100.00%, avg=29333.65, stdev=656.56, samples=119
   iops        : min= 6980, max= 7732, avg=7333.39, stdev=164.13, samples=119
  lat (usec)   : 500=0.04%, 750=0.04%, 1000=0.11%
  lat (msec)   : 2=4.13%, 4=95.53%, 10=0.14%, 20=0.01%
  cpu          : usr=2.39%, sys=4.00%, ctx=54843, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=439765,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: bw=28.6MiB/s (30.0MB/s), 28.6MiB/s-28.6MiB/s (30.0MB/s-30.0MB/s), io=1718MiB (1801MB), run=60002-60002msec
Disk stats (read/write):
  sdb: ios=54619/0, merge=384382/0, ticks=118257/0, in_queue=118258, util=99.90%

Describing Fio report

Now, let’s identify the several lines displayed in the report. They will be described individually.

slat (nsec): min=1550, max=1752.6k, avg=4170.66, stdev=9485.59

slat is the sending latency and is characterized by its minimum (min), maximum (max), average (avg), and standard deviation (stdev) values,

clat (usec): min=307, max=15484, avg=2177.77, stdev=222.49

clat denotes the completion latency,

lat (usec): min=329, max=15486, avg=2182.10, stdev=222.55

lat is the total latency (it is the sum of clat and slat),

clat percentiles (usec):     
|  1.00th=[ 1926],  5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2114],     
| 30.00th=[ 2114], 40.00th=[ 2114], 50.00th=[ 2147], 60.00th=[ 2147],     
| 70.00th=[ 2212], 80.00th=[ 2245], 90.00th=[ 2311], 95.00th=[ 2442],     
| 99.00th=[ 2704], 99.50th=[ 2868], 99.90th=[ 4359], 99.95th=[ 4948],     
| 99.99th=[ 7242]

clat percentiles describe the completion latency percentile (probably the most useful info),

bw (  KiB/s): min=27920, max=30928, per=100.00%, avg=29333.65, stdev=656.56, samples=119

bw denotes the bandwidth,

iops        : min= 6980, max= 7732, avg=7333.39, stdev=164.13, 

iops denotes the IOPS statistics based on samples,

samples=119lat (usec)   : 500=0.04%, 750=0.04%, 1000=0.11%lat (msec)   : 2=4.13%, 4=95.53%, 10=0.14%, 20=0.01%

lat is the latency distribution in both microseconds (usec) and milliseconds (msec),

cpu          : usr=2.39%, sys=4.00%, ctx=54843, majf=0, minf=26

cpu describes the CPU usage, 

IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%   
submit       : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%   
complete     : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%  
issued rwts  : total=439765,0,0,0 short=0,0,0,0 dropped=0,0,0,0   
latency      : target=0, window=0, percentile=100.00%, depth=16

IO depths is the number of I/Os issued to the OS at any given time, IO submit and IO complete denote the number of submitted and completed I/Os by Fio in a given time, IO issued rwts and IO latency denote the I/O depth required to meet the specified latency target (these values are used by simulations that employ  latency_target and related options.)

Run status group 0 (all jobs):READ: bw=28.6MiB/s (30.0MB/s), 28.6MiB/s-28.6MiB/s (30.0MB/s-30.0MB/s), io=1718MiB (1801MB), run=60002-60002msec

Subsequently, group statistics are printed according to the test. Remember, I performed a reading test (READ). Here, bw=28.6MiB/s is the aggregate bandwidth of threads, followed by its maximum and minimum values (28.6MiB/s-28.6MiB/s), io is the aggregate I/O performed of all threads in this group, run points to the smallest and longest runtimes of the threads. 

Note: Fio prints the values in base 2 (MiB/s) and base 10 (MB/s).

Disk stats (read/write):  sdb: ios=54619/0, merge=384382/0, ticks=118257/0, in_queue=118258, util=99.90%

Finally, Fio displays statistics for the analyzed disk (sdb). Here, ios is the number of I/Os, merge is the number of merges, ticks is the number of ticks we kept the disk busy (amount of time measured in ticks), in_queue is the total time spent in the disk queue, and util is the percentage of the disk utilization.


Fio is a widely accepted tool by engineers, and it is also used in infrastructures such as Red Hat, Oracle Cloud, Amazon, and Azure. Fio has cross-platform support, and you can use it to perform benchmark testing to verify storage performance and find bugs.

To learn more about Fio, go to Fio documentation.

Scroll to Top