I'd like to do some general disk io monitoring on a debian linux server. What are the tools I should know about that monitor disk io so I can see if a disk's performance is maxed out or spikes at certain time throughout the day?
For disk I/O trending there are a few options. My personal favorite is the
sar command from
sysstat. By default, it gives output like this:
09:25:01 AM CPU %user %nice %system %iowait %steal %idle 09:35:01 AM all 0.11 0.00 0.01 0.00 0.00 99.88 09:45:01 AM all 0.12 0.00 0.01 0.00 0.00 99.86 09:55:01 AM all 0.09 0.00 0.01 0.00 0.00 99.90 10:05:01 AM all 0.10 0.00 0.01 0.02 0.01 99.86 Average: all 0.19 0.00 0.02 0.00 0.01 99.78
%iowait is the time spent waiting on I/O. Using the Debian package, you must enable the stat collector via the
/etc/default/sysstat config file after package installation.
To see current utilization broken out by device, you can use the
iostat command, also from the sysstat package:
$ iostat -x 1 Linux 3.5.2-x86_64-linode26 (linode) 11/08/2012 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.84 0.00 0.08 1.22 0.07 97.80 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda 0.09 1.02 2.58 0.49 112.79 12.11 40.74 0.15 48.56 3.88 1.19 xvdb 1.39 0.43 4.03 1.82 43.33 18.43 10.56 0.66 112.73 1.93 1.13
Some other options that can show disk usage in trending graphs is munin and cacti.
Munin or Cacti are more what I'm looking for. Thanks for the pointers. Nov 9, 2012 at 4:00
15Note that %iowait only shows time when the kernel had nothing else to do but wait for IO. On a 100% CPU bound server this will be zero regardless how much IO it is doing. Apr 28, 2014 at 14:19
iostat -x dm-0 | grep Device; iostat -xd 2 -y dm-0 | grep dm-0 Apr 17, 2018 at 19:02
sar, one can simply do
"true"save the file and then do a
sudo service sysstat restart– HosseinMar 2, 2021 at 3:50
sar: command not foundSep 1, 2021 at 12:47
Have a look at
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init splash 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq] 7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
iodump, if that's more down your way of thinking.
Note: This requires at least kernel 2.6.20 to work.
This is not just Linux only, but also requires a newer (or patched) kernel.– jordanmNov 8, 2012 at 18:45
These look good for finding out what's going on right now, but I'm looking for bigger picture monitoring. Nov 9, 2012 at 4:01
14@camomileCase - you didn't specify that in your question ...– tinkJun 17, 2016 at 10:16
on ubuntu 18.04 iotop crashes with messages
OSError: Netlink error: No such file or directory (2)Sep 1, 2021 at 12:49
@user7860670 - I can't confirm this; a cursory search suggests that there's a problem with iotop in containers - are you running docker or something?– tinkSep 1, 2021 at 16:54
dstat. It can show totals and statistics per disk and even md-devices (RAID), also can use colors for better overview:
$ dstat -tdD total,sda,sdb,sdc,md1 60 ----system---- -dsk/total----dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/md1-- time | read writ: read writ: read writ: read writ: read writ 08-11 22:08:17|3549k 277k: 144k 28k: 851k 62k: 852k 60k: 25k 82k 08-11 22:09:17| 60k 258k:1775B 15k: 13k 63k: 15k 60k: 68B 74k 08-11 22:10:17| 176k 499k: 0 14k: 41k 122k: 41k 125k: 273B 157k 08-11 22:11:17| 42k 230k: 0 14k:9830B 54k: 14k 51k: 0 70k 08-11 22:11:52| 28k 132k: 0 5032B:5266B 33k:9479B 28k: 0 37k
-dfor disk statistics
-Dto specify the exact devices to report
60to average over 60 seconds. The display is updated every second, but only once per 60 seconds a new line will be started.
not used in this example, but
-ccan report wait IO percentage, which in most cases is related to the CPU waiting for data from the disks.
It is available for most Linux distributions, but sometimes needs to be installed from repositories.
I used this, and don't get the 'k' suffix, just numbers like 135 read 157 writ. What are the units here? Megabytes?– BramMay 11, 2021 at 1:31
dstat: command not foundSep 1, 2021 at 12:49
@user7860670 that means it's not installed... Run
<apt/yum/dnf> search dstatto find the package name and install it first.– cyqsimonNov 25, 2021 at 17:45
Just from files on your computer (assuming a recent kernel) you can ask
/sys/block/sda/stat (documented at kernel.org) or
/proc/diskstats. It will need some translating, however. But it's nice for a quick and dirty check.
4Big advantage for this answer is that no tools need to be installed. It's just there. (Potentially :-) Feb 3, 2017 at 5:36
4e.g. echo "read io merge sectors ticks write io merges sectors ticks inflight ioticks timeinqueu";cat /sys/block/vda/vda1/stat ;sleep 10; cat /sys/block/vda/vda1/stat Feb 6, 2017 at 0:07
Another option is to use:
sudo iotop -aoP
-a Will show accumulated output -o Will only output -P Will only show processes instead of threads
This program will tell you how much a process has written to disk and read from disk since iotop was started.
I hear something accessing the disk, but accumulated output is increasing very slowly, sometimes not at all in a second... ???– MichaelDec 11, 2018 at 1:13
If you hear the disk, it's probably old and/or slow. Modern HDDs tend to be very quiet. What kind of sounds is it making? Use
smartctlto see whether the disk is healthy. Jun 21, 2019 at 7:08
Monitoring Disk Io can be done by multiple tools like the following.
Also some important operating system concepts are very much necessary to comprehend them..read the Linux IO complete tutorial
3iozone is for testing, not monitoring: "Iozone is a filesystem benchmark tool. The benchmark generates and measures a variety of file operations." Feb 16, 2016 at 22:28
Why not try nmon. It does disks and network with processes. Originally for AIX but it has been ported to Linux for some years now.
Another great tool for a quick overview where the load comes from is
It can show you an overview over all resources (CPU, memory/swap, network and disk I/O) or you can drill down to a single resource and sort processes by how much they consume.
I am the author of
diskgraph which is a command line tool that plots the disk IO in a terminal. You select which disk, as an argument on the command-line, like
$ ./diskgraph nvme0n1
In red: the write bandwidth
In green: the read bandwidth
In orange: the number of IO operations that are in-flight.
Please note that this utility depends on features introduced in Linux 5. In other words, it will not work with Linux 4 or earlier.
If you use
zfs, then there is a built in tool for this:
% zpool iostat -T d -n -H -v 1 | head -5 Sat Dec 24 23:09:49 2022 datatank 24.8T 31.6T 39 70 5.76M 4.55M mirror-0 4.29T 4.80T 3 9 802K 765K gpt/slot_A1 - - 2 4 511K 403K gpt/slot_B7 - - 1 4 335K 405K
-T doption puts the date lines (e.g Dec 24).
-Henables scripted mode by omitting headers and making the data well formed.
-venables statistics for each individual vdev in the pool.
- Finally the 1 is how many seconds to wait for the next update.
head -5is just to demonstrate you can pipe the output.
The best thing to do if you want to see performance on a long period is to use Centreon which is a powerful tools build on Nagios. Centreon make you able to monitor by drawing graphs of resource's performances and lot of other things.
3What other things? Any screenshots? Please expand your answer. Jun 24, 2019 at 18:06