61

How can I find which process is constantly writing to disk?

I like my workstation to be close to silent and I just build a new system (P8B75-M + Core i5 3450s -- the 's' because it has a lower max TDP) with quiet fans etc. and installed Debian Wheezy 64-bit on it.

And something is getting on my nerve: I can hear some kind of pattern like if the hard disk was writing or seeking someting (tick...tick...tick...trrrrrr rinse and repeat every second or so).

In the past I had a similar issue in the past (many, many years ago) and it turned out it was some CUPS log or something and I simply redirected that one (not important) logging to a (real) RAM disk.

But here I'm not sure.

I tried the following:

ls -lR /var/log > /tmp/a.tmp && sleep 5 && ls -lR /var/log > /tmp/b.tmp && diff /tmp/?.tmp

but nothing is changing there.

Now the strange thing is that I also hear the pattern when the prompt asking me to enter my LVM decryption passphrase is showing.

Could it be something in the kernel/system I just installed or do I have a faulty harddisk?

hdparm -tT /dev/sda report a correct HD speed (130 GB/s non-cached, sata 6GB) and I've already installed and compiled from big sources (Emacs) without issue so I don't think the system is bad.

(HD is a Seagate Barracude 500GB)

Improve this question
3
  • Are you sure it's a hard drive making that noise, and not something else? (Check the fans, including PSU fan. Had very strange clicking noises once when a very thin cable was too close to a fan and would sometimes very slightly touch the blades and bounce for a few "clicks"...)
    – Mat
    Jul 27, 2012 at 6:03
  • @Mat: I'll take the hard drive outside of the case (the connectors should be long enough) to be sure and I'll report back ; )
    – Cedric Martin
    Jul 27, 2012 at 7:02
  • 2
    Make sure your disk filesystems are mounted relatime or noatime. File reads can be causing writes to inodes to record the access time.
    – camh
    Jul 27, 2012 at 9:48

9 Answers 9

Reset to default
63

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
Improve this answer
3
  • 1
    thanks for that tip. I didn't know about iotop. On Debian I did an apt-cache search iotop to find out that I had to apt-get iotop. Very cool command!
    – Cedric Martin
    Aug 2, 2012 at 15:56
  • 6
    I use iotop -o -b -d 10 which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used.
    – ndemou
    Jun 20, 2016 at 15:32
  • 1
    Unfortunately on new systems it mostly just shows jbd2 as everything goes via the journal
    – krubo
    Oct 8, 2021 at 20:36
17

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog. This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

Improve this answer
7
  • 10
    It is absolutely crazy to leave sysloging enabled when block_dump is active. Logging causes disk activity, which causes logging, which causes disk activity etc. Better stop syslog before enabling this (and use dmesg to read the messages)
    – dan3
    Jul 15, 2013 at 8:32
  • You are absolutely right, although the effect isn't as dramatic as you describe it. If you just want to have a short peek at the disk activity there is no need to stop the syslog daemon.
    – scai
    Jul 16, 2013 at 6:32
  • I've tried it about 2 years ago and it brought my machine to a halt. One of these days when I have nothing important running I'll try it again :)
    – dan3
    Jul 16, 2013 at 7:22
  • I tried it, nothing really happened. Especially because of file system buffering. A write to syslog doesn't immediately trigger a write to disk.
    – scai
    Jul 16, 2013 at 10:50
  • 1
    I would assume there is rate general rate limiting in place for the log messages, which handles this case too(?)
    – Volker Siegel
    Apr 16, 2014 at 22:57
6

You can wack on this a bit. Should narrow it down for most. 

find / -mount -newer /proc -print

Give files modified since boot on the physical device of the / files system. Knowing the files will likely help identify the writer.

Improve this answer
1
  • 1
    A variant of this that worked best for me was: rm -f a; touch a; sleep 30; find / -mount -newer a -print |less
    – Fmstrat
    Jun 23, 2020 at 20:48
6

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem, you can use the audit subsystem (install the auditd package). Put a watch on the sync calls and its friends:

auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log. Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none.

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf: if a log file name is not be preceded by -, then that log is flushed to disk after each write.

Improve this answer
1
  • @StephenKitt Huh. No. The asker mentioned Debian so I've changed it to a link to the Debian package.
    – Gilles 'SO- stop being evil'
    Mar 23, 2018 at 18:24
5

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do
  /sbin/hdparm -q -S 0 "$disk"
done
Improve this answer
6
  • 2
    that you can't query SMART readings without spinning up the drive leaves me speechless :-/ Now obviously the "spinning down" issue can become quite complicated. Regarding disabling the spinning down: wouldn't that in itself cause the HD to wear out faster? I mean: it's never ever "resting" as long as the system is on then?
    – Cedric Martin
    Aug 2, 2012 at 16:03
  • IIRC you can query some SMART values without causing the drive to spin up, but temperature isn't one of them on any of the drives i've tested (incl models from WD, Seagate, Samsung, Hitachi). Which is, of course, crazy because concern over temperature is one of the reasons for idling a drive. re: wear: AIUI 1. constant velocity is less wearing than changing speed. 2. the drives have to park the heads in a safe area and a drive is only rated to do that so many times (IIRC up to a few hundred thousand - easily exceeded if the drive is idling and spinning up every few seconds)
    – cas
    Aug 2, 2012 at 21:42
  • It's a long debate regarding whether it's better to leave drives running or to spin them down. Personally I believe it's best to leave them running - I turn my computer off at night and when I go out but other than that I never spin my drives down. Some people prefer to spin them down, say, at night if they're leaving the computer on or if the computer's idle for a long time, and in such cases the advantage of spinning them down for a few hours versus leaving them running is debatable. What's never good though is when the hard drive repeatedly spins down and up again in a short period of time.
    – micheal65536
    Mar 12, 2016 at 20:48
  • Note also that spinning the drive down after it's been idle for a few hours is a bit silly, because if it's been idle for a few hours then it's likely to be used again within an hour. In that case, it would seem better to spin the drive down promptly if it's idle (like, within 10 minutes), but it's also possible for the drive to be idle for a few minutes when someone is using the computer and is likely to need the drive again soon.
    – micheal65536
    Mar 12, 2016 at 20:51
  • I thought sure this would fix my issue as I hear the drive make a periodic clacking sound (3-4 times/second) like it's writing even when it's not mounted! But I still hear the noise after running this command. Worryingly, it's the drive I use to back up my internal SSD...
    – Michael
    Aug 14, 2019 at 2:19
3

fatrace (File Activity Trace)

fatrace reports file access events (Open, Read, Write, Close) from all running processes.

Its main purpose is to find processes which keep waking up the disk unnecessarily and thus prevent some power saving.

When running it outputs in real time one line per file event in this format:

<timestamp> <process-name(pid)>: <file-event> </path/to/file>

For example:

23:10:21.375341 Plex Media Serv(2290): W /srv/dev-disk-by-uuid-UID/Plex/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log

From which you easily get all the necessary infos

  • WHEN: Timestamp, if using the --timestamp option

  • WHO: Process name (who is causing the activity)

  • WHAT: File operation event (O-pen, R-read, W-rite, C-lose)

  • WHERE: Filepath (where is it writing to).

  • SCOPE You can limit the search scope with --current-mount to only record events on the partition/mount of your current directory.

    • So simply cd into the volume which corresponds to your spinning HDD first, and there run ftrace with the --current-mount option.
    • Without this option, all (real) partitions/mount points are being watched.

Super practical

  • In comparison to all the other answers (as of 2022-07-16) with this tool you can very quickly and efficiently find out which process so far unknown to you is causing what file activity so far unknown to you.
  • All the other tools were not useful for my purposes because they either let you monitor a particular process or a particular directory/file. But not let you "investigate the unknown" in a quick and real time manner.
Improve this answer
2
  • 1
    This is exactly what I was looking for, great tool!
    – Vincent
    Jan 16 at 20:33
  • Glad that it helped you! I hope that over time fatrace will get more popular for its investigative strengths that I tried to outlay in the answer.
    – porg
    Jan 17 at 8:41
1

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

Improve this answer
1
  • You can configure smart daemon not to scan USB disks, most good linux distributions do this by default.
    – lzap
    Jun 15, 2017 at 11:19
1

In case you need to narrow it down to an exact disk use the following:

run lsblk and look up the device number. In the case below it is 9:126

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0   7.3T  0 disk  
└─md126       9:126  0  13.8T  0 raid0 /mnt/InternalPhase
sdb           8:16   0   7.3T  0 disk  
└─md126       9:126  0  13.8T  0 raid0 /mnt/InternalPhase
sdc           8:32   0   7.3T  0 disk  
└─sdc1        8:33   0   7.3T  0 part  /mnt/InternalFBE

run lsof | grep '9,126' with the : replace with , compared to the above disk number. In my case this shows up as:

bash      389162            root  cwd       DIR              9,126      4096  449183796 /mnt/InternalPhase/0000000001/CHANNEL01/LIVE/PHASE/DATA/2018/10/04

with the PID of 389162 kill this process using:

kill -9 389162
Improve this answer
2
0

The problem is that the system needs to flush data from the disk buffers to the disk ever 5 seconds or so by default. Thus if the disk does spin down, there will be little option other than to spin back up again when a flush needs to happen. So the problem is not really avoidable other than by disabling spins downs or disk power management features altogether hdparm -B 255 /dev/hdax. This is probably the better option since restarting so often can definitely be more damaging than simply staying on all the time.

Improve this answer
1
  • 1
    It's only going to flush data if there's any data to flush. If the disk is really not in use, then there isn't going to be any buffered data to flush.
    – micheal65536
    Mar 12, 2016 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.