3

On my system, sar reports an average write to disk about 5MB/s. Using dstat I found that it's not so permanently, but there are peaks to 30-50MB/s, and then are sleeps.

Using iotop i found that most of writing is done by one application (but i don't know how sure it's).

Using strace and fatrace I saw that application opening, creating files in different directories, and then close/remove them. There are many files, but I didn't see in strace much writing. I try to find out more using /proc/pid/fd to see if there are any files constantly incremented (logs, etc) but I couldn't find any. So my conclusion is that all writing is done with small temporary files created by application.

How do I check how much is written to each file? I would like to have a report with the amount of bytes written to each file.

Is there any way to do it? I thought about rewriting syscalls like open and write to monitor changes to every file, or using systemtap but i hope that there is simpler way.

6

Brendan Gregg's iosnoop (part of his perf-tools) will give you detailed information about an application's I/O; for example:

# ./iosnoop
Tracing block I/O... Ctrl-C to end.
COMM             PID    TYPE DEV      BLOCK        BYTES     LATms
supervise        1809   W    202,1    17039968     4096       1.32
supervise        1809   W    202,1    17039976     4096       1.30
tar              14794  RM   202,1    8457608      4096       7.53
tar              14794  RM   202,1    8470336      4096      14.90
tar              14794  RM   202,1    8470368      4096       0.27
tar              14794  RM   202,1    8470784      4096       7.74
tar              14794  RM   202,1    8470360      4096       0.25
tar              14794  RM   202,1    8469968      4096       0.24
tar              14794  RM   202,1    8470240      4096       0.24
[...]

(example taken from his blog post).

You can use it to trace all your system's I/O as above, or limit it to a single process, device...

  • it would be perfect tool, but.. it doesn't give me information about changed files. Most of files after creating disapear, so i think that it's impossible to find where they were created. – undefine May 23 '15 at 20:19
  • Perhaps by combining timestamped strace (-tt to get microseconds) and iosnoop (-ts) output... – Stephen Kitt May 23 '15 at 21:08
  • maybe... i'll check how matching they're. But i'm afraid that no to so - i expect that phisical writing will be after "write" from strace due to cache/buffers/barriers configuration etc. – undefine May 24 '15 at 11:18
4

You may try inotify (often packaged inotify tools since kernel 2.6). It will monitor a part of your filesystem and inform you on events like creating/deleting files or directories.

A simple use would be:

inotifywait -m -r /tmp/

Then, when filesystem activity occurs, you'll see:

#=> I'm creating /tmp/b
/tmp/ CREATE,ISDIR b
/tmp/ OPEN,ISDIR b
/tmp/ ACCESS,ISDIR b
/tmp/ CLOSE_NOWRITE,CLOSE,ISDIR b

man inotifywait and man inotifywatch for more informations.

Beware that using inotify on wide filesystem use a lot of resource...

If you combine it with stat (to see owner for instance) and/or lsof (display all currently opened files and by which process), you may have more hints to know who to blame.

EDIT: An example of inotify and stat combination:

inotifywait -m -r /tmp --format "%w%f" -e modify | while read newfile
do
  echo $newfile $(stat -c "%s" $newfile) bytes
done

This will output something like :

/tmp/oaoa 24 bytes
/tmp/oaoa 24 bytes
/tmp/.com.google.Chrome.iLdh1J 0 bytes
/tmp/.com.google.Chrome.iLdh1J 1472 bytes
  • inotify doesn't give me more information than fatrace - i see only files which are writen to a directory, without information about amount of bytes writen to each of them. – undefine May 23 '15 at 13:08
  • @undefine yes, that's why you need to combine it with something like stat. See my edit for a simple example. – apaul May 23 '15 at 13:39
  • files are changing too fast :-) I try to do somehting like it, but - files disapear before i stat them – undefine May 23 '15 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.