How can I print current reads and writes to the filesystem on demand?

Question

I am interested in getting statistics about filesystem reads and writes, specifically to an NFS mount.

I know there are many tools such as iostat, nfsstat and inotifywait but there are issues I have with each program.

I cannot use anything that would require me to walk the filesystem or explicitly mention file names due to the complexity of the hierarchy.

Is there a program that can do the following on demand?

1. show reads / writes
2. show path of read / write
3. show byte quantity

By on demand, I mean on execution time. Point (3) is not mandatory.

For example, I would like to be able to do something like the following:

$ magicIOprogram
read, 512, /path/to/file1
read, 256, /path/to/file2
write, 15, /path/to/file3
write, 10562, /path/to/file4

If there are ways to get this information using the aforementioned tools that I am unaware of, it would also be a adequate solution.

Answer 1

You might want to try systemtap.

Here is a slightly modified example showing opens, reads and writes every 100ms:

#! /usr/bin/env stap

global fileread, filewrite

probe syscall.open.return {
    if ($return != -1) {
        printf("open, %s, %d/%d\n", user_string($filename), pid(), $return)
    }
}

probe syscall.read.return {
    p = pid()
    fd = $fd
    bytes = $return
    if (bytes > 0)
        fileread[p, fd] += bytes
}

probe syscall.write.return {
    p = pid()
    fd = $fd
    bytes = $return
    if (bytes > 0)
        filewrite[p, fd] += bytes
}

probe timer.ms(100) {
    foreach (v = [p,fd] in fileread) {
        printf("read, %d, %d/%d\n", v, p, fd)
    }                                                                                                                                                                                          
    delete fileread                                                                                                                                                                            
    foreach (v = [p,fd] in filewrite) {                                                                                                                                                        
        printf("write, %d, %d/%d\n", v, p, fd)                                                                                                                                                 
    }                                                                                                                                                                                          
    delete filewrite                                                                                                                                                                           
}

The format of output lines is:

• open, (pathname), (pid)/(fd)

• read, (bytes), (pid)/(fd)

• write, (bytes), (pid)/(fd)

Answer 2

Linux's auditd can get the information for points 1 and 2.

Assuming you are running RHEL/CentOS 6 and have an nfs share mounted as /mnt/nfs/foo:

$ tree /mnt/nfs/foo
/mnt/nfs/foo
|-- a
|   `-- foo
|-- b
    `-- bar

You will need to define the following rules in /etc/audit/audit.rules:

# Delete existing rules
-D
# Set buffer size
-b 320
# Log read and write operations
-w /mnt/nfs/foo -p r -k read -k nfs
-w /mnt/nfs/foo -p w -k write -k nfs

And then reload the auditd service with /etc/init.d/auditd reload.

Once that is done you can use ausearch and aureport to read the event logs generated by auditd:

$ cat /mnt/nfs/foo/a/foo
$ echo 'test' > /mnt/nfs/foo/b/bar
$ ausearch -k nfs | aureport -f
File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 07/15/2015 11:39:04 /mnt/nfs/foo/a/foo 2 yes /bin/cat 500 59
2. 07/15/2015 11:39:05 /mnt/nfs/foo/b/bar 2 yes /bin/bash 500 60

One caveat that it will only be able to collect the information from the client side. If your client is un-trusted you can not use this method to safely determine what they are accessing over nfs.