3

I'm writing a script that parses the output of lsof to show a list of what processes have sockets open (similar to what netstat shows). lsof also gives me a file descriptor. What I want to do now is have my script also tell how much data (in KB/sec) is being sent/received through that socket.

So far I've looked at:

  1. nethogs: Tells me a process's network I/O, but only per process, not per socket.
  2. iotop: Tells me disk I/O per process; doesn't appear to be able to tell network I/O nor I/O per socket.
  3. /proc/pid/fd/: Doesn't look like this can tell me much at all.
  4. fatrace: Tells me what files (not sockets) a process accesses.
  5. iostat: Tells me average I/O stats per disk.
  6. tcpdump: Gives me a dump of all traffic per IP; doesn't appear to be able to tell which socket the traffic belongs to.
  7. strace -p pid -e trace=network -s 0: Tells me every time the given process calls certain socket functions, which seems useful, but in practice is only giving me a lot of recvfrom(13, 0x7feed8fb3074, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
  8. strace -p pid -e trace=read,write -s: Tells me the result of each read/write call.

strace looks promising, but I'm not sure I'm working it correctly (is read,write enough?), plus it seems like there would be a lot of overhead. (For every process with an open socket I'd have to run an instance of strace and parse the output.)

Does the Linux kernel provide a nicer means to measure how much I/O is going on per fd/socket? Possibly something could be set up with iptables or hacking on nethogs?

1
1

Consider using SystemTap. It is a DTrace clone but for Linux - it compiles kernel module that dynamically patches kernel and has full access to its data (so lsof may not be needed in that case).

However, the more information you ask, more tricky and kernel-version-specific script will become.

For example, simple stat-like utility for sockets will look like:

global stats;

probe begin {
    printf("%14s %6s %12s %5s %5s %8s\n", "NAME", "PID", "EXECNAME",
                "INO", "OPS/S", "BYTES");
}

function file_ino:long (file:long)
{
    if(file == 0) return -1;
    d_inode = @cast(file, "file", "kernel")->f_inode;
    if (d_inode == 0) return -1;
    return @cast(d_inode, "inode", "kernel")->i_ino;
}

probe socket.send, socket.receive {
    if(success == 0) next;

    /* Get inode number for a socket. Depending on 
       operation, struct file is contained in different fields. 
       Determine that field and get inode number */
    ino = -1;
    if(@defined($sock)) {
        ino = file_ino($sock->file);
    }
    else if(@defined($iocb)) {
        ino = file_ino($iocb->ki_filp);
    }

    stats[pid(), execname(), ino, name] <<< size;
}

probe timer.s(1) {
    /* Every 1 second print statistics */
    foreach([pid+, ename, ino, name] in stats) {
        printf("%14s %6d %12s %5d %5d %8d\n", name, pid, ename, ino, 
                    @count(stats[pid, ename, ino, name]), 
                    @sum(stats[pid, ename, ino, name]));
    }
    delete stats;
}

I tested it on vanilla Linux 3.12, but as you can see logic of getting inode number relies on internal kernel structure.

As you can see, most of the time, it traces himself writing to SSH session:

       NAME    PID     EXECNAME   INO OPS/S    BYTES
socket.send   2655         sshd  7480     1       96
socket.send   2655         sshd  7480     1       96
socket.send   2655         sshd  7480     1       96
...

There is much more complex script in examples: https://sourceware.org/systemtap/examples/network/socktop

Warning

SystemTap is in-development kernel-level software, so there is the possibility of kernel panics or freezes. However, it is pretty rare, but be careful with it.

References

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.