5

For long running processes like init, I can do things like

$ cat /proc/[pid]/io

What can I do if I want to see stats for a briefly running process such as a command line utility like ls? I don't even know how to see the pid for such a briefly running process...

  • What specific stats are you looking to get? IOPS, bandwith usage, iowait%, front/back merges, etc, etc? There are a lot of tools out there for io stats so we need to know which ones you're interested in getting. Please update the question with this. – Bratchley Jul 22 '14 at 17:15
  • Ok, thanks for helping me clarify what I'm wanting to know. I'm more interested in finding out how to get stats of any type for briefly running processes. I'm mainly wanting to know this to get performance stats, troubleshoot, and understand the system. – labyrinth Jul 22 '14 at 17:23
  • I think the most appropriate answer here will depend on the exact kind of information you want. E.g., you can stall a process at specific points by running it in a debugger and then look at what it is doing I/O wise, but this is totally inappropriate for gauging performance. – goldilocks Jul 22 '14 at 17:37
2

Basically, it sounds like you want general advice on profiling an application's I/O at runtime. You've been doing this with /proc/$PID/io which will give you some sort of idea of how much bandwidth is being used between disk and memory for the application. Polling this file can give you a rough idea of what the process is doing but it's an incomplete picture since it only shows you how much data is being shoved to and from disk.

To solve your stated problem, you basically have the following options:

  • Use platform instrumentation. On Linux writing a SystemTap script is the most feature complete solution but depending on how hardcore you're wanting to go that may be more work than you're really willing to expend for the desired benefit.

  • Use application-based instrumentation. A lot of ways to do this but gprof profile is a popular option. Some applications may also provide their own instrumentation, but you'd have to check.

Probably, the best alternative is to use already existing platform instrumentation tools together to achieve the desired effect and to get the most out of it.


I'm not aware of a program that will kick off an application and do all this for you (doesn't mean there isn't by any means, just that I haven't heard of it) so your best bet is to just start gathering system-wide information and just filter for the PID you're concerned about after the fact (so that you get a full sample).

First things first, I would enable auditing of execve calls so that you can save the PID of the application you're kicking off. Once you have the PID, you can remove auditing.

Run mount debugfs -t debugfs /sys/kernel/debug to get the debugfs running, so you can run the blktrace.

On my system I ran blktrace -d /dev/sda -a read -a write -o - | blkparse -i - but you can adjust accordingly. Here is some example blktrace output:

8,0   15        3 1266874889.709440165 32679  Q   W 20511277 + 8 [rpc.mountd]

In the above output the fifth column (32679) is the PID associated with the application performing the write. The parts we care about are the Q (event type, queued) the W (RWBS field, W means it's an write since there's no S in that field as well the implication is that it was an asynchronous one.) and the 20511277 + 8 (operation starts at block number 20511277 and goes for another eight blocks). Determining read/write sizes should just be adding the blocks together and multiplying by block size.

blktrace will also tell you about more than just throughput, it will also let you see if there's anything going on with merges that you care about.

Once you have the blktrace running, you can spawn the process using strace -c which will give you a feeling for the average latency associated with each system call (including read and write operations). Depending on how reliable each invocation needs to be latency can be important, it can also tell you more about what the application is doing (to point out areas to explore tuning) without having any application instrumentation going.

Between those two you should get a pretty good sampling of what your program is doing without losing any data or possibly including the I/O of other applications. Obviously there are more ways to do this than what I've described but this is how I would've solved the problem.

One should also be able to collect I/O-related latency measures by messing with blkparse's output options, for instance. I just didn't because I haven't played with them enough.

| improve this answer | |
  • I hate to do it, but I edited my question again to be i/o specific. I agree that it really depends on what kind of statistics you are trying to get at, and your answer gives me a very good idea of where I would want to go if I were needing to profile i/o for briefly running processes. By rewording my question back, I can give you the point for sharing an in-depth answer that seems instructive for similar cases. – labyrinth Jul 22 '14 at 18:34
  • Overall, SystemTap and Valgrind/Callgrind/Cachegrind can collect pretty much any information you could possibly eve think of. That's hard mode though, and you usually have to install separate debuginfo versions of every package you want to profile. – Bratchley Jul 23 '14 at 14:04
1

You can start command in background then get its pid via $! variable.

Example:

$ ls & cat /proc/$!/io
[1] 6410
rchar: 7004
wchar: 0
syscr: 13
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0
| improve this answer | |
  • What if the program exits before the cat gets ran? The sample size is already going to be small so you would only be getting a single sample anyways which makes the profiling data more trivia or at best an incredibly vague reference point. – Bratchley Jul 22 '14 at 17:39
  • @JoelDavis: The cat command will fail, of course. – cuonglm Jul 22 '14 at 17:45
  • I like this answer as a quick hack, but I got a lot of inconsistent results. I kept getting "event not found /io" when using zsh (I suppose it thinks ! is a history cmd), and "rchar: 4508" in bash for any ls I did, large or small, recursive or not. I got it to give me seemingly good values a few times, but I'd have to mess with it more to figure out why I had these problems. – labyrinth Jul 22 '14 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.