Pipe Viewer - Progress monitor performance consequence

Question

I am writing a batch script to sort through gigs and gigs of data. All of the data is text but the script will take a long long time to execute. I would like to give some visual indication that the script is running. I found the program called pv which allows you to create progress bars and other nice CLI progress indicators.

My question is what type of impact does something like this have on performance and are there any other alternatives without me having to reinvent the wheel.

I have googled it to death and haven't found anything which is surprising as I would assume you would only show progress on long tasks where performance is important. But I suppose I am using clock cycles and you could also use it to measure disk I/O or network data transfers where using a few cycles for a progress indication wouldn't matter.

Any ideas?

P.S. I have also considered using the echo -ne trick to create my own but in order to make it feasible I would have to use the % operator on every loop and only take action ever 100th or so loop but that is A LOT of wasted calculations...

Answer 1

There will typically (lacking zero-copy trickery) be measurable overhead due to the extra IPC: copying the data from one process to another, rather than the "workhorse" process reading files directly. A pipe may also result in loss of performance (or functionality) for other reasons: with piped input a process cannot seek() on its input, and cannot mmap() it.

Generally though, the main performance bottlenecks are probably disk I/O, and CPU compute time (which presumably is intensive in your case). These may be very much larger than the IPC overhead, but there are many variables here (CPU type, disk type and fileystem type, available physical RAM, OS and version, libc and version — at least).

You can get a rough idea of performance with some quick tests, taking care to flush the disk cache before each one (I'm using linux, I use this method) in between each test.

# time ( pv -pt somethinglarge.iso | sha256sum )
[...]
real    0m8.066s
user    0m5.146s 
sys     0m1.075s

# time ( sha256sum somethinglarge.iso )
[...]
real    0m7.913s
user    0m5.064s
sys     0m0.309s

Note the similar real and user and times, and the marked increase in system time for the piped case due to the extra copying.

On some OSes, specifically Linux, you may be able to read per-process I/O stats from /proc (see 3.3) (you'll need CONFIG_TASKSTATS enabled in the kernel for this). This isn't as easy or as slick as pv, but it's low-overhead. pidstat uses this, it can be used to show real-time throughput (rate) on a PID, but it's less useful as a completion indicator.

A similar linux option (this one doesn't need CONFIG_TASKSTATS), given a process and file descriptor, you can track the file descriptor's offset in /proc/PID/fdinfo/FD (the pos: field). Here's a toy script that shows this:

FILE=/tmp/some-large-input
SZ=$(stat -c "%s" "$FILE")

# start slow process in background
( some-slow-command $FILE ) &
PID=$!
FD=/proc/$PID/fdinfo/3       # some experimentation required
# or iterate over /proc/$PID/fd/* with readlink 

# start %-ometer in background, exits when FD disappears
(
while nawk '/^pos:/{printf("%i\n",$2*100/'$SZ')}' $FD 2>/dev/null ; do
    sleep 5  # adjust
done | dialog --gauge "$PID: processing $FILE ($SZ bytes)" 10 60
) &

wait $PID
if [ $? -eq 0 ]; then
    echo 100 | dialog --gauge "$PID: completed $FILE ($SZ bytes)" 10 60
else 
    echo ...
fi

(Caveat: not accurate for small files, libc stdio buffering will skew the results.)

Other options that occur to me right now:

• use lsof to monitor a processes fd offsets not exactly lightweight, but multi-platform, and you can start it on any long-running process after the fact, which you cannot do with pv (it's not pretty either, since lsof refuses to give both the size and offset in one go)

• something hackish with LD_PRELOAD and some stubs which track data read/write, this too is multi-platform, but I think you'd have to write your own (I don't know any that does exactly this, but here's a related answer of mine)

• Update: someone has gone to the trouble of writing a general purpose transfer monitor tool cv for use with coreutils commands on Linux. It uses similar logic to the /proc fdinfo approach (as demonstrated in the shell hack above). It also has a background mode where it scans /proc and reports on transfers in progress as it finds them. See related question Is it possible to see cp speed and percent copied?

Answer 2

The pv command does not have much overhead on CPU in cases where I use it (mostly using zfs send and receive actions). If you just want to know that it is still processing you can just use pv to count data that is piped through it. If you want an estimate time of completion you might want to add a pre-command which counts the sum of data to be processed with pv -s <SIZE>. Such pre-calculations might cause some overhead.