I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying.

$ sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 | 
  sudo tar x -C /home/pi/test

...this works perfectly. Example output:

191MiB 0:00:03 [58.1MiB/s] [<=>       ]

...in the terminal. But I want to do something more with this output and I have simple python script /home/pi/screentest.py:

import sys
print "start"
for line in sys.stdin:
    print "line: " + line
print "end"

Test run:

$ (   sudo tar c --files-from /tmp/camfilenames | 
      pv --width 40 | 
      sudo tar x -C /home/pi/test   ) | 
   python /home/pi/screentest.py


191MiB 0:00:03 [55.1MiB/s] [<=>       ]

I understand that pv returns everything in one buffer, so how to force it to return it line by line and flush the buffer every time to output something like:

line: 0MiB 0:00:00 ...
line: 60MiB 0:00:01 ...
line: 120MiB 0:00:02 ...
line: 191MiB 0:00:03 ...
  • 1
    If you like a fast copy, I recommend to use star -copy. – schily Nov 11 '19 at 15:58

pv outputs the progress to stderr, so pipe to the python script didn't get anything from pv. It printed end after pv's output stopped probably because by then the last tar had exited, and so the script's stdin had been closed.

You might need something like:

( sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 -f | 
  sudo tar x -C /home/pi/test ) 2>&1 >/dev/null | python /home/pi/screentest.py

Where we discard the stdout of the tar pipeline, and instead send stderr to the pythons script. Note that we need to force progress output from pv with -f, since pv won't progress if stderr is not a terminal.

But you might want to look in to pv's --format option. You can use something like:

sudo tar c --files-from /tmp/camfilenames | 
  pv --width 40 --format $'%b %t\n' | 
  sudo tar x -C /home/pi/test

to get similar output.


Thank you @muru, I tried you'r solution, but in the meantime I found that main issue is with forcing PV to output new lines instead of just updating constantly one line.

So I wrote this:

$ tail -f /home/pi/log | pv -f |& stdbuf -oL tr '\r' '\n'

And It worked, so now question was - how to call my python script to do something with every line - and I found that I can use do-while loop for that:

$ tail -f /home/pi/log | pv -f |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do echo $LINE; done;)

So having this I just use my python script (little LCD handler) to show PV status on LCD:

tail -f /home/pi/log | pv -f --width 20 -i 2 |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do python /home/pi/screen.py $LINE; done;)

And I re-introduced my TAR script to copy many small files into it:

$ (sudo tar c --files-from /tmp/camfilenames | pv -f --width 20 -i 1 | sudo tar x -C /home/pi/test) |& stdbuf -oL tr '\r' '\n' | (while read -r LINE; do python /home/pi/screen.py $LINE; done;)

So at the end when I add more parameters to PV, and I made few string replaces before sending PV output to LCD, and I add few break lines I have this:

cd /home/pi/cam \ && ( \ sudo tar c --files-from /tmp/camfilenames \ | pv -f -s $( \ du -sb /home/pi/cam \ | awk '{print $1}' \ ) --width 64 -i 5 \ | sudo tar x -C /mnt/usb \ ) \ |& stdbuf -oL tr '\r' '\n' \ | ( \ while read -r LINE; \ do LINE="${LINE// /_}"; \ LINE="${LINE//_\[/ [}"; \ LINE="${LINE//\]_/] }"; \ lcd $LINE; \ done; \ ) \ && cd /home/pi

This is how PV output looks on LCD: This is how PV output looks on LCD

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.