How track progress of a command in a pipe if only the size of its input is known in advance?

Question

I would like to track progress of a slow operation using pv. The size of the input of this operation is known in advance, but the size of its output is not. This forced me to put pv to the left of the operation in the pipe.

The problem is that the long-running command immediately consumes its whole input because of buffering. This is somewhat similar to the Turn off buffering in pipe question, but in my case it is the consuming operation that is slow, not the producing one and none of the answers to the other question seem to work in this case.

Here is a simple example demonstrating the problem:

seq 20 | pv -l -s 20 | while read line; do sleep 1; done
  20 0:00:00 [13.8k/s] [=====================================>] 100%

Instead of getting updated every second, the progress bar immediately jumps to 100% and stays there for the entire 20 seconds it takes to process the input. pv could only measure the progress if the lines were processed one by one, but the entire input of the last command seems to be read into a buffer.

A somewhat longer example that also demonstrates the unknown number of output lines:

#! /bin/bash
limit=10
seq 20 | \
  pv -l -s 20 | \
  while read num
do
  sleep 1
  if [ $num -gt $limit ]
  then
    echo $num
  fi
done

Any suggestions for a workaround? Thanks!

Answer 1

In your setup the data has passed pv while it is still processed on the right side. You could try to move pv to the rightmost side like this:

seq 20 | while read line; do sleep 1; echo ${line}; done | pv -l -s 20 > /dev/null

Update: Regarding your update, maybe the easiest solution is to use a named pipe and a subshell to monitor the progress:

#! /bin/bash
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
(rm /tmp/progress.pipe; mkfifo /tmp/progress.pipe; tail -f /tmp/progress.pipe | pv -l -s 20 > /dev/null)&
limit=10
seq 20 | \
  while read num
do
  sleep 1
  if [ $num -gt $limit ]
  then
    echo $num
  fi
  echo $num > /tmp/progress.pipe
done