Making a progressbar with "dialog" from rsync output

Question

I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync. Currently I have only tested it directly at the prompt, but I'm planning to do this in a (bash) shell script.

I have looked around the internet and found bits and pieces, but I'm still missing something to make it work

(Disclaimer: This might be a totally wrong approach, and is a redirect/piping monstrosity)

What I have currently put together:

rsync -avz --progress -e "ssh" user@server:/home/user/data/ /home/user/data | awk -f /home/user/rsync.awk | sed 's/\([0-9]*\).*/\1/' | dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70

First I have the actual rsync command with the --progress option Output from rsync is piped into awk and uses the followng awk filter:

{
   if (index($0, "to-check=") > 0)
   {
        split($0, pieces, "to-check=")
        split(pieces[2], term, ")");
        split(term[1], division, "/");
        print (1-(division[1]/division[2]))*100
}
#   else
#   {
#       print "#"$0;
#   }
   fflush();
}

This filters out rsync output and provides the percentage in the following format:

53.7037
55.5556
57.4074
59.2593
61.1111
62.963

So to get rid of the decimal numbers, I feed the output to sed:

sed 's/\([0-9]*\).*/\1/'

Which gives the following output:

64
66
68
70
72
74
75
77

Those numbers are piped into dialog like this:

dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70 

As far as I know, "dialog --gauge" etc. should accept this, but it just displays progress to be 0% until it suddenly reaches 100%

Can someone point me in the right direction here? Am I far away from a working progressbar? Is there a better way to achieve this?

Regards,

Christer

EDIT: After taking @lynxlynxlynx' answer into account, the correct command line is:

rsync -avz --progress -e "ssh" user@server:/home/user/data/ /home/user/data \ 
 | awk -f /home/user/rsync.awk \ 
 | sed --unbuffered 's/([0-9]*).*/\1/' \ 
 | dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70

Answer 1

for i in 10 20 30; do echo $i; sleep 1; done | dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70

works fine, so @Shadur is right and there is buffering at play.

Adding the sed stripper into the mix shows it is the culprit (only shows 0 and 30):

for i in 10 20 30; do echo $i; sleep 1; done | sed 's/\([0-9]*\).*/\1/' | dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70

Now that the problem is known, you have multiple options. The cleanest would be to round/cut the percentage in awk with either math or string manipulation, but since you have GNU sed, just adding -u or --unbuffered should do the trick.

However for completeness' sake, a simple test case shows awk also does buffering:

for i in 10 20 30; do echo $i; sleep 1; done | awk '{print $0}' | sed -u 's/\([0-9]*\).*/\1/' | dialog --title "My Gauge" --gauge "Hi, this is a gauge widget" 20 70 

But you already handle that with fflush, so I don't expect problems.