3

I'm trying to extract the download progress of a file that's being retrieved by curl.

I tried this, but this doesn't work:

curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>/dev/stdout | sed -r 's/[# ]//g;s/^/#/g'

The sed expression seems to be fine though:

$ echo '########                      10.2%' | sed -r 's/[# ]//g;s/^/#/g'
#10.2%

Can anyone please point out what I'm doing wrong?

2
  • What is your expeted output?
    – cuonglm
    Commented Jun 30, 2014 at 16:24
  • I had thought that curl might not output progress bars to file, but it appears it does. @terdon, don't confuse stderr with "screen". Just because isatty(2) is often true, it doesn't mean they're the same thing.
    – ams
    Commented Jun 30, 2014 at 17:02

2 Answers 2

3

The main issue is that sed works on lines so it doesn't do anything until the first \n is reached and that doesn't happen until your command is finished. You can get around this by exchanging \rs with \ns:

$ curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>&1 | 
   tr $'\r' $'\n' | sed -r 's/[# ]+/#/g;'

This, however, brings you up against buffering, sed will now act on groups of lines. The final solution I hacked together was to redirect the error to a file and then deal with that file:

$ curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>er
$ while :; do 
    echo -ne "$(tr $'\r' $'\n' < er | tail -n 1 | sed -r 's/^[# ]+/#/;')\r"; 
  done

The command above will parse the error file (er) and print the result with an \r making it update constantly. You will need to break out of it manually.

Suggestion from an anonymous user: You can also put stdbuf -oL in front of tr and sed thus modifying the buffering behaviour of those commands.

2
  • Will sed's unbuffered mode work here? (since it seems you're trying to emulate just that)
    – user48923
    Commented Jun 30, 2014 at 17:20
  • @user2064000 I don't think so. The first issue will still be there and the buffering is more an issue of the pipe passing the data to sed than sed itself. I can't get it to work in any case.
    – terdon
    Commented Jun 30, 2014 at 17:25
1

Try perl:

curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>&1 | perl -015 -n -e 'print "$1\n" if (/[#]* ([\d]+)/);'

where:

  • 2>&1 assigns stderr to where ever stdout is going which in this case is a pipe to perl
  • -015 sets the input record separator to carriage return (octal 15)
  • -e is a line of code, in this case one that skips any leading #'s and extracts a subsequent integer into $1 which is printed whenever this extraction succeeds.
  • -n wraps the lines provided with -e in "while(<>){}" which reads the input record-by-record until EOF, where we have arranged for each record to be separated by CR with -0

You must log in to answer this question.