I'm trying to extract the download progress of a file that's being retrieved by curl.

I tried this, but this doesn't work:

curl --progress-bar -o test.tar.bz2 2>/dev/stdout | sed -r 's/[# ]//g;s/^/#/g'

The sed expression seems to be fine though:

$ echo '########                      10.2%' | sed -r 's/[# ]//g;s/^/#/g'

Can anyone please point out what I'm doing wrong?

  • What is your expeted output? – cuonglm Jun 30 '14 at 16:24
  • I had thought that curl might not output progress bars to file, but it appears it does. @terdon, don't confuse stderr with "screen". Just because isatty(2) is often true, it doesn't mean they're the same thing. – ams Jun 30 '14 at 17:02

The main issue is that sed works on lines so it doesn't do anything until the first \n is reached and that doesn't happen until your command is finished. You can get around this by exchanging \rs with \ns:

$ curl --progress-bar -o test.tar.bz2 2>&1 | 
   tr $'\r' $'\n' | sed -r 's/[# ]+/#/g;'

This, however, brings you up against buffering, sed will now act on groups of lines. The final solution I hacked together was to redirect the error to a file and then deal with that file:

$ curl --progress-bar -o test.tar.bz2 2>er
$ while :; do 
    echo -ne "$(tr $'\r' $'\n' < er | tail -n 1 | sed -r 's/^[# ]+/#/;')\r"; 

The command above will parse the error file (er) and print the result with an \r making it update constantly. You will need to break out of it manually.

Suggestion from an anonymous user: You can also put stdbuf -oL in front of tr and sed thus modifying the buffering behaviour of those commands.

  • Will sed's unbuffered mode work here? (since it seems you're trying to emulate just that) – user2064000 Jun 30 '14 at 17:20
  • @user2064000 I don't think so. The first issue will still be there and the buffering is more an issue of the pipe passing the data to sed than sed itself. I can't get it to work in any case. – terdon Jun 30 '14 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.