I have a bash script where I do some filtering with the sed command and then pipe the result to gzip to zip the output again. The filtered output has the format <id>,<title>,<bool (0 or 1)>. I would like to print the id but with a trailing carriage return instead of a newline so that they all are printed over each other, since the id is increasing. My approach was something like:

<chained sed commands> \
| tee >(sed -re "s/^([0-9]+).*/\1\r/") | gzip -c > outputfile.gz

This does however add a newline on every line and does not produce the desired outcome of only printing on one line.


You need to output the progess indicator to stderr so that it does not end up in the gzipped data. I would use perl for this:

<chained sed commands> | perl -pe 'print STDERR "$1\r" if /^(\d+)/' | gzip -c > outputfile.gz

perl -p means create an automatic loop which prints each input line, possibly modified. The -e defines the expression to execute inside the loop.

The expression matches a number at the beginning of the line, and if found prints that to stderr with a trailing carriage return. The input is passed on to the output to be gzipped.

A variation that might work better visually is to replace the print statement value with "\r$1 "\. This leaves the cursor at the end, the extra spaces overwrite any digits from earlier lines in case the current id is less than the previous id.

  • Even with the | gzip ... after the perl command? That is very strange. Try this: for i in $(seq 1 10); do echo $i bla; sleep 0.2; done | perl -pe 'print STDERR "\r$1 " if /^(\d+)/' | gzip > /tmp/bla – wurtel Sep 7 '18 at 11:17
  • But then you have nothing in your gzipped file. – wurtel Sep 7 '18 at 11:20
  • Using perl -p removes the need for the extra tee process. It's similar to UUOC (google that) – wurtel Sep 8 '18 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.