4

I am using a tool (openocd) which is printing a lot of garbage, then a basic progress bar slowly printing simple dots, then again some garbage.

I would like to filter this output with grep so only the line with progress bar is shown, and in real time (i.e. each dot outputted by openocd is immediately printed in the terminal):

openocd <args> |& grep '^\.'

The problem is that grep is line buffered (at best) so the progress bar will not be shown until it is finished.

How can I do with grep, or is there any standard alternative to achieve this? If there is a way through openocd configuration, this would be useful though I would prefer a more general solution.

  • Are you sure garbage and progress bar are both printed to the same file descriptor? If the garbage is printed to stderr, for example, you can redirect it out of the way. – Alessandro Dotti Contra Dec 15 '16 at 15:26
  • @adc Unfortunately, openocd is sending all to stderr – calandoa Dec 15 '16 at 16:48
3

This is a kind of hacky/unusual answer, the fact being that this is most likely possible in a not very clean way.


grep itself seems to only print output when it encounters a newline character, your progress bar likely does not introduce a newline character when it updates, hence your issue.

strace is a tool used to view the system calls that a command is calling, this includes things like reading and writing things to memory/storage, as well as things like opening/closing file descriptors.


With strace you can view what a process is accessing, in the case of your pipe stout is being passed to grep, so with strace you can view the text that's being fed to grep. strace will regularly be sent the output coming from the piped command, and you can sniff that output and display it. I was testing with rsync --progress, which seems to run into a similar scenario. I used grep on the ##% because that's what rsync uses to show progress.

rsync --progress file1 file2 | strace -e trace=read grep "[0-9]*%"

If you run this command you'll find that strace doesn't have nice output, but that when I used it strace caught a couple of reads from the rsync that grep would not normally write, showing reads for 0%, 21%, 45%, 68%, 91% and 100%, seemed to update about every second (probably based on how often rsync updates the progress).

So with that you can grep the strace output, which is not very nice, by calling the same grep again.

rsync --progress file1 file2 | strace -e trace=read grep "[0-9]*%" 2>&1 > /dev/null | grep -o "[0-9]*%"

The 2>&1 is important because strace prints on stderr. The > /dev/null redirects stdout to /dev/null to prevent the output of the first grep being reported. The end result of this was the following output:

0%
21%
45%
68%
91%
100%

You'll have to swap out the grep, but it seems like it'll do the trick. It's not pretty, but it functions, and works around the restrictions of grep. Seems like a grep -f that works like tail -f would be handy (I know grep -f is already in use).

The first grep is mostly just to filter down the text that strace will be reading, since only the matching lines will be listed in straces read calls, but you also need something for the text to be moving through so strace can watch it.

2

I am expending on Centimane answer as the final answer is quite complicated.

It was already quite hacky... it becomes now really crappy:

make flash \
    |& strace -e trace=read grep -e "^\." -e rror \
    |& stdbuf -o0 sed -ne 's/^.*"\.".*/\./p;/rror/p' \
    | stdbuf -o0 tr -d '\n' \
    ; echo

So the make flash above is calling openocd;

strace is using Centimane's hack explained above;

sed replace read(0, ".", xxx) lines by single .;

tr is keeping all the . on one line, and final echo is putting the only EOL.

In addition to that, sed and tr are called with stdbuf -o0 which sets the stdout buffer size to 0 (because EOL removed), and grep/sed are also matching (e)rror to print some garbage in case of error.

I tried to factorize all this bloat to the bare minimum, but could not do better.

Note also that I am using zsh/Ubuntu 14.04, and I am not sure this can work on other Unix.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.