I am currently using OpenCV in Python on the Raspberry Pi 4, and due to a problem with libjpeg I am getting constantly barraged with unnecessary console messages such as this:

Corrupt JPEG data: 10 extraneous bytes before marker 0xd6

My goal is to filter out these messages while still printing any other error messages (and of course normal stdout) in real time. The program is interactive, so I must still be able to provide input to the program while it is running.

I have attempted to disable printing to stderr and stdout (just in case) from within Python using this answer, but it seems that the error is being thrown by a separate process (which I assume is libjpeg) so redirecting sys.stderr from within Python is ineffective. I have also attempted creating a bash one liner to filter out error messages starting with "Corrupt" and ended up with this:

python test.py 2> >(sed -u '/^Corrupt/ d')

This partially works in that it filters out those messages, but it also doesn't display any other error message until the script has ended. So if an error is thrown, it and all other error messages are all displayed at the same time once the script has finished, but only then. And should the process be ended prematurely via CTRL-C, the errors are not shown at all.

Any ideas on how to filter the error stream without buffering all of the other error messages until the end?

| improve this question | | | | |
  • Have you tried using grep in inverse mode? Something like this: python test.py 2> >(grep -v Corrupt) – Panki Dec 24 '19 at 10:19
  • Perhaps the issue is coming from python. It doesn't print directly to the terminal anymore (default line buffering), but to something sort of like a pipe (default full buffering). Try python -u to force the output to be unbuffered. – seshoumara Dec 24 '19 at 11:38
  • @seshoumara Adding the -u flag works, thank you for the suggestion! Could you please post this comment as an answer so that I may accept it? – Idrees Dec 24 '19 at 19:16
  • Also since this is my first question in years, I was wondering if it is better for me to include the code from my program in the question or not. I am trying to balance clarity with details here. I figured since the error message wouldn't be easily reproduced without my specific setup, it wasn't worth it. – Idrees Dec 24 '19 at 19:33

When printing directly to the terminal, Python will do line buffering by default, so that you can see what happens. But when sending the output to a pipe / file, Python will buffer it entirely. This is a common behavior for many Linux commands. To get around it, please use python -u to explicitly unbuffer the output.

| improve this answer | | | | |


python test.py 2>&1 | grep -v "Corrupt JPEG data"
| improve this answer | | | | |
  • Unfortunately this still resulted in the messages buffering, though @seshoumara determined that it was actually a result of Python being weird which would explain why both grep and sed fail in this case – Idrees Dec 24 '19 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.