I'm running a command like:

parallel --spreadstdin --line-buffered 'some_command 2> `mktemp --tmpdir /tmp/stderr`' | do_something

The trick is that parallel creates a lot of processes and they all get a stderr file, most of which is uninteresting because they're empty. How can I make my shell generate a stderr file only if any standard error output actually happened?

  • 1
    wouldn't it be easier to delete empty file ?
    – Archemar
    Oct 3 '14 at 11:59
  • I'd prefer to monitor logs on the fly and empty files make it less convenient.
    – d33tah
    Oct 3 '14 at 12:06
  • @don_crissti: I'm not exactly satisfied by this solution, but thanks anyway.
    – d33tah
    Oct 3 '14 at 13:39

As far as I know, this is impossible. The shell reads right to left, so if 2> error exists. an error file will be created. I don't think there's any way around that.

You might be able to do some complex tricks such as using a named pipe and testing its output but I really don't think that's worth it. Instead, I would either use inotify as explained in another answer or create tmp files and copy them to a different location if they contain data.

tmp=$(mktemp); command 2> "$tmp"; [ -s "$tmp" ] && cp "$tmp" /path/to/log/files

parallel --spreadstdin --line-buffered 'some_command 2 \

mktemp --tmpdir /tmp/stderr' | do_something && if [test -s /tmp/stderr]; then; sleep .1; else; rm /tmp/stderr

  • One-Liner to Test if File Exists and Size is Greater than Zero for TRUE; empty if FALSE.

I.T. is about doing something that works. Stop counting your bits and bytes, and create an inotifywait trigger on startup to check your logs if empty, record results to datatable, and wipe log on complete; then program a web interface that emails you when keywords appear in your error log.

    inotifywait -e create /tmp/* && \
      if [ -s /tmp/stderr ]
        ##put it in a datatable
        ##remove the log

iNotifyWait can also monitor Kernel F/S subsystem to see if a file is changed, if you are appending. Otherwise, modify the Source Code and recompile to prevent the error from being created by modifying the try() catch() or error reporting section (the part that prints error codes to the screen).

  • I'd expect it to be more complex in the working case since you'd have to be looking at a particular temporary file and as I said, I'd rather prevent empty stderr files from being created at all.
    – d33tah
    Oct 3 '14 at 12:32
  • +1 inotifywait; -1 "Stop counting your bits and bytes ... program a web interface that emails you when keywords appear in your error log"; -2 "prevent errors" - there's nothing wrong about error reporting on stderr, it usually isn't a bug in the software that is reported there.
    – peterph
    Oct 3 '14 at 13:24
  • @peterph: in my case it is usually some kind of a bug.
    – d33tah
    Oct 3 '14 at 13:40
  • @d33tah if you think you have a bug, wouldn't it be easier to fix the bug, than to build a logging facility? Use trap to process errors.
    – eyoung100
    Oct 3 '14 at 14:10
  • First you need to trace down the bug.
    – d33tah
    Oct 3 '14 at 14:50

It’s amazing how close the answer to that other question on SO came to answering the question without hitting the mark.  Define a shell function (or write a script) like this:

    # Check that "$1" is set and either complain/abort or generate a default.
    if read -r x
        { printf "%s\n" "$x"; cat; } > "$1"

Then use as:

some_command 2> >( save_err $(mktemp --tmpdir /tmp/stderr) )

Caveats: This will fail if the stderr output is a single line not ending with a newline.  To fix that, change the guts of the function to

    if read -r x
        { printf "%s\n" "$x"; cat; } > "$1"
    elif [ "$x" != "" ]
        printf "%s" "$x" > "$1"

I used printf rather than echo to guard against text beginning with a dash or containing backslashes.  There may be issues if the first line of the stderr output contains non-ASCII characters.

Not the answer you're looking for? Browse other questions tagged or ask your own question.