How to make `xargs` ignore child's exit and keep processing further

Question

I sometimes run long xargs jobs overnight and it is really annoying to discover in the morning that xargs died somewhere in the middle, for example because of a segmentation fault in one single special case, as happened this night.

If even one xargs child is killed, it does not process any more input:

Console 1:

[09:35:48] % seq 40 | xargs -i --max-procs=4 bash -c 'sleep 10; date +"%H:%M:%S {}";'
xargs: bash: terminated by signal 15
09:35:58 3
09:35:58 4
09:35:58 2
<Exit with code 125>

Console 2:

[09:35:54] kill 5601

Can I somehow prevent xargs from stopping to process any more input once a child process dies and instead continue processing?

Answer 1

No, you can't. From the xargs sources at savannah.gnu.org:

if (WEXITSTATUS (status) == CHILD_EXIT_PLEASE_STOP_IMMEDIATELY)
  error (XARGS_EXIT_CLIENT_EXIT_255, 0,
         _("%s: exited with status 255; aborting"), bc_state.cmd_argv[0]);
if (WIFSTOPPED (status))
  error (XARGS_EXIT_CLIENT_FATAL_SIG, 0,
         _("%s: stopped by signal %d"), bc_state.cmd_argv[0], WSTOPSIG (status));
if (WIFSIGNALED (status))
  error (XARGS_EXIT_CLIENT_FATAL_SIG, 0,
         _("%s: terminated by signal %d"), bc_state.cmd_argv[0], WTERMSIG (status));
if (WEXITSTATUS (status) != 0)
  child_error = XARGS_EXIT_CLIENT_EXIT_NONZERO;

There's no flag around that check, or around the function that calls it. It does seem to be related to max procs, which I suppose makes sense: if you set max procs high enough, it won't bother checking until it's hit the limit, which you might get to be never.

A better solution for what you're trying to do might be to use GNU Make:

TARGETS=$(patsubst %,target-%,$(shell seq 1 40))

all: $(TARGETS)

target-%:
    sleep 10; date +"%H:%M:%S $*"

Then:

$ make -k -j4 

will have the same effect, and give you much better control.

Answer 2

It would seem one of the most obvious colloquialisms is only alluded to by other proposals.

That is, you can use the following:

bash -c '$PROG_WHICH_MAY_FAIL ; (true)'

in order to "force success".

Note, this is along the lines of the proposal by lornix (just not in so many words).

Anyway, since this is effectively ignoring the actual process exit status, I'd make sure you consider somehow saving the sub-process status for post-mortem analysis. E.g.:

bash -c '$PROG_WHICH_MAY_FAIL || touch failed; (true)'

The true here is somewhat redundant and so this may be better written as:

bash -c '$PROG_WHICH_MAY_FAIL || touch failed'

Since we probably would like to know when the 'failed' file couldn't be touched. In other words, we're no longer ignoring the failure, we're taking note and continuing.

And, after considering the recursive nature of this issue, perhaps we see exactly why xargs does not make ignoring failure easy. Because it is never a good idea - you should be enhancing the error handling within the process you're developing instead. I believe this notion, however, is more inherent in the "Unix philosophy" itself.

Finally, I suppose this is also what James Youngman alludes to by recommending trap, which presumably could be used in a similar manner. That is, don't ignore the problem... trap it and handle it or you make wake up one day and find that none of the sub-programs succeeded at all ;-)

Answer 3

Use trap:

$ seq 40 | xargs -i --max-procs=4 bash -c \
 'trap "echo erk; exit 1" INT TERM;  sleep 10; date +"%H:%M:%S {}";' fnord
16:07:39 2
16:07:39 4
erk
16:07:39 1
^C
erk
erk
erk
erk

Alternatively switch from shell to another language in which you can also set signal handlers.

Note also that after bash -c foo.. you should specify the value that $0 should take (here, fnord) so that the first word produced by seq doesn't get eaten.

Answer 4

Put another command in there to 'eat' the signal from the dying program.

I tried your example, initally as shown to prove the problem... 'killall sleep' kills the sleep process, interrupts bash, and xargs quits.

As a test, I stuck a 'run another command' type command in between xargs and bash... in this case '/usr/bin/time'. this time (no pun), killall sleep kills the sleep process, but the xargs continues on.

You would pipe time's output to /dev/null, and this would do exactly what you're looking for without a major rewrite of your existing process.

I imagine if I ponder for a moment I could come up with another program to do the same without the stderr chatter from '/usr/bin/time'. Or even write one myself, it's just a 'fork' (or exec() derivative).

Remember to use '/usr/bin/time', as I'm not sure the built-in 'time' from bash will do the same 'eating' of the signal.

Answer 5

Neither time nor env worked for me (they pass along the return value of their child program) so I wrote bliss:

#!/bin/sh
"$@"
exit 0

then chmod u+x ~/bliss

and something like find_or_similar | xargs ~/bliss fatally_dying_program.sh