Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I sometimes run long xargs jobs overnight and it is really annoying to discover in the morning that xargs died somewhere in the middle, for example because of a segmentation fault in one single special case, as happened this night.

If even one xargs child is killed, it does not process any more input:

Console 1:

[09:35:48] % seq 40 | xargs -i --max-procs=4 bash -c 'sleep 10; date +"%H:%M:%S {}";'
xargs: bash: terminated by signal 15
09:35:58 3
09:35:58 4
09:35:58 2
<Exit with code 125>

Console 2:

[09:35:54] kill 5601

Can I somehow prevent xargs from stopping to process any more input once a child process dies and instead continue processing?

share|improve this question
add comment

4 Answers

up vote 12 down vote accepted

No, you can't. From the xargs sources at savannah.gnu.org:

if (WEXITSTATUS (status) == CHILD_EXIT_PLEASE_STOP_IMMEDIATELY)
  error (XARGS_EXIT_CLIENT_EXIT_255, 0,
         _("%s: exited with status 255; aborting"), bc_state.cmd_argv[0]);
if (WIFSTOPPED (status))
  error (XARGS_EXIT_CLIENT_FATAL_SIG, 0,
         _("%s: stopped by signal %d"), bc_state.cmd_argv[0], WSTOPSIG (status));
if (WIFSIGNALED (status))
  error (XARGS_EXIT_CLIENT_FATAL_SIG, 0,
         _("%s: terminated by signal %d"), bc_state.cmd_argv[0], WTERMSIG (status));
if (WEXITSTATUS (status) != 0)
  child_error = XARGS_EXIT_CLIENT_EXIT_NONZERO;

There's no flag around that check, or around the function that calls it. It does seem to be related to max procs, which I suppose makes sense: if you set max procs high enough, it won't bother checking until it's hit the limit, which you might get to be never.

A better solution for what you're trying to do might be to use GNU Make:

TARGETS=$(patsubst %,target-%,$(shell seq 1 40))

all: $(TARGETS)

target-%:
    sleep 10; date +"%H:%M:%S $*"

Then:

$ make -k -j4 

will have the same effect, and give you much better control.

share|improve this answer
3  
+1 for checking the source code. –  bahamat Jul 5 '12 at 9:03
add comment

Put another command in there to 'eat' the signal from the dying program.

I tried your example, initally as shown to prove the problem... 'killall sleep' kills the sleep process, interrupts bash, and xargs quits.

As a test, I stuck a 'run another command' type command in between xargs and bash... in this case '/usr/bin/time'. this time (no pun), killall sleep kills the sleep process, but the xargs continues on.

You would pipe time's output to /dev/null, and this would do exactly what you're looking for without a major rewrite of your existing process.

I imagine if I ponder for a moment I could come up with another program to do the same without the stderr chatter from '/usr/bin/time'. Or even write one myself, it's just a 'fork' (or exec() derivative).

Remember to use '/usr/bin/time', as I'm not sure the built-in 'time' from bash will do the same 'eating' of the signal.

share|improve this answer
1  
A good alternative to time for this purpose would be env, since all it does is add zero or more optional variables to the environment of the program it runs. It emits no output of its own, and the return code of the invoked program will be passed back up to whatever called env. –  James Sneeringer Jul 6 '12 at 17:20
    
{Chuckle} I thought of that a while after I write this up. Time was the first thing that came to mind as a "run something" command. Works nicely though. Congrats and thank you. –  lornix Jul 6 '12 at 17:26
add comment

Use trap:

$ seq 40 | xargs -i --max-procs=4 bash -c \
 'trap "echo erk; exit 1" INT TERM;  sleep 10; date +"%H:%M:%S {}";' fnord
16:07:39 2
16:07:39 4
erk
16:07:39 1
^C
erk
erk
erk
erk

Alternatively switch from shell to another language in which you can also set signal handlers.

Note also that after bash -c foo.. you should specify the value that $0 should take (here, fnord) so that the first word produced by seq doesn't get eaten.

share|improve this answer
add comment

Neither time nor env worked for me (they pass along the return value of their child program) so I wrote bliss:

#!/bin/sh
"$@"
exit 0

then chmod u+x ~/bliss

and something like find_or_similar | xargs ~/bliss fatally_dying_program.sh

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.