GNU Parallel can generate all combinations of input, copy shell functions and run them on 3 remote servers without mixing the output, retrying the command if it fails:


env_parallel --session
process_output() {
  perl -ne '$/=undef; $|=1; @a=split//;
        print ".",shift @a;
env_parallel --retries 4 -Sserver{1..3},: \
  'echo X={1} Y={2} Z={3} | process_output' \
  ::: "a'  \"b"{1..3} ::: "c\"  'd"{3..4} ::: "e\$  f'"{7..8}


.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.

xargs -P$(nproc) can run one job per processor.


  • the function process_output,
  • the servers to run on (server1, server2, server3),
  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?

  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers? – RomanPerekhrest Nov 19 '17 at 8:08
  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ? – Fedor Dikarev Nov 19 '17 at 12:21
  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers. – David Favor Nov 19 '17 at 13:05
  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job? – Hannu Nov 19 '17 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.