1

GNU Parallel can generate all combinations of input, copy shell functions and run them on 3 remote servers without mixing the output, retrying the command if it fails:

#!/bin/bash

env_parallel --session
process_output() {
  perl -ne '$/=undef; $|=1; @a=split//;
      while(@a){
        print ".",shift @a;
        select($a,$a,$a,0.100);
      }';
}
env_parallel --retries 4 -Sserver{1..3},: \
  'echo X={1} Y={2} Z={3} | process_output' \
  ::: "a'  \"b"{1..3} ::: "c\"  'd"{3..4} ::: "e\$  f'"{7..8}

Output:

.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.1. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.2. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.3. .Z.=.e.$. . .f.'.8.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.7.
.X.=.a.'. . .".b.3. .Y.=.c.". . .'.d.4. .Z.=.e.$. . .f.'.8.

xargs -P$(nproc) can run one job per processor.

Given

  • the function process_output,
  • the servers to run on (server1, server2, server3),
  • the argument groups (a' "b1, a' "b2, a' "b3;c" 'd1, c" 'd2, c" 'd3;e$ f'7, e$ f'8)

how can you emulate GNU Parallel using xargs to get the same done using process_output and the servers?

4
  • you mean to just produce the same output with xargs -P<N> OR it should also fulfill all conditions like distributing process_output calls on 3 remote servers? Commented Nov 19, 2017 at 8:08
  • Based on my experience: there is no easy way to emulate "parallel" with "xargs". It is allways easier to find package of GNU Parallel on your OS and Platfrom than create something new and update and support this solution after. By the way, what are your restrictions to use xargs instead of parallel ? Commented Nov 19, 2017 at 12:21
  • I think what you're really asking is how to run parallel to execute all work on one machine/container. To do this, you simply drop the reference to your remote servers. Commented Nov 19, 2017 at 13:05
  • @ole are you debugging gnu parallell? ;-), or just asking whether xargs actually can do the same job?
    – Hannu
    Commented Nov 19, 2017 at 16:14

0

You must log in to answer this question.

Browse other questions tagged .