4

What I'm really trying to do is run X number of jobs, with X amount in parallel for testing an API race condition.

I've come up with this

echo {1..10} | xargs -n1 | parallel -m 'echo "{}"';

which prints

7 8 9
10
4 5 6
1 2 3

but what I really want to see is (note order doesn't actually matter).

1
2
3
4
5
6
7
8
9
10

and those would be processed in parallel 4 at a time (or whatever number of cpus/cores, I have, e.g. --jobs 4). For a total of 10 separate executions.

I tried this

echo {1..10} | xargs -n1 | parallel --semaphore --block 3  -m 'echo -n "{} ";

but it only ever seems to print once. bonus points if your solution doesn't need xargs which seems like a hack around the idea that the default record separator is a newline, but I haven't been able to get a space to work like I want either.

10 is a reasonably small number, but lets say it's much larger, 1000

echo {1..1000} | xargs -n1 | parallel -j1000

prints

parallel: Warning: Only enough file handles to run 60 jobs in parallel.
parallel: Warning: Running 'parallel -j0 -N 60 --pipe parallel -j0' or
parallel: Warning: raising 'ulimit -n' or 'nofile' in /etc/security/limits.conf
parallel: Warning: or /proc/sys/fs/file-max may help.

I don't actually want 1000 processes, I want 4 processes at a time, each process should process 1 record, thus by the time I'm done it will have executed 1000 times.

  • I'm getting the expected result with echo {1..10} | xargs -n1 | parallel -m 'echo "{}"';. The output is 1 2 3 4 5\n 6 7 8 9 10 (\n - linebreak in real). Works fine (GNU parallel 20141022) – RomanPerekhrest Oct 27 '17 at 20:24
  • @Roman that newline says it only executed twice, I should actually have 10 separate executions – xenoterracide Oct 27 '17 at 20:26
  • 10 separate executions is 10 jobs, as you know – RomanPerekhrest Oct 27 '17 at 20:27
  • @RomanPerekhrest yes, but I only want to have 4 jobs running at a time, 4 processes, 10 records, 10 command executions. – xenoterracide Oct 27 '17 at 20:29
5

I want 4 processes at a time, each process should process 1 record

parallel -j4 -k --no-notice 'echo "{}"' ::: {1..10}
  • -j4 - number of jobslots. Run up to 4 jobs in parallel

  • -k - keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes

  • ::: - arguments


The output:

1
2
3
4
5
6
7
8
9
10
  • Contrary to shell variables GNU Parallel replacement strings should not be quoted. In most cases (like here) it does no harm, but try this as an example of where it goes wrong: parallel "echo '{}'" ::: '$' – Ole Tange Oct 29 '17 at 23:04
1
seq 10 | parallel -j4 echo {}

Or if you have 4 cores:

seq 10 | parallel echo

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.