I am testing my web server response. Well I have a computer with 15 CPUs and 30 GB memory. How can I execute the following bash command on this computer 25000 times per second?

 curl -Z url="mywebserver.com" --data "{JSON DATA}"

I did the following which is worse and executes the shell 200 times per second.

 seq 25000 | parallel -j0 myshell
  • the -Z option is for parallel transfers, not parallel requests. Mar 27 '20 at 13:20
  • 1
    Does this answer your question? Running thousands of curl background processes in parallel in bash script Mar 27 '20 at 13:22
  • @TimKennedy Actually mine is much simpler and would like very fundamental script for doing that with GNU parallel.
    – user401012
    Mar 27 '20 at 13:43
  • What is the important requirement here? Do you need exactly 25000 times per second, no less and no more? Do you need at least 25000 per second but are fine with more? Or maybe the opposite? You can't have more than 25000? Depending on your needs, the solution can be different. Please edit and clarify.
    – terdon
    Mar 27 '20 at 14:11
  • @terdon Thanks for the comment. Indeed I need at least 25000per second if this arise would be better and doing this for period of max 20 seconds.
    – user401012
    Mar 27 '20 at 14:17

The CPU overhead of running a single GNU Parallel job is in the order of 1-10 ms. This is partly due to being written in Perl, but mostly due to a lot of safety tests happening behind the scenes.

So if you want to use GNU Parallel to run 25000 jobs per second on average, you can do:

seq 1000000 | parallel -n100 --pipe --round-robin -I ,, parallel myjob {}

You will need around 250 cores to do that.

(When you call parallel from parallel you get into the situation: Which parallel should replace the replacement strings? By using -I dummy or -I ,, in the outer parallel you can use {} in the inner parallel as you are used to.)

TL;DR: GNU Parallel is not made for very short jobs where run time is important.

xargs has very few safety tests, and that may be suitable here:

myjob() {
  curl -Z url="mywebserver.com" --data "{JSON DATA}"
export -f myjob
seq 25000 | parallel --pipe --block 100 --roundrobin xargs -n1 -P0 bash -c myjob >/dev/null

On my 48 core server this runs in 2.6s. So still not at the required 1s.

  • Could you explain what the specific options you're using are for? For example, why the -I ,,? I thought that changed the replacement string, so if you use -I ,, wouldn't you also need to say parallel myjob ,,?
    – terdon
    Mar 29 '20 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.