2

I try to run cocurrent curl but it can easily report "Could not resolve host". To run curl parallel, I use "parallel".

parallel :::: ./a.sh ./a.sh
from api server
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16    0    16    0     0  13781      0 --:--:-- --:--:-- --:--:-- 16000
from api server
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16    0    16    0     0  14925      0 --:--:-- --:--:-- --:--:-- 16000
from api server
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16    0    16    0     0  15009      0 --:--:-- --:--:-- --:--:-- 16000
from api server
from api server
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16    0    16    0     0  14324      0 --:--:-- --:--:-- --:--:-- 16000
curl: (6) Could not resolve host: curl
100    16    0    16    0     0  44198      0 --:--:-- --:--:-- --:--:-- 44198

a.sh

#!/bin/bash

curl http://127.0.0.1:81/a.php

a.php

<?php

echo "from some server\n";

How could resolve host fail for just 4 concurrent curl? I simulate this because the original problem I encounter is described in php curl localhost is slow when making concurrent requests. I really don't think it's an open file limit issue since there is just 4 concurrent curl. Can anyone explain why this occurs? By the way, the OS is ubuntu 16.04.


Well the correct way to use parallel is suggested by rudimeier. After using yes | head -n4 | parallel ./a.sh The issue no longer exists. Still, my original issue is there.

1

You should try using --dryrun when you are confused by what GNU Parallel runs:

$ parallel --dryrun :::: ./a.sh ./a.sh
#!/bin/bash #!/bin/bash
#!/bin/bash 
#!/bin/bash curl http://127.0.0.1:81/a.php
 #!/bin/bash
 curl http://127.0.0.1:81/a.php
curl http://127.0.0.1:81/a.php #!/bin/bash
curl http://127.0.0.1:81/a.php 
curl http://127.0.0.1:81/a.php curl http://127.0.0.1:81/a.php

This is clearly not what you intended. So what is going on?

If you analyse the output you see each line from the first file is combined with each line from the second file. It is even more obvious with:

file-b:
1
2
3

$ parallel --dryrun :::: b b
1 1
1 2
1 3
2 1
2 2
2 3
3 1
3 2
3 3

The mistake is in ::::. :::: makes GNU Parallel read the content of the file as arguments. And since you gave 2 files, it makes the cross-product of those.

What you wanted was ::::

$ parallel --dryrun ::: ./a.sh ./a.sh
./a.sh
./a.sh
2

Your parallel usage is wrong.

Somehow the 4th run will execute

curl http://127.0.0.1:81/a.php curl http://127.0.0.1:81/a.php

I don't fully understand the syntax but this would work:

yes | head -n4 | parallel ./a.sh

Explanation from the man page

A job can be a single command or a small script that has to be run for each of the lines in the input.

  • yes, I think you're right, the way I use parallel is incorrect. And the name resolve is correct now! Well, I guess I have to focus on my initial issue now. Thanks! – cwhsu Sep 21 '16 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.