here is a shell script which takes domain and its parameters to find status code . this runs way faster due to threading but misses lot of requests.

while IFS= read -r url <&3; do
    while IFS= read -r uri <&4; do
    urlstatus=$(curl -o /dev/null --insecure --silent --head --write-out  '%{http_code}' "${url}""${uri}" --max-time 5 ) &&
    echo "$url  $urlstatus $uri" >> urlstatus.txt &
done 4<uri.txt 
done 3<url.txt

if i ran normally it process all requests but the speed is very low. is there a way through which speed is maintained and it also not misses all requests .

  • Shell scripts don't do "threading", they do asynchronous processes. What do you mean by "misses a lot of requests"? What behaviour are you observing?
    – Kusalananda
    Apr 5 '17 at 12:19
  • Actually there is a txt file suppose contain 5 of domains. And uri file contain 100 uri. So results should be 500 output of lines of data as I run normally, but it misses lot requests like 100-200, Apr 5 '17 at 12:24
  • If I run normal it is compromise of speed. Apr 5 '17 at 12:25
  • 1
    Are you concluding that you miss a lot by counting the lines in the output file? If several processes are writing to the same file at once, you may get garbled output. All output will be there, but it may be wonky. Better to write to separate files and then concatenate these files.
    – Kusalananda
    Apr 5 '17 at 12:26
  • 1
    Try writing to separate files. Then concatenate these to one big file afterwards.
    – Kusalananda
    Apr 5 '17 at 12:34

You are experiencing the problem of appending to a file in parallel. The easy answer is: Don't.

Here is how you can do it using GNU Parallel:

doit() {
    urlstatus=$(curl -o /dev/null --insecure --silent --head --write-out  '%{http_code}' "${url}""${uri}" --max-time 5 ) &&
    echo "$url  $urlstatus $uri"
export -f doit

parallel -j200 doit :::: url uri >> urlstatus.txt

GNU Parallel defaults to serializing the output, so you will not get output from one job that is mixed with output from another.

GNU Parallel makes it easy to get the input included in the output using --tag. So unless the output format is fixed, I would do:

parallel --tag -j200 curl -o /dev/null --insecure --silent --head --write-out  '%{http_code}' {1}{2} --max-time 5 :::: url uri >> urlstatus.txt

It will give the same output - just formatted differently. Instead of:

url  urlstatus uri

you get:

url uri urlstatus
  • this works fine Apr 7 '17 at 8:54
  • actually in my previous and this script , it is not working fine due to that i not included #!/usr/bin/env bash in my script Apr 7 '17 at 9:02

While you can run multiple processes asynchronously (utilizing the "shellcmd &" syntax) in the shell, the sub-process can (and often does) end before your next command can capture its output. And, as @Ole-Tange pointed out, there is no way to ensure writing that output to a file in the correct order!

You many want to use a different scripting language where you can manage multiple threads versus using a shell script...

  • Yeah I am thinking the same, looks like here is not proper control over commands, I will try ole tange code and tell after testing. Apr 6 '17 at 17:39
  • I once read (I can't remember where right now) that a shell script is great for dealing with file manipulation (search, move, delete, etc.) and for dealing with exit errors after a daemon exits (the script is blocked until the process exits), and for some other things, but process/coprocess management isn't one of them.
    – bb_referee
    Apr 6 '17 at 20:30
  • Ole Tange code works fine. Apr 7 '17 at 8:55

To perform multiple curl transfers in parallel, we need to look at another tool: xargs.

If you aren’t familiar with xargs, it is a very powerful linux utility. With it, we can execute multiple (dynamic) curl commands in parallel with very little overhead. Example:

seq 1 3 | xargs -n1 -P3 bash -c 'i=$0; url="http://mytestserver.net/10m_test.html?run=${i}"; curl -O -s $url'

This code will run 3 curl commands in parallel. The -P parameter allows you to set the desired number of parallel executions. In this example, we are using the seq command to pass numerical arguments to our commands so that each URL is unique with a run number. The -n parameter simply limits how many arguments are passed per execution. The -c parameter is where we specify our command to be run.

Note that this example doesn’t give any output, it simply runs the transfers. If you want to save the output, you can use the previous discussion on output format to decide what you want to output and how to save it.

From here, you can expand the number of iterations, pass other interesting parameters (a list of URLs from a file, perhaps), and so on. We often use this type of command when generating background traffic to simulate particular network conditions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.