Bash : parallel curl and variables

Question

I'd like to do this script in bash, so I can understand it and write it in python (that I know much less) later.

I have some files in data. 3, nothing more.

https://www.example.com/data/file27-1.jpg
https://www.example.com/data/file51-2.jpg
https://www.example.com/data/file2576-3.jpg

URL='https://www.example.com/data/file'
i=1 ; j=1
for i in {1..3000} ; do
  curl -I ${URL}file${i}-{j}.jpg
  # here, pipe result into grep,
  ### if !200 ((i++)) (so do nothing) ;
  ### if 200 wget $URL$i$j and ((j++)), and i=i-1 so I can have the same $i at the next loop, for a incremented $j
  " and go on into the the for loop
done

But curling for 3000 links individually takes some times. I'd like to parralelize the curl -I URL in some way, and when I get a 200 response, stop all process requesting as there won't be two files with the same $j value, add 1 to $j, and take everything back to appropriate values $i and $j and go on.

I'm stuck at parallelizing (but found many threads on it), but the part that really blocks me is where a 200 would kill all curl processus, and then resume to the 200 OK $i and $j value.

I hope I've been understandable. I didn't wrote a sample script yet, I'm looking into methods of achieving it.

Thanks

---

Edit

#ps -ax | grep "curl" | grep -v "grep" | awk '{print $1}'| xargs kill -9 I figured out I can use that command to kill all curl request, that I can use in a if 200 condition, and then re-setting $i value with i=i-1, increment $j, and go on in the loop.

But at this stage, nothing is parallelized : I can find out how, with xargs, I can parralel curl request, but I can't do it to increment its value.

I think of a temporary file with URL generated in it, but I'd rather it to be generated as the script goes.

Answer 1

Here is a little snippet that can help you along with what you are trying to do, I hope the logic is fine :

#!/bin/bash
i=0
j=0
pid=0
ppid=0
#Enable job control; It's not used here but it can be usefull if you need to do more job control
set -m 
for i in {1..3000} ; do
    #Execute each curl in the background to have a sort of multi-threading and get get the HEAD response status and put it in file descriptor 3 to be gathered later
    exec 3< <(curl -I ${URL}file${i}-{j}.jpg | head -n 1 | cut -d$' ' -f2)
    #Get the pid of the background job
    pid="$!"
    #Get the parent pid of the background job
    ppid="$(ps -o ppid= -p $pid)"
    #Gather the HTTP Response code
    status="$(cat <&3)"
    #Check
    if [ "$status" -eq 200 ] ; then
        i="$(($i - 1))"
        j="$(($j + 1))" 
        echo "kill all previous background process by their parent"
        pkill -P $ppid
    else 
      i="$(($i + 1))"
    fi 
    echo " status : $status"
    echo " parent : $ppid"
    echo " child : $pid"
done

Answer 2

If you have GNU Parallel something like this should work (i=1..3000; j=1..1000):

do_j() {
  j=$1
  URL='https://www.example.com/data/file'
  seq 3000 |
    parallel --halt soon,success=1 -j100 "curl -I ${URL}file{}-${j}.jpg | grep 'HTTP.* 200 OK'"
}
export -f do_j
seq 1000 | parallel -j1 do_j

Adjust -j1 and -j100 to get more or fewer in parallel.