0

I'd like to do this script in bash, so I can understand it and write it in python (that I know much less) later.

I have some files in data. 3, nothing more.

https://www.example.com/data/file27-1.jpg
https://www.example.com/data/file51-2.jpg
https://www.example.com/data/file2576-3.jpg
URL='https://www.example.com/data/file'
i=1 ; j=1
for i in {1..3000} ; do
  curl -I ${URL}file${i}-{j}.jpg
  # here, pipe result into grep,
  ### if !200 ((i++)) (so do nothing) ;
  ### if 200 wget $URL$i$j and ((j++)), and i=i-1 so I can have the same $i at the next loop, for a incremented $j
  " and go on into the the for loop
done

But curling for 3000 links individually takes some times. I'd like to parralelize the curl -I URL in some way, and when I get a 200 response, stop all process requesting as there won't be two files with the same $j value, add 1 to $j, and take everything back to appropriate values $i and $j and go on.

I'm stuck at parallelizing (but found many threads on it), but the part that really blocks me is where a 200 would kill all curl processus, and then resume to the 200 OK $i and $j value.

I hope I've been understandable. I didn't wrote a sample script yet, I'm looking into methods of achieving it.

Thanks


Edit

#ps -ax | grep "curl" | grep -v "grep" | awk '{print $1}'| xargs kill -9 I figured out I can use that command to kill all curl request, that I can use in a if 200 condition, and then re-setting $i value with i=i-1, increment $j, and go on in the loop.

But at this stage, nothing is parallelized : I can find out how, with xargs, I can parralel curl request, but I can't do it to increment its value.

I think of a temporary file with URL generated in it, but I'd rather it to be generated as the script goes.

1

Here is a little snippet that can help you along with what you are trying to do, I hope the logic is fine :

#!/bin/bash
i=0
j=0
pid=0
ppid=0
#Enable job control; It's not used here but it can be usefull if you need to do more job control
set -m 
for i in {1..3000} ; do
    #Execute each curl in the background to have a sort of multi-threading and get get the HEAD response status and put it in file descriptor 3 to be gathered later
    exec 3< <(curl -I ${URL}file${i}-{j}.jpg | head -n 1 | cut -d$' ' -f2)
    #Get the pid of the background job
    pid="$!"
    #Get the parent pid of the background job
    ppid="$(ps -o ppid= -p $pid)"
    #Gather the HTTP Response code
    status="$(cat <&3)"
    #Check
    if [ "$status" -eq 200 ] ; then
        i="$(($i - 1))"
        j="$(($j + 1))" 
        echo "kill all previous background process by their parent"
        pkill -P $ppid
    else 
      i="$(($i + 1))"
    fi 
    echo " status : $status"
    echo " parent : $ppid"
    echo " child : $pid"
done
2
  • Oh my ! I just looked into it, haven't test it as it is since I have some more things to do with the vars but it has all the kind of feature I'd need (format date for i, hash for j...). I would edit your answer to put pkill before i and j dec in if [ "$status" -eq 200 ] tho, so background process won't change them in the short meantime. Also, As I see your code, I have some difficulty to understand how the [if] thing would be in the background. I do understand only the curl for current loop is, doesnt it ? I don't mark the topic answered for now, but I give you a "up" for that ! thanks Dec 15 '20 at 21:39
  • I see . Normally the script should work correclty, [If] statement is on the foreground but it will be executed on each iteration and <status> variable will get the response everytime a background job ended.
    – Reda Salih
    Dec 15 '20 at 22:07
1

If you have GNU Parallel something like this should work (i=1..3000; j=1..1000):

do_j() {
  j=$1
  URL='https://www.example.com/data/file'
  seq 3000 |
    parallel --halt soon,success=1 -j100 "curl -I ${URL}file{}-${j}.jpg | grep 'HTTP.* 200 OK'"
}
export -f do_j
seq 1000 | parallel -j1 do_j

Adjust -j1 and -j100 to get more or fewer in parallel.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.