0

I'd like to do this script in bash, so I can understand it and write it in python (that I know much less) later.

I have some files in data. 3, nothing more.

https://www.example.com/data/file27-1.jpg
https://www.example.com/data/file51-2.jpg
https://www.example.com/data/file2576-3.jpg
URL='https://www.example.com/data/file'
i=1 ; j=1
for i in {1..3000} ; do
  curl -I ${URL}file${i}-{j}.jpg
  # here, pipe result into grep,
  ### if !200 ((i++)) (so do nothing) ;
  ### if 200 wget $URL$i$j and ((j++)), and i=i-1 so I can have the same $i at the next loop, for a incremented $j
  " and go on into the the for loop
done

But curling for 3000 links individually takes some times. I'd like to parralelize the curl -I URL in some way, and when I get a 200 response, stop all process requesting as there won't be two files with the same $j value, add 1 to $j, and take everything back to appropriate values $i and $j and go on.

I'm stuck at parallelizing (but found many threads on it), but the part that really blocks me is where a 200 would kill all curl processus, and then resume to the 200 OK $i and $j value.

I hope I've been understandable. I didn't wrote a sample script yet, I'm looking into methods of achieving it.

Thanks


Edit

#ps -ax | grep "curl" | grep -v "grep" | awk '{print $1}'| xargs kill -9 I figured out I can use that command to kill all curl request, that I can use in a if 200 condition, and then re-setting $i value with i=i-1, increment $j, and go on in the loop.

But at this stage, nothing is parallelized : I can find out how, with xargs, I can parralel curl request, but I can't do it to increment its value.

I think of a temporary file with URL generated in it, but I'd rather it to be generated as the script goes.

2 Answers 2

1

Here is a little snippet that can help you along with what you are trying to do, I hope the logic is fine :

#!/bin/bash
i=0
j=0
pid=0
ppid=0
#Enable job control; It's not used here but it can be usefull if you need to do more job control
set -m 
for i in {1..3000} ; do
    #Execute each curl in the background to have a sort of multi-threading and get get the HEAD response status and put it in file descriptor 3 to be gathered later
    exec 3< <(curl -I ${URL}file${i}-{j}.jpg | head -n 1 | cut -d$' ' -f2)
    #Get the pid of the background job
    pid="$!"
    #Get the parent pid of the background job
    ppid="$(ps -o ppid= -p $pid)"
    #Gather the HTTP Response code
    status="$(cat <&3)"
    #Check
    if [ "$status" -eq 200 ] ; then
        i="$(($i - 1))"
        j="$(($j + 1))" 
        echo "kill all previous background process by their parent"
        pkill -P $ppid
    else 
      i="$(($i + 1))"
    fi 
    echo " status : $status"
    echo " parent : $ppid"
    echo " child : $pid"
done
2
  • Oh my ! I just looked into it, haven't test it as it is since I have some more things to do with the vars but it has all the kind of feature I'd need (format date for i, hash for j...). I would edit your answer to put pkill before i and j dec in if [ "$status" -eq 200 ] tho, so background process won't change them in the short meantime. Also, As I see your code, I have some difficulty to understand how the [if] thing would be in the background. I do understand only the curl for current loop is, doesnt it ? I don't mark the topic answered for now, but I give you a "up" for that ! thanks Commented Dec 15, 2020 at 21:39
  • I see . Normally the script should work correclty, [If] statement is on the foreground but it will be executed on each iteration and <status> variable will get the response everytime a background job ended.
    – Reda Salih
    Commented Dec 15, 2020 at 22:07
1

If you have GNU Parallel something like this should work (i=1..3000; j=1..1000):

do_j() {
  j=$1
  URL='https://www.example.com/data/file'
  seq 3000 |
    parallel --halt soon,success=1 -j100 "curl -I ${URL}file{}-${j}.jpg | grep 'HTTP.* 200 OK'"
}
export -f do_j
seq 1000 | parallel -j1 do_j

Adjust -j1 and -j100 to get more or fewer in parallel.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .