5

I want my machine to automatically download some files. That doesn't have to be very efficient. So I decided to do this with a bash script.

It works so far when I encode the URL hardly. But I want to get the files retrieved in irregular order and I thought I would use simple variables. How do I get the random number into my variable?

My approach

data_link0="https://example.com/target1.html"
data_link1="https://example.com/target2.html"
data_link2="https://example.com/target3.html"
data_link3="https://example.com/target4.html"

useragent0="Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
useragent1="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
useragent3="Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"

wget --user-agent="$user_agent[$((RANDOM % 3))]" "$datei_link$((RANDOM % 3))"

unfortunately does not work.

  • 1
    You should know that $(( RANDOM % k )) does not generate a random number uniformly from 0 to k - 1, unless k is a power of 2. (Proof: k does not divide 32768.) If it is important that your distribution be uniform, you'll need to take extra measures to effect this. – wchargin Mar 14 '18 at 4:22
8

As far as you need to retrieve all the urls, a better way would be using (GNU/linux coreutils) (or sort -R coreutils too):

shuf file | xargs wget

File :

$ cat file
"https://example.com/target1.html"
"https://example.com/target2.html"
"https://example.com/target3.html"
"https://example.com/target4.html"

man 1 shuf

NAME

shuf - generate random permutations


New comments, new needs, new code :

(requiring random user-agent)

$ cat uas
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.52 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.52 Safari/537.36 OPR/15.0.1147.100
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko

Code :

shuf file | while read url; do
    wget --user-agent="$(shuf -n1 uas)" "$url"
done

If you prefer to keep your way (one url) :

data_link=(
    "https://example.com/target1.html"
    "https://example.com/target2.html"
    "https://example.com/target3.html"
    "https://example.com/target4.html"
)
user_agent=(
    "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
    "Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"
)

wget --user-agent="${user_agent[RANDOM % ${#user_agent[@]} ]}" "${data_link[RANDOM % ${#data_link[@]}]}"

Your way for all urls and user-agent (both randomized) :

for i in $(seq 0 $((${#data_link[@]} -1)) | shuf); do
    wget -U "${user_agent[RANDOM % ${#user_agent[@]}]}" "${data_link[i]}"
done
  • Wow, it runs really good. But ... … now I can not fill in the option "--user-agent=" How can I fill in an randomized *user agent* from a file? – Wilhelm Mar 13 '18 at 21:56
  • Something like uas=( firefox waterfox chromium opera ); shuf file | xargs wget --user-agent="${uas[ RANDOM % ${#uas[@]}]}" (adapt to your needs, this is not real user-agents) – Gilles Quenot Mar 13 '18 at 22:05
  • 1
    Out of curiosity, is there any reason to prefer shuf over sort -R? – martin Mar 14 '18 at 8:47
  • 1
    Both works, up to OP :) Added it at the beginning of the post – Gilles Quenot Mar 14 '18 at 12:49
4

Instead of defining separate variables for each string, define an array. Use ${ar[123]} to access element 123 of the array ar and ${#ar[@]} to get the size of the array.

data_link=(
    "https://example.com/target1.html"
    "https://example.com/target2.html"
    "https://example.com/target3.html"
    "https://example.com/target4.html"
)
user_agent=(
    "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1"
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
    "Mozilla/5.0 (Windows 7; ) Gecko/geckotrail Firefox/firefoxversion"
)

wget --user-agent="${user_agent[RANDOM % ${#user_agent[@]}]}" "${data_link[RANDOM % ${#data_link[@]}]}"
  • Good if OP want only to retrieve one url at a time ;) (And I get a syntax error) – Gilles Quenot Mar 13 '18 at 22:17
  • And no need $(( )) : uas=( foo bar base); echo "${uas[RANDOM % ${#uas[@]}]}" – Gilles Quenot Mar 13 '18 at 22:22
  • I tested it, but I get this error 'Syntax error: "(" unexpected' – Wilhelm Mar 13 '18 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.