3

Let's say I have a .txt file where I have a list of image links that I want to download. example:

image.jpg
image2.jpg
image3.jpg

I use: cat images.txt | xargs wget and it works just fine

What I want to do now is to provide another .txt file with the following format:

some_id1 image.jpg
some_id2 image2.jpg
some_id3 image3.jpg

What I want to do is to split each line in the ' ' , download the link to the right, and change the downloaded file-name with the id provided to the left.

I want to somehow use wget image.jpg -O some_id1.jpg for each separate line.

So the output will be:

some_id1.jpg
some_id2.jpg
some_id3.jpg

Any ideas ?

  • awk can give solution. it can parse file line by line and compose commands. But there is danger to first and second column be nonunique (eg try to download file with name already used), so better to download to temporary folder. – mmv-ru Jul 4 '16 at 15:41
4

This might do your job,

while read a b
 do
    wget "$b" -O "$a".jpg
    printf "$a".jpg"\n%s" >> newfile
done  < images.txt
  • I should be more careful before posting this. please see the updated one, earlier one was having redirection bug. Try this one, this should work flawlessly. – Rahul Jul 7 '16 at 12:38
2

To test:

$ awk '{ ext=gensub(/.*(\.[^.]*$)/,"\\1",1);
         print "wget " $2 " -O " $1 ext}' images.txt
wget image.jpg -O some_id1.jpg
wget image2.jpg -O some_id2.jpg
wget image3.jpg -O some_id3.jpg

To run, pipe into bash or sh like so:

awk '{ ext=gensub(/.*(\.[^.]*$)/,"\\1",1);
       print "wget " $2 " -O " $1 ext}' images.txt | 
  bash

Add single-quotes in the print statement if needed (e.g. if the fields are separated by tabs and the fields can contain spaces or other characters problematic to the shell. or if field 2 is a complete URL with ampersands &, semi-colons ; etc). e.g.

awk '{ ext=gensub(/.*(\.[^.]*$)/,"\\1",1); 
       print "wget '\''" $2 "'\'' -O '\''" $1 ext"'\''"}' images.txt
wget 'image.jpg' -O 'some_id1.jpg'
wget 'image2.jpg' -O 'some_id2.jpg'
wget 'image3.jpg' -O 'some_id3.jpg'

BTW, that's how you embed single-quotes in single-quoted strings: '\'' (end single-quote, a backslash-escaped single-quote, start single-quote again)


Extra Info:

1. You may need to do extra processing on $2 (with awk's gsub() function) before the print statement if there's any chance that $2 might contain one-or-more apostrophes. e.g.

gsub(/'\''/,"'"'\\\\''"'",$2);

That almost-unreadable mess of single-quotes, double-quotes, slashes, and backslashes transforms all 's in the second field to '\''. Quoting can be a real pain sometimes (although it wouldn't be quite as ugly in a #!/usr/bin/awk -f script rather than a single-quoted one-liner).

2. Alternatively, if $2 is a URL, it might be worth changing single-quotes to %27 instead, which is the URL entity encoding for an apostrophe. The shell doesn't care about % symbols, certainly not if they're inside single-quotes, but it cares a lot about quotes.

Similarly, you might want to exclude all punctuation characters when extracting the .extension. e.g.

ext=gensub(/.*(\.[^[:punct:]]*)/,"\\1",1); 

NOTE, though, that this can get quite complicated, depending on exactly where and how the image filename is embedded in, e.g., a URL.

0

Using awk:

awk '{split($2, p, "."); system(sprintf("wget -O %s.%s %s\n", $1, p[length(p)], $2))}' images.txt

Break down

  1. First we use awk to print out roughly the command we want:

    awk '{printf("wget -O %s %s\n", $1, $2)}' images.txt
    
  2. Then we extract the extension by splitting the URL on '.' and append it to the first argument:

    awk '{split($2, p, "."); printf("wget -O %s.%s %s\n", $1, p[length(p)], $2)}' images.txt
    
  3. Finally we change the printf to an sprintf and wrap is in a system command to cause awk to execute the command:

    awk '{split($2, p, "."); system(sprintf("wget -O %s.%s %s\n", $1, p[length(p)], $2))}' images.txt
    

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.