1

I have a CSV with 4 fields (id1,id2,id3,filename) like this:

2,2314,78,/image/1289.jpg
1,19825,179,/image/01288.jpg
2,2585,2281,/image/01287.jpg
3,35879,82,/image/01286.jpg

There is a common URL endpoint for the images like

http://example.com/assets/?mediafile=

The last column is the unique portion of the URL to the image.

I'm trying to download each image, and name of each image is based on the first 3 fields:

id1 is always 1 digit
id2 is really 5 digits with prepending zeros (so 2314 needs to become 02314)
id3 is really 4 digits (78 becomes 0078),

so in total a 10 digit code. So the first image name should be 2023140078.jpg

If I were to download a single file manually, I'd do something like this:

wget -O 2023140078.jpg http://example.com/assets/?mediafile=/image/1289.jpg

Also if I could write back some sort of status or error code to the row or write an error log, would be great.

2
awk -F, '{printf("%d%.5d%.4d %s\n",$1,$2,$3,$4)}' input.csv | while read newfile filename 
do 
wget -O "${newfile}.jpg" http://example.com/assets/?mediafile=${filename} 
done
  • After running this for 200,000 rows I realized that not all filenames are unique, and some IDs have several images, how can I fix this? – zevnyc Jan 17 '17 at 20:00
  • what you mean by its not unique ? are you ok to add the incremental value in the filename ? eg: 2023140078_1.jpg 2023140078_2.jpg ..... 2023140078_200000.jpg – Kamaraj Jan 18 '17 at 8:28
  • Some of the entries have several URLs for the same filename so wget is downloading them all, replacing the first file with the next. Most have only 1 file, but some have several. I'd like for the 1st to stay in the original format and the rest add _1 etc. – zevnyc Jan 18 '17 at 10:25
1

You can set the IFS to a comma and use a while loop to read the 4 variables that uses the file as input:

IFS=','
while read c1 c2 c3 c4; do
  wget -O "$(printf "%d%05d%04d.jpg" "$c1" "$c2" "$c3")" "http://example.com/assets/?mediafile=${c4}"
done < file.txt
  • if second column has the value of 12345, then the filename will be created with 11 digit – Kamaraj Jan 17 '17 at 6:30
  • Sorry, I misread your padding requirements. You can use printf command substitution for that: $ IFS=','; while read c1 c2 c3 c4; do echo wget -O "${c1}$(printf %05d ${c2})$(printf %04d ${c3})" "example.com/assets/?mediafile=${c4}"; done < file.txt – Kyle Jan 17 '17 at 6:31
  • You should edit your answer to improve it.  Comments are often overlooked and sometimes deleted. And you don't need two printfs; just do $(printf "%d%05d%04d.jpg" "$c1" "$c2" "$c3"). – Scott Jan 17 '17 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.