3

I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.

1,text1,text2,http://image_a.jpg
2,text1,text2,http://image_b.jpg
3,text1,text2,http://image_c.jpg
.
.
.

The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script

while IFS=, read col1 col2 col3 col4
do
   curl ${col4} > ${col1}.jpg
done < myfile.csv

Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.

I realize gnu parallel and wget may be better options but they are not available in my unix environment.

Thanks.

6

2 Answers 2

2

IF Scott can use one of the proposed ways to get GNU Parallel https://oletange.wordpress.com/2018/03/28/excuses-for-not-installing-gnu-parallel/ then this might work:

parallel --colsep , 'curl {4} > {1}.jpg' :::: input.csv
1

It seems to me that you'd have to build up the curl commands and then pass them to xargs:

awk -F, '{print "curl '\''" $4 "'\'' > '\''" $1".jpg'\''"}' < input.csv | xargs -P2 -I {} sh -c '{}'

The command looks a little ugly because I wrapped single quotes around the URL and sequence number/filenames, in case there are any shell-special characters in them (e.g. &).

2
  • I needed to remove the -n1 parameter to xargs and then this works. With that param, each token of the print was passed to sh separately. Nov 2, 2018 at 18:04
  • sorry for the confusion, @Scott! I've updated the post to remove the -n1
    – Jeff Schaller
    Nov 2, 2018 at 18:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .