I have a csv file with several columns including a sequence number and an image URL. The file is 450+ lines.


The goal is to download the image at the URL with a filename being the sequence. I accomplished this successfully with the following script

while IFS=, read col1 col2 col3 col4
   curl ${col4} > ${col1}.jpg
done < myfile.csv

Now I want to run the curl commands in parallel using xargs -P. (I don't want to just background using & as I want to limit the number of processes.) I see examples using -I and -R but cannot figure out how to use multiple fields from a line of the csv file in the xargs command.

I realize gnu parallel and wget may be better options but they are not available in my unix environment.



It seems to me that you'd have to build up the curl commands and then pass them to xargs:

awk -F, '{print "curl '\''" $4 "'\'' > '\''" $1".jpg'\''"}' < input.csv | xargs -P2 -I {} sh -c '{}'

The command looks a little ugly because I wrapped single quotes around the URL and sequence number/filenames, in case there are any shell-special characters in them (e.g. &).

  • I needed to remove the -n1 parameter to xargs and then this works. With that param, each token of the print was passed to sh separately. – Scott Nelson Nov 2 '18 at 18:04
  • sorry for the confusion, @Scott! I've updated the post to remove the -n1 – Jeff Schaller Nov 2 '18 at 18:06

IF Scott can use one of the proposed ways to get GNU Parallel https://oletange.wordpress.com/2018/03/28/excuses-for-not-installing-gnu-parallel/ then this might work:

parallel --colsep , 'curl {4} > {1}.jpg' :::: input.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.