2

I have a CSV with 2 columns, ID and a list of image urls separated by |, the number of files is variable, example:

ID_0;https://via.placeholder.com/350x150.jpg|https://via.placeholder.com/250x150.jpg
ID_1;https://via.placeholder.com/350x150.jpg|https://via.placeholder.com/250x150.jpg
ID_2;https://via.placeholder.com/450x150.jpg

Is there a way with wget to download all the images in a folder named with the value of ID?

EDIT:

This is what I've tried so far:

#!/bin/bash

while IFS=';' read -r product_code img
do
    wget $img -P $product_code
done < test.csv
6
  • Yes, there is a way. What have you tried so far? Dec 19, 2022 at 8:12
  • CSV with both ; and | as delimiter ? oO Dec 19, 2022 at 8:24
  • @GillesQuenot consider it an array of images
    – LS_
    Dec 19, 2022 at 8:26
  • @MarcusMüller I've added an example of what I've tried so far, problem is I don't understand how to separate the images list
    – LS_
    Dec 19, 2022 at 8:28
  • @LS_ that looks pretty promising! Glad that Gilles answered so quickly based on that :) Dec 19, 2022 at 8:39

4 Answers 4

2

One simple way:

#!/bin/bash

while read -r id urls; do
    ( mkdir -p "$id" && cd "$id" && wget $urls )
done < <(sed 's/[|;]/ /g' file)

Output

ls ID_*
ID_0:
250x150.jpg  350x150.jpg

ID_1:
250x150.jpg  350x150.jpg
4
  • Thank you! Sorry but I forgot to mention that the images column might have different number of files, will add it to the question.
    – LS_
    Dec 19, 2022 at 8:34
  • @LS_ is the number unlimited, or is it always between 0 and 2? (in that case you get empty url2, and albeit that'll make wget emit an error, it's not a problem at all) Dec 19, 2022 at 8:39
  • @MarcusMüller it is unlimited, I don't know beforehand the maximum number of files
    – LS_
    Dec 19, 2022 at 8:41
  • 1
    looks a lot cleaner than mine, thank you :)
    – LS_
    Dec 19, 2022 at 9:09
1

This is the solution I came up with:

#!/bin/bash

while IFS=';' read -r product_code image_list # Parse each line of the CSV and assing its ; separated values
do
    img_array=( $(echo "$image_list" | tr "|" "\n") ) # Create an Array of urls starting from a | separated string
    for image in "${img_array[@]}" # Parse each element of the Array
    do
        wget "$image" -P "$product_code" # Download Image in specific folder
    done
done < test.csv

Regarding the input file, these are conditions:

  • The first column product_code is a string which can contain A-z, 0-9, _ and spaces;
  • The second column image_list is a list of urls separated by a | character, with variable length, each file name can contain A-z, 0-9, _;
0
1
tr ';|' '\t\t' <file | xargs -L 1 sh -c 'dir=$1; shift; wget -q -P "$dir" "$@"' sh

This assumes that the ID and URLs do not contain the character ; or |, or any whitespace character. It replaces each ; and | character with a tab and then repeatedly calls a short inline shell script using xargs with each line of the input (broken up into separate strings on the tabs) as the arguments.

The short inline shell script shifts off the first command line argument, the ID, into the variable dir and then calls wget with this as the argument to the -P option (--directory-prefix) and the rest of the arguments as the URL operands.

Testing:

$ tree
.
`-- file

0 directories, 1 file
$ tr ';|' '\t\t' <file | xargs -L 1 sh -c 'dir=$1; shift; wget -q -P "$dir" "$@"' sh
$ tree
.
|-- ID_0
|   |-- 250x150.jpg
|   `-- 350x150.jpg
|-- ID_1
|   |-- 250x150.jpg
|   `-- 350x150.jpg
|-- ID_2
|   `-- 450x150.jpg
`-- file

3 directories, 6 files

If your xargs support it, this could run in parallel using -P n (with n being some number, like 4) with xargs.


If you're lazy, you can shorten the above into

tr ';|' '\t\t' <file | xargs -L 1 sh -c 'wget -q -P "$0" "$@"'

... where we abuse the fact that the first operand of sh -c is what's stored in $0, and that this is not part of the list of positional parameters, "$@". However, it's customary to put the name of the script or shell into $0 (which is what the trailing sh is in the code above the divider). The shell will use $0 in its diagnostic messages.

1
  • Beautiful :) Thank you very much for the detailed explanation, you gave me lot of material to look up
    – LS_
    Dec 19, 2022 at 22:31
1

If you have GNU Parallel:

cat file |
  parallel --colsep '[|;]' 'mkdir -p {1} && cd {1} && wget {=1 $arg[1]=""; $_="@arg"; uq(); =}'

Explanation:
  $arg[1]="" -- remove first argument
  $_="@arg" -- join arguments with " "
  uq() -- do not quote the result

Or:

cat file |
  parallel --colsep ';' 'mkdir -p {1} && cd {1} && wget {=2 s/\|/ /g; uq(); =}'

Explanation:
  2 s/\|/ /g -- replace | with " " in argument 2

This may be handy to download more urls in parallel.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .