0

I have a csv-file that looks like this (short example, there are actually about 20 images per id).

2463666;client@email.fi;yleiskuva;7;kivirealty-original;7;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/original/209.15555/img483903555766418535.jpg;pääkuva;1;kivirealty-original;1;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/original/209.15555/img2055776735947406126.jpg;yleiskuva;9;kivirealty-www;9;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/www/209.15555/img7955434706446319955.jpg
2538034;client@email.fi;yleiskuva;8;kivirealty-original;8;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/original/209.15555/img3622394024988955125.jpg;yleiskuva;7;kivirealty-original;7;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/original/209.15555/img7456562650934206782.jpg;yleiskuva;7;kivirealty-thumb;7;http://d372r717gpt3jp.cloudfront.net/itemimages/realty/thumb/209.15555/img3483011994211655969.jpg
...

The structure is like this:

id;email;type_of_image;image_order;image_quality;image_transfer_id;image_url;

The bold version is repeated as many times as the source xml has these fields, so there can be (most likely is) different amount of images per id. I need to create a folder and download these images in that folder based on the id, for example

some_path/2463666/img483903555766418535.jpg

There are also image files that are thumbnails and "www-sized", but I only need the ones that have kivirealty-original as a field before the image location.

How can I parse the files I need and then wget those files to a folder that is named by it's id?

The example file is generated by a shell script, and I would need a code that would do this after that file is done.

  • You can parse CSV file with Perl or Python. Since CSV files can have newlines in cell-content, using line oriented tools is a shortcut only beginners try to take. Once you have the cell contents call wget or use the language build in retrieval tools. – Anthon Feb 10 '15 at 11:51
  • Thanks for the help, however I'm in a need for bit more detailed examples. I am fairly new to shell scripting. I have been using awk to get only the main images to use as preview pictues by looking for value pääkuva;1;kivirealty-original;1;, but now I need to download them all (not just pääkuva = main image) and in different folders. – SamSwampwater Feb 10 '15 at 12:00
0
while IFS=";" read id a remain
do
    while [ "$remain" ]
    do
        IFS=";" read a b test d i_url remain <<< "$remain"
        if [ "$test" == "kivirealty-original" ]
        then
            if [ ! -d "some_path/$id" ]
            then
                mkdir -p "some_path/$id"
            fi
            wget -nd -P "some_path/$id" "$i_url"
        fi
    done
done < file.csv
  • This did just what I needed, thank you very much! – SamSwampwater Feb 10 '15 at 14:28
0

I'd use awk to create the commands and pipe the output into sh if all works as desired. Here's a code skeleton (you'll have to add the desired wget options)...

awk -F\; '
{
    printf "mkdir -p %s/%s\n", "somepath", $1
    for (i=7; i<=NF; i+=5) {
        printf "wget ... %s ... %s/%s ...\n", $i, "somepath", $1
    }
}
' | sh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.