Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very efficient. Is there some way to ignore the small ones when downloading?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

If you use a http HEAD request, only the headers will be returned.

Here's a sketchy approach (assuming you have a list of urls). threshold=expr 100 \* 1024

for url in ${list_of_urls} ; do    
    size=`curl -s --head ${url} | grep 'Content-Length:' | cut -d ' ' -f 2` 

    if [ ${size} -gt ${threshold} ] ; then  
        curl -s ${url}
    fi    
done  
share|improve this answer
    
It is unclear, how this is relevant to the problem of downloading images. –  rozcietrzewiacz Nov 18 '11 at 22:34
    
@rozcietrzewiacz That's apartial answer, showing how to determine which images match the length requirement and download them. The missing piece is parsing the web pages to gather the image URLs. –  Gilles Nov 19 '11 at 2:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.