Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I’m trying to recursively download a directory using this command:

wget -r -l2 --no-parent -A "*.pdf" -nd -N -e robots=off http://…

The script gets executed every 15 minutes, so I want to reduce the requests as much as possible. I saw that there are a lot of requests like ?C=M;O=A which originate from the directory order modifier buttons like Name, Date or Size (example). How can I prevent these requests from happening?

share|improve this question
    
Related: serverfault.com/questions/156045/… –  Evan Teitelman Apr 22 '13 at 22:52
add comment

2 Answers

up vote 3 down vote accepted

As pointed out in an answer to a related question, wget's documention says:

Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings.

You will probably have to use a different tool such as pavuk or httrack. These two programs provide name filtering that is more robust than wget's.

As an alternative, you can download the directory using wget and filter it afterwards.

find . -name '*\?' -delete
# Here is a less dangerous pattern that matches the string you provided more closely:
find . -name '*\??=?;?=?' -delete
share|improve this answer
add comment

I'd use httrack if you're trying to mirror a directory of *.pdf files. A command like this worked for me:

 % httrack http://my.server.com/dir/with/pdfs/ +mime:text/html +*.pdf --update

You can check out more about httrack at the following URLS:

I should mention one major advantage to using httrack over wget is that httrack will only download files that have changed/updated.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.