How do I prevent wget from loading Apache directory listings in different orders?

Question

I’m trying to recursively download a directory using this command:

wget -r -l2 --no-parent -A "*.pdf" -nd -N -e robots=off http://…

The script gets executed every 15 minutes, so I want to reduce the requests as much as possible. I saw that there are a lot of requests like ?C=M;O=A which originate from the directory order modifier buttons like Name, Date or Size (example). How can I prevent these requests from happening?

Answer 1

As pointed out in an answer to a related question, wget's documention says:

Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings.

You will probably have to use a different tool such as pavuk or httrack. These two programs provide name filtering that is more robust than wget's.

As an alternative, you can download the directory using wget and filter it afterwards.

find . -name '*\?' -delete
# Here is a less dangerous pattern that matches the string you provided more closely:
find . -name '*\??=?;?=?' -delete

Answer 2

I'd use httrack if you're trying to mirror a directory of *.pdf files. A command like this worked for me:

 % httrack http://my.server.com/dir/with/pdfs/ +mime:text/html +*.pdf --update

You can check out more about httrack at the following URLS:

• Basic Users Guide
• Download Links
• example commands
• Blog article with additional examples

I should mention one major advantage to using httrack over wget is that httrack will only download files that have changed/updated.