I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line:

wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/archive/xfce-4.6.2/src/

It only downloads the index page and then deletes it automatically.


--2016-07-01 16:56:02--  http://www.xfce.org/archive/xfce-4.6.2/src/
Resolving www.xfce.org (www.xfce.org)...
Connecting to www.xfce.org (www.xfce.org)||:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://archive.xfce.org/xfce/4.6.2/src/ [following]
--2016-07-01 16:56:17--  http://archive.xfce.org/xfce/4.6.2/src/
Resolving archive.xfce.org (archive.xfce.org)...
Connecting to archive.xfce.org (archive.xfce.org)||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                [ <=>                                                                      ]   8.05K  --.-KB/s   in 0.03s  

2016-07-01 16:56:23 (247 KB/s) - ‘index.html’ saved [8239]

Removing index.html since it should be rejected.

FINISHED --2016-07-01 16:56:23--
Total wall clock time: 21s
Downloaded: 1 files, 8.0K in 0.03s (247 KB/s)

The web-directory contains a lot of tar.bz2 files. Can anyone tell me where I'm going wrong?

My wget version is 1.16.3

  • try adding --no-parent option – Rahul Jul 1 '16 at 11:42
  • It didn't work. – Abhimanyu Saharan Jul 1 '16 at 11:43
  • It is still not working – Abhimanyu Saharan Jul 1 '16 at 11:45
  • do you want to exclude index.html ? – Rahul Jul 1 '16 at 11:46
  • Yes. That would be good. But, before that I want to download the files from the directory – Abhimanyu Saharan Jul 1 '16 at 11:47

Seems like it is not trivial to get directory listing over http; I could get the bz2 files using bellow:

wget -k -l 0 "http://archive.xfce.org/xfce/4.6.2/src/" -O index.html ; cat index.html | grep -o 'http://archive.xfce.org/xfce/4.6.2/src/[^"]*.bz2' | uniq -c | xargs wget
  • Can you please explain what you have done over here? – Abhimanyu Saharan Jul 1 '16 at 12:39
  • First download the index file of the page; pipe it to grep to fetch the bz2 file names. Then uses another wget to fetch those files. Not very efficient but it works I think. Did you try it? – coffeMug Jul 1 '16 at 12:43
  • sort is not necessary and uniq is to avoid duplicates. – coffeMug Jul 1 '16 at 12:44
  • Yes, it worked. – Abhimanyu Saharan Jul 1 '16 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.