1

I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line:

wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/archive/xfce-4.6.2/src/

It only downloads the index page and then deletes it automatically.

Output:

--2016-07-01 16:56:02--  http://www.xfce.org/archive/xfce-4.6.2/src/
Resolving www.xfce.org (www.xfce.org)... 138.48.2.103
Connecting to www.xfce.org (www.xfce.org)|138.48.2.103|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://archive.xfce.org/xfce/4.6.2/src/ [following]
--2016-07-01 16:56:17--  http://archive.xfce.org/xfce/4.6.2/src/
Resolving archive.xfce.org (archive.xfce.org)... 138.48.2.107
Connecting to archive.xfce.org (archive.xfce.org)|138.48.2.107|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                [ <=>                                                                      ]   8.05K  --.-KB/s   in 0.03s  

2016-07-01 16:56:23 (247 KB/s) - ‘index.html’ saved [8239]

Removing index.html since it should be rejected.

FINISHED --2016-07-01 16:56:23--
Total wall clock time: 21s
Downloaded: 1 files, 8.0K in 0.03s (247 KB/s)

The web-directory contains a lot of tar.bz2 files. Can anyone tell me where I'm going wrong?

My wget version is 1.16.3

8
  • try adding --no-parent option
    – Rahul
    Jul 1, 2016 at 11:42
  • It didn't work. Jul 1, 2016 at 11:43
  • It is still not working Jul 1, 2016 at 11:45
  • do you want to exclude index.html ?
    – Rahul
    Jul 1, 2016 at 11:46
  • Yes. That would be good. But, before that I want to download the files from the directory Jul 1, 2016 at 11:47

1 Answer 1

1

Seems like it is not trivial to get directory listing over http; I could get the bz2 files using bellow:

wget -k -l 0 "http://archive.xfce.org/xfce/4.6.2/src/" -O index.html ; cat index.html | grep -o 'http://archive.xfce.org/xfce/4.6.2/src/[^"]*.bz2' | uniq -c | xargs wget
4
  • Can you please explain what you have done over here? Jul 1, 2016 at 12:39
  • First download the index file of the page; pipe it to grep to fetch the bz2 file names. Then uses another wget to fetch those files. Not very efficient but it works I think. Did you try it?
    – Vombat
    Jul 1, 2016 at 12:43
  • sort is not necessary and uniq is to avoid duplicates.
    – Vombat
    Jul 1, 2016 at 12:44
  • Yes, it worked. Jul 1, 2016 at 15:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .