1

I'm using wget --mirror --convert-links to mirror a web site, and I find that when the site contains a link to a subdirectory like:

<a href="subdir">

which would display the contents of subdir/index.html in a web browser, wget is fetching the documents that are linked to in the index.html, but not the index.html itself.

If I change the link to:

<a href="subdir/">

then the index.html is also fetched.

Is there a flag I can put on the wget command to get it to do this automatically?

I'm using wget version 1.14 under CentOS 7.

2 Answers 2

0

Wget should save the page as subdir. Target path and filename are based on the link structure (and not on the final URLs after applying any redirections).

You could add option -E to save the document as subdir.html. I know, it's not perfect.

5
  • 1
    What I find is that wget creates a directory called subdir, and that contains any files with URLs of the form "subdir/file.html". But the index for that subdirectory (index.html) is never fetched. Jan 13, 2021 at 19:33
  • 1
    According to my log the index is fetched, but gets overwritten by the subdirectory with the same name if there are files under subdir. :(
    – Freddy
    Jan 13, 2021 at 20:04
  • Bummer. It would be nice if wget let me supply a file name to be used for the index in cases like that. E.g., "--index-file=index.html". Jan 13, 2021 at 20:15
  • 1
    There is one, --default-page=xxx (default index.html), but that only works for links with slashes (subdir/). Have you tried -E?
    – Freddy
    Jan 13, 2021 at 20:24
  • -E doesn't do it (and it also mis-names some files that really should have extensions other than .html). Jan 13, 2021 at 21:16
0

I encountered this problem too. It seems to be long known and impossible to do that with pure wget. I opened an issue on wget tracker. Here I propose several solutions I came up with.

Manually download directories

First, you need to download all you can:

wget -c --no-check-certificate --recursive --page-requisites --no-clobber --domains mysite.org --exclude-directories=/login https://mysite.org

Then, manually find a list of local directories (-type d) you need to save:

find mysite.org -path 'mysite.org/static' -prune -o -type d

Since static won't likely be browsed by people, we don't save it as index.html, and find action -prune disables search on that path. It is explained in more details on stackoverflow. When you want to disable several paths, use parentheses. The full script would be:

for dir in `find mysite.org \( \
        -path 'mysite.org/static' -o \
        -path 'mysite.org/media' \)
    -prune -o -type d`
do
    echo $dir
    wget --force-directories $dir -O $dir/index.html
done

Since we download files one by one, we save them with full local path with --force-directories (otherwise they would be saved to the current directory). Also, we need to manually provide the output file name with -O option (otherwise wget will refuse to save the result to an existent directory).

Unfortunately, find will also list directories static and media, so you can use grep after that (I ignored that). You could also have saved the needed directory names to a file and use that with wget -i links.txt - however, in that case you wouldn't be able to provide output file names for those links.

Download only directories

I tried to download only directories (links ending on slash) with:

wget -pr -A '/,index.html' https://mysite.org/

Here, -A/--accept is a comma-separated list of file name suffixes or patterns to accept. -p, as before, is to save page requisites (it also conserves local directory structure) and -r is for recursive. The first suffix means that we have a directory (ending with '/'), while the second is necessary to save that: without that the resulting index.html will be immediately deleted.

Unfortunately, this didn't work for my site, because some links ended with slashes, and some did not. There is no way to otherwise distinguish a "directory" path from a final link, and a slash in the end of a link looks like a redundant symbol, so I gave up this idea.

Final remarks

Downloading a site can take a while, so if you do it from a server and want to disconnect after some time, use nohup.

I'd like to mention a curious thing I discovered. Sometimes wget saved html pages together with server response, that is one could see at the start of a page several lines of technical information ("HTTP/1.1 200 OK", etc). I'm not sure whether that was a problem with wget or with my old nginx server, but it can also be manually fixed by re-downloading the broken pages:

grep -RIl nginx * > bad.txt
wget -x -i bad.txt

Make sure to remove broken pages first, because wget won't re-download them if they exist (I could not find a key to overwrite that without -r or -p).

Maybe this detail is not relevant to the question, but it shows that a manual intervention could still be needed for wget results, and you should be ready for that.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .