Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using wget to create an HTML mirror of a php templated site on localhost. I am encountering a problem with some directories being mirrored as extensionless documents.

What could be causing this problem?

I am using this command:

$ wget -mk http://www.example.com/

A screenshot of corrupted directories.

share|improve this question
    
Is there some significance to the spelling wGet rather than wget? –  Keith Thompson Apr 14 at 18:13
    
I've seen it written that way in some docs. Is that incorrect? Changed it, since official docs don't capitalize. –  itsjustluck Apr 14 at 18:15
    
what's the URL that leads to becoming-rental-pro? http://localhost/becoming-rental-pro or http://localhost/becoming-rental-pro.html? –  user61786 Apr 14 at 18:20
    
The command name is wget, and Unix command names are case-sensitive. I've seen documentation refer to it as Wget, but I've never seen it referred to as wGet. –  Keith Thompson Apr 14 at 18:21
    
@awk_FTW http://localhost/becoming-rental-pro –  itsjustluck Apr 14 at 18:59

2 Answers 2

What could be causing this problem?

Have you checked that the actual site always returns documents with an extension? There is no protocol which requires this and it is completely normal to not use them; the document type is determined by the http headers and not any suffix on the address. Of course, your file browser does use file extensions to identify documents, so this may be an annoyance.

For example, this page is literally http://unix.stackexchange.com/questions/124741/wget-corrupted-directories, and if you fetch it with wget, it will produce a file wget-corrupted-directories with no extension, which contains html data.

share|improve this answer
    
I'm not sure what you mean. I mean to create directories, not files. To my knowledge directories have no extensions. –  itsjustluck Apr 14 at 19:40
    
This page is not a directory, right? But do you see an .html extension in the browser nav bar? No, because there is no extension associated with the address. It is still an html page, and this is indicated in the http headers sent by the server. Your browser does not use or care about filename extensions, so they are considered superfluous in delivering content. If I download this page with wget, I get a file with no extension. If I download it with wget --html-extension, I get the exact same file and wget adds a .html because the http headers indicate this. –  goldilocks Apr 14 at 19:47
    
Put another way: Do not confuse URLs with file paths. URLs are not file paths or file names, and don't expect them to work like such. They may (or may not) correspond to static files stored on the server, but that is just happenstance. There does not need to be any such relation, and if there is, the names do not have to be in any way the same, because again, a URL is not a file name (that wikipedia page linked is another example -- an html page, but the URL does not end with '.html'). –  goldilocks Apr 14 at 19:50

You are missing --html-extension flag.

I would suggest using this syntax instead:

wget \
 --recursive \
 --no-clobber \
 --page-requisites \
 --html-extension \
 --convert-links \
 --restrict-file-names=windows \
 --domains website.org \
 --no-parent \
     www.website.org
share|improve this answer
    
--html-extension produces the same result. –  itsjustluck Apr 14 at 19:04
    
Also this answer only produces the root index file. –  itsjustluck Apr 14 at 19:10
    
@itsjustluck you'll need to add -m or --mirror –  user61786 Apr 14 at 19:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.