3

I'm trying to mirror a website locally. However, I've been running into a segmentation fault at some consistent point in the download, on a different domain than the site I'm targeting (probably due to --page-requisites).

2018-04-09 04:58:32 (346 KB/s) - './not-website.com/2017/06/28/xyz/index.html' saved [145810]

29247 Segmentation Fault      (core dumped) wget --directory-prefix="${DL_ROOT}" --recursive --page-requisites --span-hosts --tries="${TRIES_NUM}" --timeout="${TIMEOUT_NUM}" --reject="*.tar" --convert-links --adjust-extension --continue --no-check-certificate "http://website.com/"

As a result, I assume that the segmentation fault is due to a wget trying to download a specific website but failing.

However, the error message doesn't seem to tell me what address wget is failing on. It only tells me the last successful download. How can I figure out where/why wget fails with this segfault?

There is a 55M core file that the error seems to reference in (core dumped), but it's not in plain text. Is the information I need in there, and how do I extract that?

I have tested this across distros (Solaris, Debian, Raspbian), and this segfault is consistent, and always after the same address (not-website.com/... in the error message above).

I'm using the command:

$ wget \
    --directory-prefix="${DL_ROOT}" \
    --recursive \
    --page-requisites \
    --span-hosts \
    --tries="${TRIES_NUM}" \
    --timeout="${TIMEOUT_NUM}" \
    --reject="*.tar" \
    --convert-links \
    --adjust-extension \
    --continue \
    --no-check-certificate \
  "http://website.com/"

Additional Information

It's a big site, with quite a bit of media. At the point of failure, the downloaded directory size is about 252M.

Tested on:

GNU Wget 1.18 built on solaris2.10.

-cares +digest -gpgme +https +ipv6 -iri +large-file -metalink -nls 
+ntlm +opie -psl +ssl/openssl 

and

GNU Wget 1.18 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls 
+ntlm +opie +psl +ssl/gnutls

and

GNU Wget 1.16 built on linux-gnueabihf.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +psl +ssl/gnutls
  • Hi and welcome to unix.SE! Can you get the backtrace using gdb core + the bt command? Excellently formatted and formulated first post, by the way! – l0b0 Apr 8 '18 at 22:30
1

Segmentation Fault means the program, in this case, wget, tried to access an invalid memory address and therefore was terminated by the kernel. This typically happen due to a program bug, so while it is quite likely it is being triggered by a specific website or web page (considering you seem to be able to reproduce it quite consistently, on multiple platforms, at the same point), it is still likely you have exposed a bug in wget itself.

In order to find where in wget the segmentation fault is happening, you can use the gdb program (GNU debugger) to get a stack trace of wget at the time it crashed, which is possible since you have a core file. (A core dump is a copy of the image of the running program at the time it was terminated due to an invalid operation such as a Segmentation Fault.)

In order to do so, use the following command:

$ gdb wget core

Which will start the debugger on the wget binary (from the path) and restore the core file (in the current directory) as the image of the running program.

gdb will then print some information about the program and give you a prompt:

$ gdb wget core
GNU gdb (GDB) 7.9
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
Core was generated by `wget --directory-prefix=... --recursive --page-requisites --span-hosts --tries=... --timeout=... --reject=*.tar --convert-links --adjust-extension --continue --no-check-certificate http://website.com/'.
Program terminated with signal SIGSEGV, Segmentation Fault.
(gdb) _

At that point, you can use the command bt (short for "backtrace") to show you what was being executed at the time the program crashed. Which is usually a good place to start looking for the bug.

For instance, you might see something like this:

(gdb) bt
#0  0x00007f5371206363 in __select_nocancel () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000559e5acbf21c in select_fd ()
#2  0x0000559e5acf0bde in wgnutls_poll ()
#3  0x0000559e5acbf3a2 in poll_internal ()
#4  0x0000559e5acbf6ed in fd_peek ()
#5  0x0000559e5ace423d in fd_read_hunk ()
#6  0x0000559e5acd5ef9 in gethttp ()
#7  0x0000559e5acd9b26 in http_loop ()
#8  0x0000559e5ace53c8 in retrieve_url ()
#9  0x0000559e5ace273b in retrieve_tree ()
#10 0x0000559e5acbe67d in main ()

You can then quit gdb with the q (for "quit") command:

(gdb) q

It's usually helpful if you have the "debug symbols" installed. These are the information generated by the compiler for debugging binaries, which is usually stripped for binaries that are installed on a system, so they're smaller in size. That information can be saved to an alternate location (typically under /usr/lib/debug) that can be located by gdb while trying to debug a binary.

With that information present, your backtraces will typically have more information attached to them, such as the name of all internal functions.

On Debian, you can install the debug info for wget with the following command:

$ sudo apt-get install wget-dbgsym

You might also want to install the debug symbols for glibc:

$ sudo apt-get install libc6-amd64-dbgsym

Having said that, before you start looking at why wget crashed, you might want to try the latest version of wget, which it seems is version 1.9.4 that you can download here. That is a source package, so you might need to build from sources to get it to work in your system.

This is because a segmentation fault is typically caused by a bug, and it's quite possible this bug was already fixed in wget and the fix is present in the latest version.

In case you get the same problem in the latest version, consider getting a core file and using gdb to get a backtrace, then report the bug to wget maintainers so they have a chance to address it.

In case it's fixed on latest wget 1.9.4 but it exists in a version of Debian you are using, consider reporting this to Debian, so they can have a chance to backport the patch to their version of wget.

There's also a new project called wget2, it looks like they're trying to replace wget with a new codebase. You might want to check whether that one works or not... It seems recent Debian ship it under the name "wget2".

I hope these pointers are helpful too!

  • 1
    Thanks! Fixed by using wget 1.19.4. I should also note that the script in my original post, since it includes --span-hosts, without --level, was attempting to download the internet. – ning Apr 11 '18 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.