New answers tagged

3

Posting an answer as requested: Use the --spider option: wget -r -nv --spider http://example.com Then you can parse the structure of the site from the output. This won't download files that stand no chance to contain links, such as images.


0

Try using the --reject-regex switch of wget. You could probably do something like: wget --recursive --no-parent --reject-regex '[^?]' url


2

Just checked the sources of bash 4.2: the proxy support for /dev/tcp and /dev/udp is not implemented. The only option you have is to start a copy of bash with tsocks library preloaded (see man tsocks for details) So, you would also need to set up a socks proxy server.


0

The below solution will only work for a not formatted, standard apache2 generated directory index. You can wget the index file and parse it with grep and cut for example : #this will download the directory listing index.html file for /folder/ wget the.server.ip.address/folder/ #this will grep for the table of the files, remove the top line (parent ...


1

A good way (not involving the webserver logs) is to use the --debug flag and grep for ^Referer: On the command line: wget -r -nv --spider http://www.domain.com/ 2>&1 | egrep -A 1 '(^---response end---$|^--[0-9]{4}-[0-9]{2}-[0-9]{2}|^[0-9]{4}-[0-9]{2}-[0-9]{2} ERROR|^Referer:|^Remote file does not)' You can do similar grepping on your log. Caveat: ...


2

curl -o dl.html -s -m 10 --retry 0 "$URL" Note: --retry is not really necessary, but is here for completeness' and because the defaults might be set in an environment or rc file.



Top 50 recent answers are included