Tag Info

Hot answers tagged

36

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything but curl Features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


30

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


26

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in not-actually-informative messages. The practical reason is that in unix's very early days, there still ...


19

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


18

If I understand your question correctly (it's not clear what machine is supposed to do what), you are logged into some machine myclient, you have ssh access to another machine myserver, and you want to download a file over HTTP from a remove server www.example.com, with the requirement that the HTTP download must be performed to myclient but the data needs ...


15

(Strange situation, doesn't something like the triangle inequality hold for internet routing?) Anyway, try the following, on A, ssh into B with a -D argument, ssh -D 1080 address-of-B which acts as a SOCKS5 proxy on 127.0.0.1:1080, which can be used by anything supporting SOCKS5 proxied connections. Apparently, wget can do this, by using the environment ...


14

In the unix world, each tool is designed to do one job and do it well. Why would cp worry about outputting progress when another tool like pv does it already? In the same vein, why do so many programs dump stuff to the screen without any pagination? Because there are already tools for that job such as more (or less). Why do most programs that require editing ...


13

If the server is blocking wget, it is most likely doing it on the basis of the "User-agent:" field of the http header, since that is the only way for it to know in the first place. It could also be blocking your IP, in which case using different software won't help, or some scheme which identifies automation on the basis of how rapid a set of requests are ...


11

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document. You could, for example, use lynx -dump -nolist, if you have lynx around. lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, ...


11

As you note, you don't need the whole ImageMagick package. You just need identify. You will also need the libraries the executable links to (and the libraries those libraries link to). > whereis identify identify: /bin/identify /usr/bin/identify /usr/share/man/man1/identify.1.gz > ldd /bin/identify ldd will show a list. When I did this, it ...


11

Use a different separator that doesn't contain any of the characters in the variable. For example, sed "s|^|$URL|" (If you use / as the separator and the pattern or replacement also contain /, then you'd need to escape those.)


10

2 refers to the second file descriptor of the process, i.e. stderr. > means redirection. &1 means the target of the redirection should be the same location as the first file descriptor, i.e. stdout. So this command first redirects stdout to /dev/null and then redirects stderr there as well. This effectively silences all output (regular or error) ...


9

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page: --max-filesize Specify the maximum size (in bytes) of a file to download. If the file ...


9

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So wget ...


9

Wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots= off, wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/


8

On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs. For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...


8

You can do it by telling wget to output its payload to stdout: wget -qO- your_link_here | tar xvz To specify a target directory: wget -qO- your_link_here | tar xvz -C /target/directory Update If you happen to have GNU tar wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -xvz should allow you to do it all in one step.


8

What you do until now can be replaced by one awk call: wget ... | awk -F\" '$6 ~ "gz$" { lastline=thisline; thisline=$6;}; '\ 'END {print lastline; print thisline;}' And, of course, awk can add the URL, too: awk -F\" -v baseurl="http://ftp.gnu.org/gnu/wget/" \ '$6 ~ "gz" { lastline=thisline; thisline=$6;}; '\ 'END {print baseurl lastline; print ...


7

You can use curl to download parts of the image. It all depends on how robust it has to be. A test-case could be first 500 bytes. Seems to work for a lot of png and jpg, then use identifyor the like to check the size. curl -o 500-peek -r0-500 "http://example.net/some-image.png" Edit: Long time since I wrote image parsers, but gave it some thought and ...


6

You can tell wget to include or exclude files and directories based on wildcard patterns. Untested: wget -R \ -X '/*/?*dna,/*/pep' \ -A '*toplevel*' -R '*_rm.toplevel*' \ ftp://ftp.ensembl.org/pub/current_fasta/ For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I ...


6

Try this: wget -r -l1 --no-parent -A ".deb" http://www.shinken-monitoring.org/pub/debian/ -r recursively -l1 to a maximum depth of 1 --no-parent ignore links to a higher directory -A "*.deb" your pattern


6

lftp: $ lftp http://repo.xplico.org/pool/ cd ok, cwd=/pool lftp repo.xplico.org:/pool> ls drwxr-xr-x -- / drwxr-xr-x - 2012-02-13 09:48 main lftp repo.xplico.org:/pool> cd main lftp repo.xplico.org:/pool/main> ls drwxr-xr-x -- .. drwxr-xr-x - 2012-02-13 09:48 x Directory listings only work for websites that do send ...


6

You could specify a list of allowed resp. disallowed filename patterns: Allowed: -A LIST --accept LIST Disallowed: -R LIST --reject LIST LIST is comma-separated list of filename patterns/extensions. You can use the following reserved characters to specify patterns: * ? [ ] Examples: only download PNG files: -A png don't download CSS files: -R ...


5

Dixit man wget: -O file --output-document=file The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.) Use of -O is ...


5

This kind of site tries to make it difficult not to use a graphical web browser, because if you use wget you'll be missing all these ads that pay for the bandwidth. Some sites don't make advanced checks and can be tricked easily: tell wget to pretend that it's really Mozilla and that it's coming from the download site. wget --user-agent='Mozilla/5.0 ...


5

I would suggest using curl to do this instead of wget. It can follow the redirection using the switches -L, -J, and -O. curl -O -J -L http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.1/bitcoin-0.8.1-linux.tar.gz/download switch definitions -O/--remote-name Write output to a local file named like the remote file we get. (Only the ...


5

You can use a for loop: for x in a b c d e f g h ; do wget http://url.com/$x.doc done which can be shortened in bash to for x in {a..h} ; do wget ... done But you can use this shorthand even without a for loop: wget http://url.com/{a..h}.doc If the names are not sequential and you need to read them from a file, use a while loop: while read ...


5

Try: wget -r -np -k -p http://www.site.com/dir/page.html The args (see man wget) are: r Recurse into links, retrieving those pages too (this has a default max depth of 5, can be set with -l). np Never enter a parent directory (i.e., don't follow a "home" link and mirror the whole site; this will prevent going above ccc in your example). k Convert links ...


5

I usually use httrack for downloading/mirroring web content from a site. $ httrack http://2011.example.com -K -w -O . -%v --robots=0 -c1 %e0 After it runs you're left with a directory structure that's local and browseable. For example: $ ls -l total 304 -rw-r--r-- 1 saml saml 4243 Aug 17 10:20 backblue.gif -rw-r--r-- 1 saml saml 828 Aug 17 10:20 ...


5

What you want is a HEAD request, but wget does not support it; curl does. Your distribution most probably has curl in repositories. curl -s -I $URL -I | awk '/Location: (.*)/ {print $2}' | tail -n 1 $ URL=http://unix.stackexchange.com/questions/89282/ $ curl -s -I $URL | awk '/Location: (.*)/ {print $2}' | tail -n 1 ...



Only top voted, non community-wiki answers of a minimum length are eligible