Tag Info

Hot answers tagged

101

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


59

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


48

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


35

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in not-actually-informative messages. The practical reason is that in unix's very early days, there still ...


26

Usually, we see that when we have stopped a download and the continued/resumed with it again. That way, we are downloading only portion which has not been downloaded already. This happens when you use the -c switch. For example $ wget https://help.ubuntu.com/10.04/serverguide/serverguide.pdf 53% [=======================> ] 531,834 ...


25

wget -P /var/cache/foobar/ [...] wget --directory-prefix=/var/cache/foobar/ [...]


24

A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


19

So you are logged into a machine myclient and have ssh access to another machine myserver. You want to download a file over HTTP from a remove server www.example.com to myclient but the data needs to be saved on myserver. This should do it: wget -O - http://www.example.com/whatever | ssh myserver 'cat >/path/to/remote/destination' Alternatively, you ...


19

(Strange situation, doesn't something like the triangle inequality hold for internet routing?) Anyway, try the following, on A, ssh into B with a -D argument, ssh -D 1080 address-of-B which acts as a SOCKS5 proxy on 127.0.0.1:1080, which can be used by anything supporting SOCKS5 proxied connections. Apparently, wget can do this, by using the environment ...


15

In the unix world, each tool is designed to do one job and do it well. Why would cp worry about outputting progress when another tool like pv does it already? In the same vein, why do so many programs dump stuff to the screen without any pagination? Because there are already tools for that job such as more (or less). Why do most programs that require editing ...


15

You can do it by telling wget to output its payload to stdout: wget -qO- your_link_here | tar xvz To specify a target directory: wget -qO- your_link_here | tar xvz -C /target/directory Update If you happen to have GNU tar wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -xvz should allow you to do it all in one step.


14

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document. You could, for example, use lynx -dump -nolist, if you have lynx around. lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, ...


14

If the server is blocking wget, it is most likely doing it on the basis of the "User-agent:" field of the http header, since that is the only way for it to know in the first place. It could also be blocking your IP, in which case using different software won't help, or some scheme which identifies automation on the basis of how rapid a set of requests are ...


14

Do you really want to tar the file or are you looking for downloading a file into a compressed form. Tarring a file is just bundling (uncompressed) files into an (uncompressed) archive. If you want to download a file into a compressed file you can use: wget -qO - <url>|gzip -c - > file.gz


13

As you note, you don't need the whole ImageMagick package. You just need identify. You will also need the libraries the executable links to (and the libraries those libraries link to). > whereis identify identify: /bin/identify /usr/bin/identify /usr/share/man/man1/identify.1.gz > ldd /bin/identify ldd will show a list. When I did this, it ...


13

2 refers to the second file descriptor of the process, i.e. stderr. > means redirection. &1 means the target of the redirection should be the same location as the first file descriptor, i.e. stdout. So this command first redirects stdout to /dev/null and then redirects stderr there as well. This effectively silences all output (regular or error) ...


12

If you know the name of the file ahead of time, you can use the -O option to wget to tell it where to write the file, but you can't just specify a directory. wget -O /var/cache/foobar/stackexchange-site-list.txt http://url.to/stackexchange-site-list.txt


12

Wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots= off, wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/


11

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page: --max-filesize Specify the maximum size (in bytes) of a file to download. If the file ...


11

Use a different separator that doesn't contain any of the characters in the variable. For example, sed "s|^|$URL|" (If you use / as the separator and the pattern or replacement also contain /, then you'd need to escape those.)


11

I think these switches will do what you want with wget: -A acclist --accept acclist -R rejlist --reject rejlist Specify comma-separated lists of file name suffixes or patterns to accept or reject. Note that if any of the wildcard characters, *, ?, [ or ], appear in an element of acclist or rejlist, it will be treated as a ...


11

I think your ? gets interpreted by shell (Correction by vinc17: more likely, it's the & which gets interpreted). Just try with simple quotes around your URL: wget 'http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file' Note that the file you are requesting is a .tar file but the above command will save it as ...


10

You could specify a list of allowed resp. disallowed filename patterns: Allowed: -A LIST --accept LIST Disallowed: -R LIST --reject LIST LIST is comma-separated list of filename patterns/extensions. You can use the following reserved characters to specify patterns: * ? [ ] Examples: only download PNG files: -A png don't download CSS files: -R ...


10

I was able to download a public shared file like this: $ wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' \ -O FILENAME Where FILEID must be replaced by the actual file ID. FILENAME is the path/filename where download will be stored. Note you cannot use a folderid instead of fileid. I have used view source in a ...


9

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So wget ...


9

Try this: wget -r -l1 --no-parent -A ".deb" http://www.shinken-monitoring.org/pub/debian/ -r recursively -l1 to a maximum depth of 1 --no-parent ignore links to a higher directory -A "*.deb" your pattern


9

I would suggest using curl to do this instead of wget. It can follow the redirection using the switches -L, -J, and -O. curl -O -J -L http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.1/bitcoin-0.8.1-linux.tar.gz/download switch definitions -O/--remote-name Write output to a local file named like the remote file we get. (Only the ...


9

You can use curl to download parts of the image. It all depends on how robust it has to be. A test-case could be first 500 bytes. Seems to work for a lot of png and jpg, then use identifyor the like to check the size. curl -o 500-peek -r0-500 "http://example.net/some-image.png" Edit: Long time since I wrote image parsers, but gave it some thought and ...


8

Use the --base option: wget -i file-of-filenames --base http://example.com/fetch/from/here/


8

Actually, this might be possible: wget -O - http://example.com/file | tee file | md5sum > file.md5 However, this assumes you have the hash in a file beforehand.



Only top voted, non community-wiki answers of a minimum length are eligible