Hot answers tagged

164

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered "...


70

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


63

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


54

wget -P /var/cache/foobar/ [...] wget --directory-prefix=/var/cache/foobar/ [...]


39

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in not-actually-informative messages. The practical reason is that in unix's very early days, there still ...


29

You can do it by telling wget to output its payload to stdout: wget -qO- your_link_here | tar xvz To specify a target directory: wget -qO- your_link_here | tar xvz -C /target/directory Update If you happen to have GNU tar wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -xvz should allow you to do it all in one step.


26

A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


26

Usually, we see that when we have stopped a download and the continued/resumed with it again. That way, we are downloading only portion which has not been downloaded already. This happens when you use the -c switch. For example $ wget https://help.ubuntu.com/10.04/serverguide/serverguide.pdf 53% [=======================> ] 531,834 ...


25

(Strange situation, doesn't something like the triangle inequality hold for internet routing?) Anyway, try the following, on A, ssh into B with a -D argument, ssh -D 1080 address-of-B which acts as a SOCKS5 proxy on 127.0.0.1:1080, which can be used by anything supporting SOCKS5 proxied connections. Apparently, wget can do this, by using the environment ...


25

I would suggest using curl to do this instead of wget. It can follow the redirection using the switches -L, -J, and -O. curl -O -J -L http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.1/bitcoin-0.8.1-linux.tar.gz/download switch definitions -O/--remote-name Write output to a local file named like the remote file we get. (Only the ...


24

If you know the name of the file ahead of time, you can use the -O option to wget to tell it where to write the file. wget -O /var/cache/foobar/stackexchange-site-list.txt http://url.to/stackexchange-site-list.txt


21

So you are logged into a machine myclient and have ssh access to another machine myserver. You want to download a file over HTTP from a remove server www.example.com to myclient but the data needs to be saved on myserver. This should do it: wget -O - http://www.example.com/whatever | ssh myserver 'cat >/path/to/remote/destination' Alternatively, you ...


19

2 refers to the second file descriptor of the process, i.e. stderr. > means redirection. &1 means the target of the redirection should be the same location as the first file descriptor, i.e. stdout. So this command first redirects stdout to /dev/null and then redirects stderr there as well. This effectively silences all output (regular or error) ...


19

I think these switches will do what you want with wget: -A acclist --accept acclist -R rejlist --reject rejlist Specify comma-separated lists of file name suffixes or patterns to accept or reject. Note that if any of the wildcard characters, *, ?, [ or ], appear in an element of acclist or rejlist, it will be treated as a ...


17

Wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots= off, wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/


17

I was able to download a public shared file like this: $ wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' \ -O FILENAME Where FILEID must be replaced by the actual file ID. FILENAME is the path/filename where download will be stored. Note you cannot use a folderid instead of fileid. I have used view source in a ...


17

Since you didn't specify, I'm assuming you are using wget to download the file. If this is the case, try using it with the -c option (e.g. wget -c <URL>). Please notice that in case the protocol used is ftp (the URL looks like ftp://...) there is a chance the remote server uses an old/ancient ftp daemon which doesn't support resuming downloads (newer ...


15

In the unix world, each tool is designed to do one job and do it well. Why would cp worry about outputting progress when another tool like pv does it already? In the same vein, why do so many programs dump stuff to the screen without any pagination? Because there are already tools for that job such as more (or less). Why do most programs that require editing ...


15

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document. You could, for example, use lynx -dump -nolist, if you have lynx around. lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, ...


14

By default, wget writes to a file whose name is the last component of the URL that you pass to it. Many servers redirect URLs like http://www.url1.com/app?q=123&gibb=erish&gar=ble to a different URL with a nice-looking file name like http://download.url1.com/files/something.pdf. You can tell wget to use the name from the redirected URL (i.e. ...


14

wget follows redirects automatically. Just give the URL as an argument wget 'http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz'


14

If the server is blocking wget, it is most likely doing it on the basis of the "User-agent:" field of the http header, since that is the only way for it to know in the first place. It could also be blocking your IP, in which case using different software won't help, or some scheme which identifies automation on the basis of how rapid a set of requests are (...


14

I think your ? gets interpreted by shell (Correction by vinc17: more likely, it's the & which gets interpreted). Just try with simple quotes around your URL: wget 'http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file' Note that the file you are requesting is a .tar file but the above command will save it as index.html?acc=GSE48191&...


14

Do you really want to tar the file or are you looking for downloading a file into a compressed form. Tarring a file is just bundling (uncompressed) files into an (uncompressed) archive. If you want to download a file into a compressed file you can use: wget -qO - <url>|gzip -c - > file.gz


14

Even though not a direct answer, git blame and git log reveal that this section was introduced in commit 2c41d783 by a committer called hniksic, who turns out to be Hrvoje Niksic. His eMail-Adress can be found in wget's ChangeLog file (I won't publish it here for the obvious reasons). I'd suggest asking him directly, as he might be the best to give a more ...


13

Try this: wget -r -l1 --no-parent -A ".deb" http://www.shinken-monitoring.org/pub/debian/ -r recursively -l1 to a maximum depth of 1 --no-parent ignore links to a higher directory -A "*.deb" your pattern


13

You could specify a list of allowed resp. disallowed filename patterns: Allowed: -A LIST --accept LIST Disallowed: -R LIST --reject LIST LIST is comma-separated list of filename patterns/extensions. You can use the following reserved characters to specify patterns: * ? [ ] Examples: only download PNG files: -A png don't download CSS files: -R ...


13

As you note, you don't need the whole ImageMagick package. You just need identify. You will also need the libraries the executable links to (and the libraries those libraries link to). > whereis identify identify: /bin/identify /usr/bin/identify /usr/share/man/man1/identify.1.gz > ldd /bin/identify ldd will show a list. When I did this, it ...


12

You may use the following command to apply the changes for all connections: $ echo insecure >> ~/.curlrc On Windows just create .curlrc text file with 'insecure' text in it in your HOME dir.


11

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page: --max-filesize Specify the maximum size (in bytes) of a file to download. If the file ...



Only top voted, non community-wiki answers of a minimum length are eligible