Tag Info

Hot answers tagged

51

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


44

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


29

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


28

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in not-actually-informative messages. The practical reason is that in unix's very early days, there still ...


25

Usually, we see that when we have stopped a download and the continued/resumed with it again. That way, we are downloading only portion which has not been downloaded already. This happens when you use the -c switch. For example $ wget https://help.ubuntu.com/10.04/serverguide/serverguide.pdf 53% [=======================> ] 531,834 ...


19

So you are logged into a machine myclient and have ssh access to another machine myserver. You want to download a file over HTTP from a remove server www.example.com to myclient but the data needs to be saved on myserver. This should do it: wget -O - http://www.example.com/whatever | ssh myserver 'cat >/path/to/remote/destination' Alternatively, you ...


18

A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


17

(Strange situation, doesn't something like the triangle inequality hold for internet routing?) Anyway, try the following, on A, ssh into B with a -D argument, ssh -D 1080 address-of-B which acts as a SOCKS5 proxy on 127.0.0.1:1080, which can be used by anything supporting SOCKS5 proxied connections. Apparently, wget can do this, by using the environment ...


14

In the unix world, each tool is designed to do one job and do it well. Why would cp worry about outputting progress when another tool like pv does it already? In the same vein, why do so many programs dump stuff to the screen without any pagination? Because there are already tools for that job such as more (or less). Why do most programs that require editing ...


13

If the server is blocking wget, it is most likely doing it on the basis of the "User-agent:" field of the http header, since that is the only way for it to know in the first place. It could also be blocking your IP, in which case using different software won't help, or some scheme which identifies automation on the basis of how rapid a set of requests are ...


13

As you note, you don't need the whole ImageMagick package. You just need identify. You will also need the libraries the executable links to (and the libraries those libraries link to). > whereis identify identify: /bin/identify /usr/bin/identify /usr/share/man/man1/identify.1.gz > ldd /bin/identify ldd will show a list. When I did this, it ...


12

wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document. You could, for example, use lynx -dump -nolist, if you have lynx around. lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, ...


11

If you are scripting downloads, you should consider using curl instead. Wget can parse output and recursively fetch whole sites, but curl has way more options relating to the actual download of a specific file. Here is the relevant option in the man page: --max-filesize Specify the maximum size (in bytes) of a file to download. If the file ...


11

Wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots= off, wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/


11

Use a different separator that doesn't contain any of the characters in the variable. For example, sed "s|^|$URL|" (If you use / as the separator and the pattern or replacement also contain /, then you'd need to escape those.)


10

You can do it by telling wget to output its payload to stdout: wget -qO- your_link_here | tar xvz To specify a target directory: wget -qO- your_link_here | tar xvz -C /target/directory Update If you happen to have GNU tar wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -xvz should allow you to do it all in one step.


10

2 refers to the second file descriptor of the process, i.e. stderr. > means redirection. &1 means the target of the redirection should be the same location as the first file descriptor, i.e. stdout. So this command first redirects stdout to /dev/null and then redirects stderr there as well. This effectively silences all output (regular or error) ...


9

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So wget ...


9

I think your ? gets interpreted by shell (Correction by vinc17: more likely, it's the & which gets interpreted). Just try with simple quotes around your URL: wget 'http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file' Note that the file you are requesting is a .tar file but the above command will save it as ...


8

Try this: wget -r -l1 --no-parent -A ".deb" http://www.shinken-monitoring.org/pub/debian/ -r recursively -l1 to a maximum depth of 1 --no-parent ignore links to a higher directory -A "*.deb" your pattern


8

On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs. For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...


8

You can use curl to download parts of the image. It all depends on how robust it has to be. A test-case could be first 500 bytes. Seems to work for a lot of png and jpg, then use identifyor the like to check the size. curl -o 500-peek -r0-500 "http://example.net/some-image.png" Edit: Long time since I wrote image parsers, but gave it some thought and ...


8

What you do until now can be replaced by one awk call: wget ... | awk -F\" '$6 ~ "gz$" { lastline=thisline; thisline=$6;}; '\ 'END {print lastline; print thisline;}' And, of course, awk can add the URL, too: awk -F\" -v baseurl="http://ftp.gnu.org/gnu/wget/" \ '$6 ~ "gz" { lastline=thisline; thisline=$6;}; '\ 'END {print baseurl lastline; print ...


8

You can use wget's --base option here: wget -qO- http://ftp.gnu.org/gnu/wget/ | cut -d\" -sf6 | grep '\.tar\.gz' | tail -n2 | wget -i - --base=http://ftp.gnu.org/gnu/wget/


7

The network scanning app nmap would be a much more apt tool to scan a batch of IPs to figure out if the host is up and if a particular port is open.


7

Use the --base option: wget -i file-of-filenames --base http://example.com/fetch/from/here/


7

wget -P /var/cache/foobar/ [...] wget --directory-prefix=/var/cache/foobar/ [...]


7

The wget redirection problem can be solved by using wget --trust-server-names http://www.example.com/X?1234


7

You could specify a list of allowed resp. disallowed filename patterns: Allowed: -A LIST --accept LIST Disallowed: -R LIST --reject LIST LIST is comma-separated list of filename patterns/extensions. You can use the following reserved characters to specify patterns: * ? [ ] Examples: only download PNG files: -A png don't download CSS files: -R ...


6

You can tell wget to include or exclude files and directories based on wildcard patterns. Untested: wget -R \ -X '/*/?*dna,/*/pep' \ -A '*toplevel*' -R '*_rm.toplevel*' \ ftp://ftp.ensembl.org/pub/current_fasta/ For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I ...



Only top voted, non community-wiki answers of a minimum length are eligible