Tag Info

Hot answers tagged

35

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything but curl Features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


30

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


19

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


11

There is a << operator (here document) but it is not what is used here. You have first an input redirection < which says pick the input from the following argument then there is the <(command) known as process substitution which says replace that by a file descriptor containing the output of the command. This command could have been written ...


10

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect. Sometimes, ...


10

You can do curl -L http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz > genromfs.tar.gz to download the file. The -L tells curl to follow any redirects, which sourceforge normally does. If wget is available, that would be far simpler.


9

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...


8

On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs. For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...


8

curl support url-encode internally with "--data-urlencode" $ curl -G -v "http://localhost:30001/data" --data-urlencode "msg=hello world" Trace headers > GET /data?hello%20world HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > Host: localhost > Accept: */*


7

Try this: $ LOCATION=`curl -I http://raspberrypi.stackexchange.com/a/1521/86 | perl -n -e '/^Location: (.*)$/ && print "$1\n"'` $ echo "$LOCATION" /questions/1508/how-do-i-access-the-distributions-name-on-the-command-line/1521#1521 Google Redirects Google redirect URLs are slightly different. They return a Javascript redirect, which could easily ...


7

Use empty: With the password safely stored (it's a way of saying...) $ echo password > pwd-file Start process with empty. (You would omit -L log in the real case.) $ empty -f -i fifo1 -o fifo2 -L log curl -u user http://example.com Send the contents of pwd-file to empty's input pipe, which the process sees as both its stdin and /dev/tty. $ empty ...


6

There's a system log facility: syslog. From a shell script, call the logger command. logger -p 'user.info' "hello world" However, this is for system logs, not for logs intended for a particular user. If the logs are intended for the user who's running the application, then a log file is more appropriate (something like ~/.application-name/downloads.log). ...


6

lftp: $ lftp http://repo.xplico.org/pool/ cd ok, cwd=/pool lftp repo.xplico.org:/pool> ls drwxr-xr-x -- / drwxr-xr-x - 2012-02-13 09:48 main lftp repo.xplico.org:/pool> cd main lftp repo.xplico.org:/pool/main> ls drwxr-xr-x -- .. drwxr-xr-x - 2012-02-13 09:48 x Directory listings only work for websites that do send ...


5

Try this construct instead: while read author; do curl "http://www.test.com/authors?string=$author&proc=39" done < authors This will save you the cat, having to do things in backticks, etc. It would also allow you to read several columns out of the input file if you wanted to, but specifying just one variable to fetch like in my example ...


5

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies. ...


5

Dixit man wget: -O file --output-document=file The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.) Use of -O is ...


5

It seems curl does print the progress stats only when stdout is not a terminal. (e.g. curl -I sandbox.dev|cat would give you these results, too) You can disable these, however. From the manpage -s/--silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.


5

What you want is a HEAD request, but wget does not support it; curl does. Your distribution most probably has curl in repositories. curl -s -I $URL -I | awk '/Location: (.*)/ {print $2}' | tail -n 1 $ URL=http://unix.stackexchange.com/questions/89282/ $ curl -s -I $URL | awk '/Location: (.*)/ {print $2}' | tail -n 1 ...


4

As I said in a comment, the netrw plugin, which handles http://, doesn't handle https:// (this from looking at the plugin code). I went into the code and got https:// working (no guarantees it's bug-free, but it worked for the couple sites I tried). I've created a pair of patches, for netrw.vim and netrwPlugin.vim. Here's how to apply them, keeping the ...


4

-s/--silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. So your res should look like res=`curl -s -I $1 | grep HTTP/1.1 | awk {'print $2'}` Result is Error 301 on google.com, for example.


4

You're looking at it the wrong way. The no_proxy environment variable lists the domain suffixes, not the prefixes. From the documentation: no_proxy: This variable should contain a comma-separated list of domain extensions proxy should not be used for. So for IPs, you have two options: 1) Add each IP in full: printf -v no_proxy '%s,' ...


4

You are confusing a few things. "Onclick" actions refer to JavaScript and are client-side. You would have to examine what the JavaScript hook on those links does to unravel the URL. However, there are no onclick actions in play here. What the web site in question does is referrer checking, also known as "hotlink protection". The browser sends a referrer ...


4

one <<(two) runs the command two and pipes its output into one. The more common way of writing it is two | one, so your command can also be written: curl -s https://rvm.beginrescueend.com/install/rvm | bash That link is broken now, but the website says to use this instead: curl -L get.rvm.io | bash -s stable curl outputs what it downloads, so ...


4

You probably have to use something more sophisticated. FTP itself (if I remember correctly) is not able to transfer entire directories, all it knows about is transferring a file. Thus the client has to translate your request "send this directory" into a sequence of upload this file (possibly preceded by make this directory). One such sophisticated client ...


4

I think you're having the same issue as posted here: http://serverfault.com/questions/296602/dns-wildcard-and-etc-resolv-conf-problem Appending a dot to your fqdn queries should work around the issue: curl http://jksodf89s9df9sudfisdf.com. Or, if you don't rely on the wildcard DNS entry, just replace it by specific DNS entries for your subdomains (that's ...


4

Why not rsync instead? You may have to install it (use your package manager), but it's made for this kind of thing. RSYNC_PASSWORD='password' rsync -rtv /path/to/dir/ username@sftp.example.com:remote/dir -r will recursively rsync, -t preserves time stamps so existing files aren't re-uploaded (effectively), and -v is verbose. If you create and copy ...


4

- is commonly used to represent standard input and < is commonly used to represent redirection from a file. I believe those syntaxes come from early shells. Together, they imply taking in standard input and sending/redirecting it elsewhere. The syntax is almost natural. Looking at the cURL revision history, the < syntax was added to cURL in mid-2000. ...


4

If you grab the perl script TTYtter you can make a post from the command line: perl TTYtter -status="Hello World from Linux Shell #bash" Run TTYtter once without any switches to setup the OAuth.


4

What about -sSf? From the man pages: -s/--silent Silent or quiet mode. Do not show progress meter or error messages. Makes Curl mute. -S/--show-error When used with -s it makes curl show an error message if it fails. -f/--fail (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable ...



Only top voted, non community-wiki answers of a minimum length are eligible