Hot answers tagged curl
25
The main differences are:
wget's major strong side compared to curl is its ability to download recursively.
wget is command line only. There's no lib or anything but curl Features and is powered by libcurl.
curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...
11
There is a << operator (here document) but it is not what is used here.
You have first an input redirection < which says pick the input from the following argument then there is the <(command) known as process substitution which says replace that by a file descriptor containing the output of the command.
This command could have been written
...
8
Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page.
curl
--limit-rate <speed>
Specify the maximum transfer rate you want curl to use.
This feature is useful if you have a limited pipe and
you'd like your transfer not to use your entire bandwidth.
...
7
For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect.
Sometimes, ...
7
Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...
6
There's a system log facility: syslog. From a shell script, call the logger command.
logger -p 'user.info' "hello world"
However, this is for system logs, not for logs intended for a particular user. If the logs are intended for the user who's running the application, then a log file is more appropriate (something like ~/.application-name/downloads.log). ...
6
You can do
curl -L http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz > genromfs.tar.gz
to download the file.
The -L tells curl to follow any redirects, which sourceforge normally does.
If wget is available, that would be far simpler.
6
Try this:
$ LOCATION=`curl -I http://raspberrypi.stackexchange.com/a/1521/86 | perl -n -e '/^Location: (.*)$/ && print "$1\n"'`
$ echo "$LOCATION"
/questions/1508/how-do-i-access-the-distributions-name-on-the-command-line/1521#1521
Google Redirects
Google redirect URLs are slightly different. They return a Javascript redirect, which could easily ...
6
On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs.
For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...
5
lftp:
$ lftp http://repo.xplico.org/pool/
cd ok, cwd=/pool
lftp repo.xplico.org:/pool> ls
drwxr-xr-x -- /
drwxr-xr-x - 2012-02-13 09:48 main
lftp repo.xplico.org:/pool> cd main
lftp repo.xplico.org:/pool/main> ls
drwxr-xr-x -- ..
drwxr-xr-x - 2012-02-13 09:48 x
Directory listings only work for websites that do send ...
5
Use empty:
With the password safely stored (it's a way of saying...)
$ echo password > pwd-file
Start process with empty. (You would omit -L log in the real case.)
$ empty -f -i fifo1 -o fifo2 -L log curl -u user http://example.com
Send the contents of pwd-file to empty's input pipe, which the process sees as both its stdin and /dev/tty.
$ empty ...
5
Yes. From the manpage:
-k, --insecure
(SSL) This option explicitly allows curl to perform "insecure"
SSL connections and transfers. All SSL connections are attempted
to be made secure by using the CA certificate bundle installed
by default. This makes all connections considered ...
4
Dixit man wget:
-O file
--output-document=file
The documents will not be written to the appropriate files, but all will be concatenated together
and written to file. If - is used as file, documents will be printed to standard output,
disabling link conversion. (Use ./- to print to a file literally named -.)
Use of -O is ...
4
As I said in a comment, the netrw plugin, which handles http://, doesn't handle https:// (this from looking at the plugin code). I went into the code and got https:// working (no guarantees it's bug-free, but it worked for the couple sites I tried). I've created a pair of patches, for netrw.vim and netrwPlugin.vim. Here's how to apply them, keeping the ...
4
You are confusing a few things. "Onclick" actions refer to JavaScript and are client-side. You would have to examine what the JavaScript hook on those links does to unravel the URL. However, there are no onclick actions in play here.
What the web site in question does is referrer checking, also known as "hotlink protection". The browser sends a referrer ...
4
one <<(two) runs the command two and pipes its output into one. The more common way of writing it is two | one, so your command can also be written:
curl -s https://rvm.beginrescueend.com/install/rvm | bash
That link is broken now, but the website says to use this instead:
curl -L get.rvm.io | bash -s stable
curl outputs what it downloads, so ...
4
You probably have to use something more sophisticated. FTP itself (if I remember correctly) is not able to transfer entire directories, all it knows about is transferring a file. Thus the client has to translate your request "send this directory" into a sequence of upload this file (possibly preceded by make this directory).
One such sophisticated client ...
4
I think you're having the same issue as posted here:
http://serverfault.com/questions/296602/dns-wildcard-and-etc-resolv-conf-problem
Appending a dot to your fqdn queries should work around the issue:
curl http://jksodf89s9df9sudfisdf.com.
Or, if you don't rely on the wildcard DNS entry, just replace it by specific DNS entries for your subdomains (that's ...
3
curl has no idea that it failed - it had well-defined input, it communicated with the server and got back a 200 OK response. Because of this, you can't rely on the exit code of curl (which is what && does).
Instead, we have to use something else to determine success or failure, such as the output generated by curl.
As a side note, it is generally ...
3
You could run rsync over ssh. Set up ssh keys so you can do it non-interactively (i.e. avoid having to use a password). Use cron to schedule it.
Check out this article. http://www.askapache.com/security/mirror-using-rsync-ssh.html
3
My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies.
...
3
Try this construct instead:
while read author; do
curl "http://www.test.com/authors?string=$author&proc=39"
done < authors
This will save you the cat, having to do things in backticks, etc. It would also allow you to read several columns out of the input file if you wanted to, but specifying just one variable to fetch like in my example ...
3
-C - automatically determines how to resume the transfer, based upon the input and output files.
From man curl (note the second paragraph):
-C/--continue-at <offset>
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be
skipped, counting from ...
3
It will just resume the download from the last position, from man curl:
-C/--continue-at <offset>
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be
skipped, counting from the beginning of the source file before
it is transferred ...
3
It seems curl does print the progress stats only when stdout is not a terminal. (e.g. curl -I sandbox.dev|cat would give you these results, too) You can disable these, however.
From the manpage
-s/--silent
Silent or quiet mode. Don't show progress meter or
error messages. Makes Curl mute.
3
No, but you could build Git without the Curl dependency on libcurl. It will disable features. Remember that wget is just a binary, whereas Curl provides a shared library as well and that is used by Git. Three options here:
./configure Git with the option --without-curl. Docs say:
--with-curl support http(s):// transports (default is YES)
...
2
The curl project site has RPM files for Redhat Enterprise Linux 5, which will work just fine on your CentOS5.6 box:
http://curl.haxx.se/download.html
You will want to get both the curl and libcurl packages, along (possibly) with libcurl-dev.
Note that you will run into some dependency errors, i.e., a number of things may depend on libcurl.so.3.
2
You can read content out of html downloaded via curl, using a mixture of the regular UNIX commands (so grep, awk, etc.) Exactly how depends on exactly what the form looks like and how you want to interpret it.
You can also do it with perl and other languages which have libraries which make it easier.
However, input forms only have content in the fields ...
2
The short answer is - no you will need to try the FTP connection via libcurl and see if the authentication succeeds.
The username/password only exist on the remote server, and you don't know if they are being changed or altered at any stage (for legitimate reasons). Hence, your code will have to take credentials from the user, and basically try an FTP ...
Only top voted, non community-wiki answers of a minimum length are eligible
