Tag Info

Hot answers tagged

51

Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


44

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


29

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


23

It is specific to curl. From man curl: --data-binary <data> (HTTP) This posts data exactly as specified with no extra processing whatsoever. If you start the data with the letter @, the rest should be a filename. Data is posted in a similar manner as --data-ascii does, except that newlines are preserved and conversions are never done. If ...


18

A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


14

Yes. curl has two options, --connect-timeout and --max-time. Quoting from the manpage: --connect-timeout <seconds> Maximum time in seconds that you allow the connection to the server to take. This only limits the connection phase, once curl has connected this option is of no more use. Since 7.32.0, this option accepts ...


13

You can do curl -L http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz > genromfs.tar.gz to download the file. The -L tells curl to follow any redirects, which sourceforge normally does. If wget is available, that would be far simpler.


11

There is a << operator (here document) but it is not what is used here. You have first an input redirection < which says pick the input from the following argument then there is the <(command) known as process substitution which says replace that by a file descriptor containing the output of the command. This command could have been written ...


11

curl supports url-encoding internally with --data-urlencode: $ curl -G -v "http://localhost:30001/data" --data-urlencode "msg=hello world" Trace headers > GET /data?hello%20world HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > Host: localhost > Accept: */*


10

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect. Sometimes, ...


9

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...


8

Use: curl http://mysite.com/myfile.jpg > myfile.jpg


8

Try this: $ LOCATION=`curl -I http://raspberrypi.stackexchange.com/a/1521/86 | perl -n -e '/^Location: (.*)$/ && print "$1\n"'` $ echo "$LOCATION" /questions/1508/how-do-i-access-the-distributions-name-on-the-command-line/1521#1521 Google Redirects Google redirect URLs are slightly different. They return a Javascript redirect, which could easily ...


8

On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs. For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...


7

Here's a simple way that comes to mind echo 'GET / HTTP/1.1 Host: google.com ' | openssl s_client -quiet -connect google.com:443 2>/dev/null


7

Use empty: With the password safely stored (it's a way of saying...) $ echo password > pwd-file Start process with empty. (You would omit -L log in the real case.) $ empty -f -i fifo1 -o fifo2 -L log curl -u user http://example.com Send the contents of pwd-file to empty's input pipe, which the process sees as both its stdin and /dev/tty. $ empty ...


7

You can do this by using tool netcat(also called as nc). Just from command line, do the following steps to send the modified request. $ nc <ip-address-of-webserver> 80 FOOBAR NOT REALLY HTTP Host: example.com


7

You can pass data into curl via STDIN like so: echo -e '...data...\n' | curl -X POST --data-binary @- http://foo.com The @- tells curl to pull in from STDIN. To pipe binary data to curl (for example): echo -e '\x03\xF1' | curl -X POST --data-binary @- http://foo.com


6

There's a system log facility: syslog. From a shell script, call the logger command. logger -p 'user.info' "hello world" However, this is for system logs, not for logs intended for a particular user. If the logs are intended for the user who's running the application, then a log file is more appropriate (something like ~/.application-name/downloads.log). ...


6

you want ... res=$( curl -w %{http_code} -s --output /dev/null $1) ... That will give you the HTTP_STATUS code as the only output.


6

If you want to use curl, this should work: curl -D - https://www.google.com/ Note, however, that this is not exactly the raw response. For instance chunked transfer encoding will not be visible in the response. Using --raw solves this, also verbose mode (-v) is useful, too and -i shows the headers before the response body: curl -iv --raw ...


6

This error message is because Twitter needs OAuth for authentication. Take a look at TTYtter if you don't mind a Perl app: Noooo, not another Twitter client! Yes, another Twitter client. The difference here is that you're dealing with a multi-functional, fully 100% text, Perl command line client. In interactive mode, it is a fully interactive ...


6

Actually, the major difference is that curl includes a library (libcurl), and that library is widely used by other applications. wget is standalone.


6

It seems curl does print the progress stats only when stdout is not a terminal. (e.g. curl -I sandbox.dev|cat would give you these results, too) You can disable these, however. From the manpage -s/--silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.


6

lftp: $ lftp http://repo.xplico.org/pool/ cd ok, cwd=/pool lftp repo.xplico.org:/pool> ls drwxr-xr-x -- / drwxr-xr-x - 2012-02-13 09:48 main lftp repo.xplico.org:/pool> cd main lftp repo.xplico.org:/pool/main> ls drwxr-xr-x -- .. drwxr-xr-x - 2012-02-13 09:48 x Directory listings only work for websites that do send ...


5

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies. ...


5

Try this construct instead: while read author; do curl "http://www.test.com/authors?string=$author&proc=39" done < authors This will save you the cat, having to do things in backticks, etc. It would also allow you to read several columns out of the input file if you wanted to, but specifying just one variable to fetch like in my example ...


5

You're looking at it the wrong way. The no_proxy environment variable lists the domain suffixes, not the prefixes. From the documentation: no_proxy: This variable should contain a comma-separated list of domain extensions proxy should not be used for. So for IPs, you have two options: 1) Add each IP in full: printf -v no_proxy '%s,' ...


5

I'm assuming your cron email settings are all correct, and you otherwise get emails. Your sending all stdout to /dev/null, so anything that prints error messages must output them to stderr. You might want to make sure everything in the script is outputting correctly. At times I have had to work with third party code, which was sending everything to ...


5

Dixit man wget: -O file --output-document=file The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.) Use of -O is ...



Only top voted, non community-wiki answers of a minimum length are eligible