Tag Info

Hot answers tagged


Yes. From the manpage: -k, --insecure (SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered ...


The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl features and is powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


Yes. curl has two options: --connect-timeout and --max-time. Quoting from the manpage: --connect-timeout <seconds> Maximum time in seconds that you allow the connection to the server to take. This only limits the connection phase, once curl has connected this option is of no more use. Since 7.32.0, this option accepts ...


Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


curl supports url-encoding internally with --data-urlencode: $ curl -G -v "http://localhost:30001/data" --data-urlencode "msg=hello world" -G is also necessary to append the data to the URL. Trace headers > GET /data?msg=hello%20world HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > Host: localhost > Accept: */*


curl writes the output to stderr, so redirect that and also suppress the progress: curl -v --silent https://google.com/ 2>&1 | grep expire The reason why curl writes the information to stderr is so you can do: curl <url> | someprgram without that information clobbering the input of someprogram


A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


It is specific to curl. From man curl: --data-binary <data> (HTTP) This posts data exactly as specified with no extra processing whatsoever. If you start the data with the letter @, the rest should be a filename. Data is posted in a similar manner as --data-ascii does, except that newlines are preserved and conversions are never done. If ...


If you want to use curl, this should work: curl -D - https://www.google.com/ Note, however, that this is not exactly the raw response. For instance chunked transfer encoding will not be visible in the response. Using --raw solves this, also verbose mode (-v) is useful, too and -i shows the headers before the response body: curl -iv --raw ...


You can do curl -L http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz > genromfs.tar.gz to download the file. The -L tells curl to follow any redirects, which sourceforge normally does. If wget is available, that would be far simpler.


You can pass data into curl via STDIN like so: echo -e '...data...\n' | curl -X POST --data-binary @- http://foo.com The @- tells curl to pull in from STDIN. To pipe binary data to curl (for example): echo -e '\x03\xF1' | curl -X POST --data-binary @- http://foo.com


For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect. Sometimes, ...


There is a << operator (here document) but it is not what is used here. You have first an input redirection < which says pick the input from the following argument then there is the <(command) known as process substitution which says replace that by a file descriptor containing the output of the command. This command could have been written ...


Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...


It appears that the address needs to be a numeric IP address, not a hostname. Try this: curl --resolve foo.example.com:443: https://foo.example.com:443/


Since you didn't specify, I'm assuming you are using wget to download the file. If this is the case, try using it with the -c option (e.g. wget -c <URL>). Please notice that in case the protocol used is ftp (the URL looks like ftp://...) there is a chance the remote server uses an old/ancient ftp daemon which doesn't support resuming downloads (newer ...


Use: curl http://mysite.com/myfile.jpg > myfile.jpg


You're looking at it the wrong way. The no_proxy environment variable lists the domain suffixes, not the prefixes. From the documentation: no_proxy: This variable should contain a comma-separated list of domain extensions proxy should not be used for. So for IPs, you have two options: 1) Add each IP in full: printf -v no_proxy '%s,' ...


Here's a simple way that comes to mind echo 'GET / HTTP/1.1 Host: google.com ' | openssl s_client -quiet -connect google.com:443 2>/dev/null


It seems curl does print the progress stats only when stdout is not a terminal. (e.g. curl -I sandbox.dev|cat would give you these results, too) You can disable these, however. From the manpage -s/--silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.


On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs. For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...


You're missing a semicolon after the URL. You have: for ((i=1;i<=100;i++)); do curl -v --header "Connection: keep-alive" "localhost:8080/user?uuid=52010&model_id=20&attr=0" done It should be: for ((i=1;i<=100;i++)); do curl -v --header "Connection: keep-alive" "localhost:8080/user?uuid=52010&model_id=20&attr=0"; done


From man curl -s, --silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it. So if you don't want any output use: curl -s 'http://example.com' > /dev/null


Wrap the URL in single-quotes. What is happening is that the ampersand is getting interpreted by the shell. curl ''


-s/--silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. So your res should look like res=`curl -s -I $1 | grep HTTP/1.1 | awk {'print $2'}` Result is Error 301 on google.com, for example.


you want ... res=$( curl -w %{http_code} -s --output /dev/null $1) ... That will give you the HTTP_STATUS code as the only output.


Try this: $ LOCATION=`curl -I http://raspberrypi.stackexchange.com/a/1521/86 | perl -n -e '/^Location: (.*)$/ && print "$1\n"'` $ echo "$LOCATION" /questions/1508/how-do-i-access-the-distributions-name-on-the-command-line/1521#1521 Google Redirects Google redirect URLs are slightly different. They return a Javascript redirect, which could easily ...


Use empty: With the password safely stored (it's a way of saying...) $ echo password > pwd-file Start process with empty. (You would omit -L log in the real case.) $ empty -f -i fifo1 -o fifo2 -L log curl -u user http://example.com Send the contents of pwd-file to empty's input pipe, which the process sees as both its stdin and /dev/tty. $ empty ...


You can do this by using tool netcat(also called as nc). Just from command line, do the following steps to send the modified request. $ nc <ip-address-of-webserver> 80 FOOBAR NOT REALLY HTTP Host: example.com


The most common way to verify the integrity of downloaded files is to use MD5 checksums. This assumes that the site you are downloading from actually published MD5 checksums of their files. You can verify a MD5 checksum by creating your own checksum of the downloaded file and comparing it to the published checksum. If they are identical the file you have ...

Only top voted, non community-wiki answers of a minimum length are eligible