cURL is a CLI tool for issuing HTTP GET, POST, & PUT request.

learn more… | top users | synonyms

51
votes
2answers
48k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
40
votes
3answers
10k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
23
votes
4answers
10k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
17
votes
3answers
6k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
16
votes
1answer
1k views

What does the @ (at) symbol before a filename mean in a curl command?

I am looking at an example of a curl command that uses the @ symbol before the filename to post data to the server. curl http://localhost/ --data-binary @file.txt What does this mean? Does it ...
15
votes
3answers
22k views

Does curl have a timeout?

So far I couldn't find anything really, but is it true that curl doesn't really time out at all? user@host:~# curl http://localhost/testdir/image.jpg I'm asking because I'm redirecting any request ...
10
votes
2answers
13k views

Any way to encode the url in curl command?

I have some url which has space in it's query param. I want to use this in curl, e.g. curl -G "http://localhost:30001/data?zip=47401&utc_begin=2013-8-1 00:00:00&utc_end=2013-8-2 ...
9
votes
4answers
5k views

Most straightforward way of getting a raw, unparsed HTTPS response

What would be the most straightforward way of making a GET request to a url over HTTPS, and getting the raw, unparsed response? Could this be achieved with curl? If so, what options would you need to ...
9
votes
3answers
6k views

Get URL's redirect target with curl

I would like to check where a single URL redirects. An example of that could be a link from Google's search result page (where a click always goes through Google server). Can I do that with curl?
8
votes
3answers
2k views

Automating web requests using curl?

I have some "corporative" systems I have to use at work. Some of them are terrible, with frames, pop-ups and such. I was thinking about automating some stuff using curl or something like that. I need ...
7
votes
6answers
2k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
7
votes
1answer
5k views

Passing binary data to curl without using a @file

Is it possible to use curl and post binary data without passing in a file name? For example, you can post a form using binary via --data-binary: curl -X POST --data-binary @myfile.bin http://foo.com ...
7
votes
5answers
1k views

Read and confirm shell script before piping from curl to sh (curl -s [url] | sh)

Whenever I have to execute a shell script from the web curl -s [url] | sh, I first open url in my web browser to make sure the script isn't malicious and is safe to run. I remember seeing a command ...
7
votes
2answers
433 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
6
votes
3answers
4k views

How do I use curl to download content from sourceforge?

Simply typing http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz works fine on a browser, but I'm trying to download from a CLI environment with limited utilities. ...
6
votes
2answers
2k views

Can't understand this script. bash <<(curl -s http://xxx.com)

In a rails book, the author wrote the script to install ruby rvm. bash <<(curl -s https://rvm.beginrescueend.com/install/rvm) But I don't understand how it works. Could you explain this?
6
votes
2answers
1k views

Pipe Password to Application When Asked

The following should be done in a bash script: curl --digest --user schmijos https://bitbucket.org/u/p/get/tip.zip -o tip.zip How can I automatically submit a password to curl when it asks for it? ...
6
votes
3answers
1k views

How to tweet using terminal?

I would like to tweet a message using terminal. I tried something like: curl -u 'TwitterUsername':'TwitterPassword' -d status=”Your Message Here” https://twitter.com/statuses/update.xml but seems ...
6
votes
1answer
525 views

How to download all images whose size>100kb in a webpage?

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very ...
5
votes
4answers
10k views

How do I get the http status of a site in bash?

I figure curl would do the job. I wrote in a bash script: #!/bin/sh function test { res=`curl -I $1 | grep HTTP/1.1 | awk {'print $2'}` if [ $res -ne 200 ] then echo "Error $res ...
5
votes
2answers
6k views

loop curl get request bash

I'm trying to automate a curl script and eventually make it loop also. I found that I can't use any loop like: for i in `cat authors` do curl *line here* ...
5
votes
3answers
818 views

Crontab job not emailing on failure

I have a cron job on Debian: 1 * * * * /home/paradroid/bin/myscript.sh >/dev/null There is an installed and configured MTA, and I have received emails when the script has a syntax error, so I ...
5
votes
2answers
338 views

Make vim / curl recognize https:// links

I recently noticed vim/vimdiff can open links (using curl, it seems), which is an awesome feature. I wanted to use it today, like this: vimdiff deps ...
5
votes
2answers
226 views

cURL <- feature

$ cat file | curl -F 'sprunge=<-' http://sprunge.us So the output of echo gets passed as a POST parameter to cURL. Is this a cURL specific feature?
4
votes
3answers
17k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
4
votes
3answers
243 views

Send first line of the HTTP Header

A normal HTTP request may look something like this: GET / HTTP/1.1 Host: example.com I would like to change the first line of this request, with a custom one, e.g.: FOOBAR NOT REALLY HTTP Host: ...
4
votes
2answers
719 views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
4
votes
1answer
909 views

'watch'-ing curl yields unexpected output

I'm trying to watch curl for live feedback on HTTP tweaks. However, when I execute the command, the output is garbled with some download stats. I've tried sending curl's stderr to /dev/null, to no ...
4
votes
3answers
495 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
2k views

How to check whether a command such as curl completed without error?

I am using curl to upload a file to a server via an HTTP post. curl -X POST -d@myfile.txt server-URL When I manually execute this command on the command line, I get a response from the server like ...
4
votes
3answers
2k views

How to download videos from Coursera with curl?

Downloading videos from Coursera is not as easy as copy and pasting the link in terminal. I also have tried exporting cookies from my browser in cookie.txt file and doing a: curl --cookie cookie.txt ...
3
votes
1answer
342 views

How to grep the output of cURL?

I need to retrieve the expiry date of an SSL cert. The curl application does provide this information: $ curl -v https://google.com/ * Hostname was NOT found in DNS cache * Trying ...
3
votes
3answers
2k views

Curl with ipv6 not working by default

I am trying to curl ipv6 addresses, but without success. If I try lets say: curl https://google.com I wait for a timeout and get network unreachable If I force ipv4, everything is fine. ...
3
votes
2answers
160 views

How can I check GMail messages with prompt for password from a script?

I'm using zsh on Ubuntu 14.04 over SSH using Putty and I'm setting up the key bindings for my keyboard. Because zsh doesn't seem to make use of my function keys I thought I'd setup scripts to do ...
3
votes
2answers
7k views

Using curl to access basic auth protected website via proxy (polipo)

I am trying to download a file protected via basic auth: curl "http://User:Pass@example.com/blub/bla.bin" This is working fine. As soon as I tell curl to use my locally installed polipo it fails: ...
3
votes
1answer
370 views

Given a mime-type get the corresponding file extension

I want to download a file using curl to a temporary directory with the appropriate extension. Currently I do something like this: tmp="$(mktemp -t 'curl')" curl -o "$tmp" ...
3
votes
1answer
305 views

How do I enable rsync in libcurl?

I'm on Arch. When pacman tries to download a package from an ftp server, It fails with the error message Protocol rsync not supported or disabled in libcurl This has been bugging me for a little ...
2
votes
2answers
1k views

twitter status updates from command line (via cURL) [duplicate]

Based on some samples: http://www.commandlinefu.com/commands/view/176/update-twitter-via-curl I'm trying to update my twitter status from command line (via cURL), but always get error. Any idea what ...
2
votes
2answers
736 views

what does Curl's stand-alone hyphen (-) mean?

what's the stand-alone hyphen (between the -C & -O) stand for in this command? curl -C - -O http://www.intersil.com/content/dam/Intersil/documents/fn67/fn6742.pdf FWIW- I'm following a ...
2
votes
3answers
355 views

Source (.) piped curl output

I have a file, f, on my webserver with the following contents: alias ll='ls -l' I would like to (from another linux box) do the following, but I can't get the command right: curl http://myserver/f ...
2
votes
2answers
207 views

Is there a OS/user-wide log to log to?

I'm using curl to dowload a jar from a TESTDRIVE script and for debugging purposes it helps to log the http headers returned to a log file. But logging them in the working dir (where the script is ...
2
votes
1answer
2k views

How can I queue up an entire directory for FTP?

Currently I know how to do this: curl -T file.zip ftp://server.domain.com/ --user username:password Which will let me do a single file. However I can't figure out how to do it for a directory. I ...
2
votes
1answer
3k views

curl --resolve appears to do nothing

The --help output for curl lists a --resolve option, which states --resolve <host:port:address> Force resolve of HOST:PORT to ADDRESS I'm not having any luck getting it to work though. The ...
2
votes
2answers
858 views

Puzzle: transfering files between two servers (ssh, ftp, http multipart…)

A puzzle: files being uploaded in realtime by SCP to SERVER1 need to be transfered to SERVER2. But: SERVER1 doesn't actually have network access to SERVER2 (NAT), however SERVER2 has SSH access to ...
2
votes
1answer
898 views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
2
votes
1answer
538 views

Strange issue with curl/lynx and non resolving domains

I have a weird issue on my server in that if I use curl or lynx to try and access a URL that doesn't exist (say a non registered domain, or a domain that's not in DNS), I don't get an error, it ...
2
votes
2answers
3k views

curl several files and save output in one file

I am using curl to grab the HTML of a website. I want to only grab the first 20 lines, and save this in an external file. I need to grab about 10 URLs, and want to save all of them in the same file. ...
2
votes
2answers
296 views

Using “&&” is not effective for cURL submit form (don't execute second line if first line fails)

I'm submitting to my website using cURL. I make cURL post it in timely fashion. In below code; first line is curl submit; second line is for queuing: curl -b cookies.txt \ -d title="$(sed ...
2
votes
4answers
107 views

Set timeout for web page response with curl

I use a tool to monitor if the web-page is up and running. The tool uses curl command internally to fetch the output. However, when a web-page takes longer time to respond, it results back with a ...
2
votes
1answer
250 views

Understanding two flags and a dollar sign in a CURL command

I right clicked a post request in Chrome and selected Copy as cURL I got a cURL command that includes the following --data-binary $'------WebKitFormBound I am used to seeing cURL requests that ...