Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [curl]

cURL is a CLI tool for issuing HTTP GET, POST, & PUT request.

12
votes
3answers
13k views

Automating web requests using curl?

I have some "corporative" systems I have to use at work. Some of them are terrible, with frames, pop-ups and such. I was thinking about automating some stuff using curl or something like that. I need ...
8
votes
2answers
1k views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
52
votes
5answers
120k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
67
votes
3answers
5k views

How does curl protect a password from appearing in ps output?

I noticed some time ago that usernames and passwords given to curl as command line arguments don't appear in ps output (although of course they may appear in your bash history). They likewise don't ...
28
votes
8answers
139k views

Health check of web page using curl

I'd like to do a health check of a service by calling a specific url on it. Feels like the simplest solution would be to use cron to do the check every minute or so. In case of errors, cron sends me ...
29
votes
2answers
46k views

Passing binary data to curl without using a @file

Is it possible to use curl and post binary data without passing in a file name? For example, you can post a form using binary via --data-binary: curl -X POST --data-binary @myfile.bin http://foo.com ...
9
votes
7answers
8k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
229
votes
5answers
352k views

Does curl have a timeout?

So far I couldn't find anything really, but is it true that curl doesn't really time out at all? user@host:~# curl http://localhost/testdir/image.jpg I'm asking because I'm redirecting any request ...
2
votes
1answer
3k views

using CURL to transmit binary data over POST parameter

How can i use CURL to send binary data through a specific post parameter? E.g in: curl 'http://www.example.com' -H 'hi:hi' --data 'utf8=%E2%9C%93&_method=put&file=binarydatahere&submit=...
11
votes
5answers
3k views

Read and confirm shell script before piping from curl to sh (curl -s [url] | sh)

Whenever I have to execute a shell script from the web curl -s [url] | sh, I first open url in my web browser to make sure the script isn't malicious and is safe to run. I remember seeing a command ...
8
votes
6answers
4k views

How to tweet using terminal?

I would like to tweet a message using terminal. I tried something like: curl -u 'TwitterUsername':'TwitterPassword' -d status=”Your Message Here” https://twitter.com/statuses/update.xml but seems ...
8
votes
3answers
3k views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
9k views

trying to install curl with homebrew, getting “incompatible library version” for libcurl.4.dylib

I am on OSX 10.9.5. I would like to use a later version of CURL than what is bundled with OSX, but I do not want to remove the OSX binary just in case, so I wanted to opt to install it via homebrew. ...
2
votes
1answer
301 views

Alternatives to cURL/wget for remote API calls

I'm on IBM AIX 7.1 where cURL and wget are not available. Also, I'm not in a position to install either. Are there alternatives that would be available that would enable remote API calls using GET/...
1
vote
1answer
479 views

Using cURL command-line tool on Mac, how does one fetch stock data which comes back *without* commas? [duplicate]

The below script is not working. cd desktop/quoteUpdate while true do curl-o quotes.txt -s "http://download.finance.yahoo.com/d/quotes.csv?s=goog,aapl&f=sl1" sed -i '.bak' 's/,/ /g' quotes.txt ...
1
vote
1answer
3k views

How to delete the oldest file over FTP using CURL?

I am trying to write a script that will use CURL to FTP to a folder on a file server, then count the files in that folder. If there are more than 5 files, delete the oldest one. I know how to delete ...
1
vote
1answer
4k views

How to curl and unzip to a certain directory? [duplicate]

How to curl and unzip to a certain directory? you curl a file to stdout then unzip to some dir.
0
votes
1answer
2k views

Using the cURL command, how does one fetch stock data which comes back *without* commas and spaces?

What can I add to the below script so that it fetches the stock data and brings it back WITHOUT any commas and spaces? For example GOOG Outstanding shares is 675,000,000. I desire the output of ...
230
votes
3answers
118k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
47
votes
4answers
163k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
83
votes
3answers
62k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
23
votes
3answers
20k views

How do I use curl to download content from sourceforge?

Simply typing http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz works fine on a browser, but I'm trying to download from a CLI environment with limited utilities. ...
15
votes
1answer
17k views

cUrl : Argument list too long

I'm getting the error: Argument list too long when trying to use cUrl to send a file in base64 inside the body of my JSON. I'm using something like this: DATA=$( base64 "$FILE" ) curl -X POST -H "...
11
votes
3answers
4k views

What is most secure and simplest way to have a user-typed password on bash become part of stdin to a program?

I'm looking for the (1) most secure and (2) simplest way to have a user type a password on a bash shell prompt and to have that password become part of stdin to a program. This is what the stdin ...
6
votes
2answers
38k views

cURL doesn't connect to HTTPS while wget does (NSS error -12286)

I'm getting error NSS error -12286 while downloading a file from HTTPS using curl. I can download the same file without issues using wget so I can exclude any firewall or blacklist issues. Already ...
2
votes
1answer
1k views

Command-line cURL submit form from other files

In this command: curl -d param1=value1 -d param2=value2 http://example.com/submit If value1 is located in file1.txt, how can I make cURL include file1.txt in the command?
9
votes
1answer
3k views

How to pipe html into lynx?

I am testing some ajax call from the console. I like get rid of the meta-information noise that is html markup and just read the content, hence I like to pipe it into a terminal browser, e.g. lynx. ...
2
votes
1answer
2k views

How to download .pdf files from pubmed

I can't download .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles, I already have some document IDs such as PMC3404089. I tried to issue the following command: curl -o file.pdf http://www....
2
votes
1answer
2k views

Strange issue with curl/lynx and non resolving domains

I have a weird issue on my server in that if I use curl or lynx to try and access a URL that doesn't exist (say a non registered domain, or a domain that's not in DNS), I don't get an error, it ...
6
votes
2answers
4k views

Pipe Password to Application When Asked

The following should be done in a bash script: curl --digest --user schmijos https://bitbucket.org/u/p/get/tip.zip -o tip.zip How can I automatically submit a password to curl when it asks for it? ...
5
votes
5answers
4k views

OSX curl error with unprivileged user dyld: Library not loaded

My MAC OS is Yosemite 10.10, which has an old version of curl. I have installed Xcode and Xcode command line tools. I run cURL with the "sudo" command as “sudo curl”: sudo curl Password: ...
2
votes
1answer
328 views

Alert when a process exits

I often download files using wget and cURL. Is there anyway I can make my Linux box give a visual alert when either of those processes exit (i.e: the download completes) PS : I am using Ubuntu ...
2
votes
2answers
453 views

Using “&&” is not effective for cURL submit form (don't execute second line if first line fails)

I'm submitting to my website using cURL. I make cURL post it in timely fashion. In below code; first line is curl submit; second line is for queuing: curl -b cookies.txt \ -d title="$(sed '1,/...
2
votes
3answers
9k views

echo text after curl command on same line

#!/bin/bash echo "$(curl -s -I https://google.com|grep Server)" abc returns: abcer: gws and bash -x shows: ++ grep Server ++ curl -s -I https://google.com ' abco 'Server: gws abcer: gws but ...
2
votes
1answer
5k views

compiling curl with openssl giving wrong openssl version

I have found various answers on this, but I can't get anything that works for me. I am trying to compile curl with nghttp2 on my CentOS server. I have compiled it all ok, but my problem is that curl -...
1
vote
1answer
593 views

How to send a file to an FTP server that doesn't properly terminate the connection on success?

(moved from SO) My goal is to upload a file to an FTP server in an embedded device. The file is a configuration file. Once the file is accepted, the server reboots. What makes this a challenge is ...
1
vote
2answers
65 views

Execute 2 or more remote scripts sharing the same curl pattern, without redundancy

I use Ubuntu 16.04 and I execute a list of remote scripts that are in the same directory (a GitHub repository): curl -s https://raw.githubusercontent.com/${user}/${repo}/master/1.sh | tr -d '\r' | ...
1
vote
1answer
571 views

Pipe multiple files from curl to other program

I would like to download several images (from a server with a self-signed certificate) and view them all with feh. Is this possible with a single command? Something like curl -sk -b ...
1
vote
1answer
679 views

How do I create a local copy of a complete website section from OSX using curl?

I'm trying to download Mathematics of the Discrete Fourier Transform so that I can read through it off-line on my iPad (which doesn't have an Internet connection) I have used wget in Linux. It seems ...
0
votes
2answers
2k views

Curl and grep html text

Is there a way to grep html text from a curl request and output grab the value that is between two html tags? More specifically the input value of a form?
0
votes
1answer
747 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url --no-check-...
0
votes
1answer
840 views

Curl fails to download a file during start up

I am adding startup script in Ubuntu with update-rc.d. My script says : #!/bin/bash nohup curl -C - -o ~/Downloads/linuxmint-17-xfce-dvd-64bit.iso.part 'http://mirror.umd.edu/linuxmint/images/...
0
votes
1answer
442 views

How to pass header from stdin or local file into remote curl?

The following curl command works as expected: $ curl -H @- -vso/dev/null http://www.example.com <<<"Foo:Bar" * Rebuilt URL to: http://www.example.com/ ... > Accept: */* > Foo:Bar > ...
-1
votes
1answer
381 views

Does `curl -v` show the complete HTTP request including the body?

I thought user=user&password=password will be in the body of the request, but I can't find it. Where is it? Does -v show the complete request including the body? Thanks. $ curl --data "user=user&...
-8
votes
1answer
84 views

Does HTTP `TRACE` method do the same thing as `curl --trace`?

Does HTTP TRACE method do the same thing as curl --trace? Stephen said at Does `curl -v` show the complete HTTP request including the body?: the TRACE HTTP method affects the server’s behaviour, ...