Questions tagged [curl]

cURL is a CLI tool for issuing HTTP GET, POST, & PUT request.

Filter by
Sorted by
Tagged with
12
votes
3answers
14k views

Automating web requests using curl?

I have some "corporative" systems I have to use at work. Some of them are terrible, with frames, pop-ups and such. I was thinking about automating some stuff using curl or something like that. I need ...
8
votes
2answers
1k views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
56
votes
5answers
128k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
67
votes
3answers
5k views

How does curl protect a password from appearing in ps output?

I noticed some time ago that usernames and passwords given to curl as command line arguments don't appear in ps output (although of course they may appear in your bash history). They likewise don't ...
32
votes
8answers
150k views

Health check of web page using curl

I'd like to do a health check of a service by calling a specific url on it. Feels like the simplest solution would be to use cron to do the check every minute or so. In case of errors, cron sends me ...
29
votes
2answers
49k views

Passing binary data to curl without using a @file

Is it possible to use curl and post binary data without passing in a file name? For example, you can post a form using binary via --data-binary: curl -X POST --data-binary @myfile.bin http://foo.com ...
9
votes
7answers
8k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
239
votes
5answers
374k views

Does curl have a timeout?

So far I couldn't find anything really, but is it true that curl doesn't really time out at all? user@host:~# curl http://localhost/testdir/image.jpg I'm asking because I'm redirecting any request ...
2
votes
1answer
4k views

using CURL to transmit binary data over POST parameter

How can i use CURL to send binary data through a specific post parameter? E.g in: curl 'http://www.example.com' -H 'hi:hi' --data 'utf8=%E2%9C%93&_method=put&file=binarydatahere&submit=...
11
votes
5answers
3k views

Read and confirm shell script before piping from curl to sh (curl -s [url] | sh)

Whenever I have to execute a shell script from the web curl -s [url] | sh, I first open url in my web browser to make sure the script isn't malicious and is safe to run. I remember seeing a command ...
8
votes
6answers
4k views

How to tweet using terminal?

I would like to tweet a message using terminal. I tried something like: curl -u 'TwitterUsername':'TwitterPassword' -d status=”Your Message Here” https://twitter.com/statuses/update.xml but seems ...
9
votes
3answers
3k views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
9k views

trying to install curl with homebrew, getting “incompatible library version” for libcurl.4.dylib

I am on OSX 10.9.5. I would like to use a later version of CURL than what is bundled with OSX, but I do not want to remove the OSX binary just in case, so I wanted to opt to install it via homebrew. ...
2
votes
1answer
434 views

Alternatives to cURL/wget for remote API calls

I'm on IBM AIX 7.1 where cURL and wget are not available. Also, I'm not in a position to install either. Are there alternatives that would be available that would enable remote API calls using GET/...
1
vote
1answer
3k views

How to delete the oldest file over FTP using CURL?

I am trying to write a script that will use CURL to FTP to a folder on a file server, then count the files in that folder. If there are more than 5 files, delete the oldest one. I know how to delete ...
1
vote
1answer
5k views

How to curl and unzip to a certain directory? [duplicate]

How to curl and unzip to a certain directory? you curl a file to stdout then unzip to some dir.
1
vote
1answer
480 views

Using cURL command-line tool on Mac, how does one fetch stock data which comes back *without* commas? [duplicate]

The below script is not working. cd desktop/quoteUpdate while true do curl-o quotes.txt -s "http://download.finance.yahoo.com/d/quotes.csv?s=goog,aapl&f=sl1" sed -i '.bak' 's/,/ /g' quotes.txt ...
0
votes
1answer
2k views

Using the cURL command, how does one fetch stock data which comes back *without* commas and spaces?

What can I add to the below script so that it fetches the stock data and brings it back WITHOUT any commas and spaces? For example GOOG Outstanding shares is 675,000,000. I desire the output of ...
245
votes
3answers
127k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
52
votes
4answers
174k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
91
votes
3answers
65k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
27
votes
3answers
21k views

How do I use curl to download content from sourceforge?

Simply typing http://downloads.sourceforge.net/project/romfs/genromfs/0.5.2/genromfs-0.5.2.tar.gz works fine on a browser, but I'm trying to download from a CLI environment with limited utilities. ...
15
votes
1answer
18k views

cUrl : Argument list too long

I'm getting the error: Argument list too long when trying to use cUrl to send a file in base64 inside the body of my JSON. I'm using something like this: DATA=$( base64 "$FILE" ) curl -X POST -H "...
11
votes
3answers
5k views

What is most secure and simplest way to have a user-typed password on bash become part of stdin to a program?

I'm looking for the (1) most secure and (2) simplest way to have a user type a password on a bash shell prompt and to have that password become part of stdin to a program. This is what the stdin ...
3
votes
1answer
1k views

Command-line cURL submit form from other files

In this command: curl -d param1=value1 -d param2=value2 http://example.com/submit If value1 is located in file1.txt, how can I make cURL include file1.txt in the command?
7
votes
2answers
42k views

cURL doesn't connect to HTTPS while wget does (NSS error -12286)

I'm getting error NSS error -12286 while downloading a file from HTTPS using curl. I can download the same file without issues using wget so I can exclude any firewall or blacklist issues. Already ...
9
votes
1answer
4k views

How to pipe html into lynx?

I am testing some ajax call from the console. I like get rid of the meta-information noise that is html markup and just read the content, hence I like to pipe it into a terminal browser, e.g. lynx. ...
2
votes
1answer
2k views

Strange issue with curl/lynx and non resolving domains

I have a weird issue on my server in that if I use curl or lynx to try and access a URL that doesn't exist (say a non registered domain, or a domain that's not in DNS), I don't get an error, it ...
2
votes
1answer
2k views

How to download .pdf files from pubmed

I can't download .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles, I already have some document IDs such as PMC3404089. I tried to issue the following command: curl -o file.pdf http://www....
6
votes
2answers
5k views

Pipe Password to Application When Asked

The following should be done in a bash script: curl --digest --user schmijos https://bitbucket.org/u/p/get/tip.zip -o tip.zip How can I automatically submit a password to curl when it asks for it? ...
6
votes
5answers
4k views

OSX curl error with unprivileged user dyld: Library not loaded

My MAC OS is Yosemite 10.10, which has an old version of curl. I have installed Xcode and Xcode command line tools. I run cURL with the "sudo" command as “sudo curl”: sudo curl Password: ...
3
votes
3answers
10k views

echo text after curl command on same line

#!/bin/bash echo "$(curl -s -I https://google.com|grep Server)" abc returns: abcer: gws and bash -x shows: ++ grep Server ++ curl -s -I https://google.com ' abco 'Server: gws abcer: gws but ...
2
votes
1answer
14k views

wget and cURL an explicit IPv6 Address

You can use the -6 flag in cURL or wget to use the IPv6 address of a domain, like google.com. I was unable to fetch pages by explicitly passing an IPv6 address. I tried: wget -6 http://[fe80::a00:...
2
votes
1answer
5k views

compiling curl with openssl giving wrong openssl version

I have found various answers on this, but I can't get anything that works for me. I am trying to compile curl with nghttp2 on my CentOS server. I have compiled it all ok, but my problem is that curl -...
2
votes
1answer
344 views

Alert when a process exits

I often download files using wget and cURL. Is there anyway I can make my Linux box give a visual alert when either of those processes exit (i.e: the download completes) PS : I am using Ubuntu ...
2
votes
2answers
474 views

Using “&&” is not effective for cURL submit form (don't execute second line if first line fails)

I'm submitting to my website using cURL. I make cURL post it in timely fashion. In below code; first line is curl submit; second line is for queuing: curl -b cookies.txt \ -d title="$(sed '1,/...
1
vote
1answer
682 views

How do I create a local copy of a complete website section from OSX using curl?

I'm trying to download Mathematics of the Discrete Fourier Transform so that I can read through it off-line on my iPad (which doesn't have an Internet connection) I have used wget in Linux. It seems ...
1
vote
1answer
633 views

How to send a file to an FTP server that doesn't properly terminate the connection on success?

(moved from SO) My goal is to upload a file to an FTP server in an embedded device. The file is a configuration file. Once the file is accepted, the server reboots. What makes this a challenge is ...
1
vote
2answers
66 views

Execute 2 or more remote scripts sharing the same curl pattern, without redundancy

I use Ubuntu 16.04 and I execute a list of remote scripts that are in the same directory (a GitHub repository): curl -s https://raw.githubusercontent.com/${user}/${repo}/master/1.sh | tr -d '\r' | ...
1
vote
1answer
602 views

Pipe multiple files from curl to other program

I would like to download several images (from a server with a self-signed certificate) and view them all with feh. Is this possible with a single command? Something like curl -sk -b ...
0
votes
1answer
799 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url --no-check-...
0
votes
1answer
543 views

How to pass header from stdin or local file into remote curl?

The following curl command works as expected: $ curl -H @- -vso/dev/null http://www.example.com <<<"Foo:Bar" * Rebuilt URL to: http://www.example.com/ ... > Accept: */* > Foo:Bar > ...
0
votes
2answers
3k views

Curl and grep html text

Is there a way to grep html text from a curl request and output grab the value that is between two html tags? More specifically the input value of a form?
0
votes
1answer
907 views

Curl fails to download a file during start up

I am adding startup script in Ubuntu with update-rc.d. My script says : #!/bin/bash nohup curl -C - -o ~/Downloads/linuxmint-17-xfce-dvd-64bit.iso.part 'http://mirror.umd.edu/linuxmint/images/...
0
votes
1answer
3k views

Does `curl -v` show the complete HTTP request including the body?

I thought user=user&password=password will be in the body of the request, but I can't find it. Where is it? Does -v show the complete request including the body? Thanks. $ curl --data "user=user&...
0
votes
1answer
25 views

How to import pgp keys from a search on an keyserver using curl?

Is there a standard method for searching and importing a pgp key using curl or wget? I was going through verifying the download with a gpg signed shasum256 file and encountered the problem of trying ...
-8
votes
1answer
252 views

Does HTTP `TRACE` method do the same thing as `curl --trace`?

Does HTTP TRACE method do the same thing as curl --trace? Stephen said at Does `curl -v` show the complete HTTP request including the body?: the TRACE HTTP method affects the server’s behaviour, ...