Hot answers tagged

679

Yes. From the manpage: -k, --insecure (TLS) By default, every SSL connection curl makes is verified to be secure. This option allows curl to proceed and operate even for server connections otherwise considered insecure. The server connection is verified by making sure the server's certificate contains the right name and verifies ...


378

Yes. Timeout parameters curl has two options: --connect-timeout and --max-time. Quoting from the manpage: --connect-timeout <seconds> Maximum time in seconds that you allow the connection to the server to take. This only limits the connection phase, once curl has connected this option is of no more use. Since 7.32.0, this ...


372

From man curl -s, --silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it. So if you don't want any output use: curl -s 'http://example.com' > /dev/null


311

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl's features are powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


191

curl supports url-encoding internally with --data-urlencode: $ curl -G -v "http://localhost:30001/data" --data-urlencode "msg=hello world" --data-urlencode "msg2=hello world2" -G is also necessary to append the data to the URL. Trace headers > GET /data?msg=hello%20world&msg2=hello%20world2 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-...


144

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


115

curl writes the output to stderr, so redirect that and also suppress the progress: curl -v --silent https://google.com/ 2>&1 | grep expire The reason why curl writes the information to stderr is so you can do: curl <url> | someprgram without that information clobbering the input of someprogram


92

This one looks more elegant to me: curl --silent --output /dev/null http://example.com Also, if you want to see the HTTP code: curl --write-out '%{http_code}' --silent --output /dev/null http://example.com Full documentation is here.


81

When the kernel executes a process, it copies the command line arguments to read-write memory belonging to the process (on the stack, at least on Linux). The process can write to that memory like any other memory. When ps displays the argument, it reads back whatever is stored at that particular address in the process's memory. Most programs keep the ...


75

They were made for different purposes wget is a tool to download files from servers curl is a tool that let's you exchange requests/responses with a server wget Wget solely lets you download files from an HTTP/HTTPS or FTP server. You give it a link and it automatically downloads the file where the link points to. It builds the request automatically. ...


63

There is a redirect on the webserver-side to the following URL: http://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg. Because it's a CDN, the exact behaviour (whether you get redirected or not) might depend on your location. curl does not follow redirects by default. To tell it to do so, add the -L argument: curl -L -O https://dev....


55

Since you didn't specify, I'm assuming you are using wget to download the file. If this is the case, try using it with the -c option (e.g. wget -c <URL>). Please notice that in case the protocol used is ftp (the URL looks like ftp://...) there is a chance the remote server uses an old/ancient ftp daemon which doesn't support resuming downloads (newer ...


50

Do it in jq jq -r '.host_components[].HostRoles.host_name | join(",")' No, that's wrong. This is what you need: jq -r '.host_components | map(.HostRoles.host_name) | join(",")' Demo: jq -r '.host_components | map(.HostRoles.host_name) | join(",")' <<DATA {"host_components":[ {"HostRoles":{"host_name":"one"}}, {"HostRoles":{"host_name":"two"}}, ...


48

How about simply using wget? $ wget http://picasaweb.google.com 2>&1 | grep Location: Location: /home [following] Location: https://www.google.com/accounts/ServiceLogin?hl=en_US&continue=https%3A%2F%2Fpicasaweb.google.com%2Flh%2Flogin%3Fcontinue%3Dhttps%253A%252F%252Fpicasaweb.google.com%252Fhome&service=lh2&ltmpl=gp&passive=true [...


47

You can pass data into curl via STDIN like so: echo -e '...data...\n' | curl -X POST --data-binary @- http://foo.com The @- tells curl to pull in from STDIN. To pipe binary data to curl (for example): echo -e '\x03\xF1' | curl -X POST --data-binary @- http://foo.com


46

You're missing a semicolon after the URL. It should be: for ((i=1;i<=100;i++)); do curl -v --header "Connection: keep-alive" "localhost:8080/user?uuid=52010&model_id=20&attr=0"; done


43

What about -sSf? From the man pages: -s/--silent Silent or quiet mode. Do not show progress meter or error messages. Makes Curl mute. -S/--show-error When used with -s it makes curl show an error message if it fails. -f/--fail (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable ...


43

There is a practical difference. curl -sSL https://get.docker.com/ | sh starts curl and sh at the same time, connecting the output of curl with the input of sh. curl will carry out with the download (roughly) as fast as sh can run the script. The server can detect the irregularities in the timing and inject malicious code not visible when simply downloading ...


42

A HTTP request may contain more headers that are not set by curl or wget. For example: Cookie: this is the most likely reason why a request would be rejected, I have seen this happen on download sites. Given a cookie key=val, you can set it with the -b key=val (or --cookie key=val) option for curl. Referer (sic): when clicking a link on a web page, most ...


42

It is specific to curl. From man curl: --data-binary <data> (HTTP) This posts data exactly as specified with no extra processing whatsoever. If you start the data with the letter @, the rest should be a filename. Data is posted in a similar manner as --data-ascii does, except that newlines are preserved and conversions are never done. If ...


42

netcat is one option. nc -zv kafka02 6667 -z = sets nc to simply scan for listening daemons, without actually sending any data to them -v = enables verbose mode


38

You may use the following command to apply the changes for all connections: $ echo insecure >> ~/.curlrc On Windows just create _curlrc text file with 'insecure' text in it in your %HOME%, %CURL_HOME%, %APPDATA%, %USERPROFILE% or %USERPROFILE%\Application Data directory. Advantage of using above solution is that it works for all curl commands, but ...


35

lftp (Wikipedia) is good for that. It supports a number of protocols, can download files using several concurrent parallel connections (useful where there's a lot of packet loss not caused by congestion), and can automatically resume downloads. It's also scriptable. Here including the fine-tuning you came up with (credits to you): lftp -c 'set net:idle 10 ...


29

If using Bash Shell, then you can use its feature to check if a port is open or closed: (timeout 1 bash -c '</dev/tcp/127.0.0.1/17500 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null PORT OPEN (timeout 1 bash -c '</dev/tcp/127.0.0.1/7500 && echo PORT OPEN || echo PORT CLOSED') 2>/dev/null PORT CLOSED Note that if the server ...


28

You can using the -H/--header argument: You could spoof your ip address: curl --header "X-Forwarded-For: 192.168.0.2" http://example.com Example: client $ curl http://webhost.co.uk web host $ tailf access.log | grep 192.168.0.54 192.168.0.54 - - [10/Nov/2014:15:56:09 +0000] "GET / HTTP/1.1" 200 14328 "-" "curl/7.19.7 (x86_64-redhat-linux-...


26

You can use -m option: -m, --max-time <seconds> Maximum time in seconds that you allow the whole operation to take. This is useful for preventing your batch jobs from hang‐ ing for hours due to slow networks or links going down. See also the --connect-timeout option. If ...


26

When debugging curl issues, the -v option is often helpful. In this particular instance, you’re running afoul of the If-None-Match header, which tells the server that you already have the file matching “6931c3b4d0e94743bb93a36ed8e8c3f5add12f9a” and that you’re not interested in retrieving it again if it hasn’t changed. -v shows you this by indicating that ...


25

I realize that there is already an accepted answer to this question but I feel I should point out that there actually is a way to do this. The --remote-name-all option tells curl to behave as if you used -O or --remote-name for each file. https://curl.haxx.se/docs/manpage.html#--remote-name-all This option has been available since version 7.19.0 https://...


24

The gold standard is undoubtedly nmap (nmap.org), but it typically requires root for “best results”. However, standalone binaries are available and it is possible to run it as an unprivileged user, just with degraded capabilities. For example, instead of a stealth syn scan (-sS), it falls back to a standard TCP connect scan (-sT). This is functionally ...


23

You might be able to use curl's --fail option, though you should test it once first. man curl -f, --fail (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to ...


Only top voted, non community-wiki answers of a minimum length are eligible