4
votes
3answers
91 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
1
vote
1answer
66 views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
0
votes
2answers
220 views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
40 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
1
vote
1answer
571 views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
2
votes
1answer
357 views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
0
votes
3answers
240 views

Custom name output file wget or curl

For example I have a link http://www.abc.com/123/def/ghi/jkl.mno. I want to download it using wget or curl and get the name of output file as def_ghi_jkl.mno, where the part def_ghi is taken from the ...
1
vote
1answer
160 views

How to figure out where a link gets redirected

I currently working on a script that updates a file. The file has several versions. The important ones have static links. These get redirected to the actual file which then gets downloaded using wget. ...
0
votes
2answers
320 views

how do I get the response ouput from the url .download it in file,not download the code

My file like this with some number or id 404000847307897 404000857827460 404001044194601 404000835511595 404000338418501 404000338654003 404000290144612 404000344175621 404000290144612 ...
0
votes
1answer
168 views

403 http code appears when I go to a certain website

I wrote a PHP script to a particular site using a home computer for debugging. Script worked properly, when I tested it on a home computer. The script has stopped working when I deploy the script on ...
2
votes
1answer
336 views

Why is wget skipping files while trying to mirror a site

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html wget --recursive --no-clobber --page-requisites --html-extension --convert-links ...
26
votes
2answers
20k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
0
votes
1answer
529 views

make install - how to use wget, not curl

I do not have root access on my Ubuntu but want to install git. I was following this tutorial but when running make or make install it says curl is not installed. Can I use wget instead if this is a ...
7
votes
6answers
1k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
1
vote
1answer
630 views

How can I update certificates on my system to fix wget's error: “Unable to establish SSL connection.”?

Sometimes I notice that I am unable to connect via SSL with curl and wget to websites that have certificates that should be valid. One example might be https://squareup.com/ which has a Verisign ...
2
votes
1answer
412 views

Predict filename before downloading from a URL, in shell script

I have a shell script that downloads files from a list using wget, and resumes automatically if there is any non-critical error (because of unstable WIFI during storms). The problem is, I want to ...
31
votes
2answers
6k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them... Why are there two different methods?
12
votes
3answers
6k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
0
votes
1answer
2k views

How to use wget or other tool to download a file, for which the link is hidden(activates on-click only)

I want to to download some subtitle files, stored in rar files form http://subs.sab.bz/ The site provides rss feeds for its new releases. Unfortunately, the link provided will open a download page, ...
1
vote
1answer
216 views

What is wget idiom for curl -o?

So I use code in some function like if [ ! -e $DOWNLOADS/$1 ]; then echo_run curl -L http://$4/$5/$1 -o $DOWNLOADS/$1 fi I wonder how to achieve the same thing without curl?
1
vote
1answer
133 views

Alert when a process exits

I often download files using wget and cURL. Is there anyway I can make my Linux box give a visual alert when either of those processes exit (i.e: the download completes) PS : I am using Ubuntu ...
6
votes
1answer
459 views

How to download all images whose size>100kb in a webpage?

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very ...
4
votes
2answers
10k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
6
votes
2answers
326 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...