wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

33
votes
2answers
29k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
33
votes
2answers
8k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them... Why are there two different methods?
25
votes
2answers
7k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
16
votes
3answers
764 views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
16
votes
3answers
7k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
12
votes
5answers
15k views

How to download a file through an SSH server?

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C, The issue is, it is very slow to download a file direct from A, so I need download the ...
9
votes
1answer
12k views

How to wget a file to a remote machine over SSH?

I'd like to basically pipe a wget command to a file on a remote server over SSH. How can I do this? I know I could simply ssh into the server and have it download the file, but I'd much rather use the ...
9
votes
2answers
7k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
9
votes
1answer
7k views

How to download specific files from some url path with wget

If I don't want to have to download the files found in a specific url path manually, what options do I have? Using wildcards fail: $ wget 'http://www.shinken-monitoring.org/pub/debian/*deb' Warning: ...
7
votes
6answers
1k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
7
votes
2answers
3k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
7
votes
2answers
365 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
6
votes
4answers
495 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
6
votes
3answers
4k views

Can one limit the max file size to download using wget?

I want to use wget (executed from a php script) to download image files, but don't want to download files over a certain size. Can I limit file size with wget? If not, what is a better way to do ...
6
votes
2answers
3k views

Download recursively with wget

I have a problem with the following wget command: wget -nd -r -l 10 http://web.archive.org/web/20110726051510/http://feedparser.org/docs/ It should download recursively all of the linked documents ...
6
votes
1answer
5k views

How to get text of a page using wget without html?

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c ...
6
votes
3answers
328 views

How you stop 'wget' after it gets a 404?

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease: $ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' It fetches the first 10 ...
6
votes
3answers
103 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
6
votes
2answers
279 views

downloading files using wget

I am trying to download an http link from here. The URL is http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file When I use wget ...
6
votes
1answer
476 views

How to download all images whose size>100kb in a webpage?

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very ...
5
votes
3answers
1k views

How to wget Youtube horrible URL

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date: wget ...
5
votes
3answers
4k views

wget - How to download recursively and only specific mime-types/extensions (i.e. text only)

How to download a full website, but ignoring all binary files. wget has this functionality using the -r flag but it downloads everything and some websites are just too much for a low-resources ...
5
votes
1answer
468 views

Download tool needed — with custom headers, resume, retry, custom filename output, and dynamic redirection

Simple problem -- I need a download manager, console, GUI, does not matter. The catch is in the features: custom header -- I have to pass a cookie in order to download (authorization) resume -- so ...
5
votes
2answers
4k views

“Malformed patch” error while applying the diff to wget source with patch

I would like to apply this patch so I copied code beginning at "Index: src/options.h" and ending with "+@item" and put it into a new file created in wget's source code folder. Then I did: $ patch -p0 ...
5
votes
1answer
4k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...
4
votes
2answers
675 views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
4
votes
2answers
2k views

Download list of files if they don't already exist

If I have a list of URLs separated by \n, are there any options I can pass to wget to download all the URLs and save them to the current directory, but only if the files don't already exist?
4
votes
2answers
3k views

wget recursive for only files that pass a regexp

I would like to use wget recursively on an ftp directory like this: ftp://ftp.ensembl.org/pub/current_fasta/ Which contains another 2 directory levels inside, the first with the species name, and ...
4
votes
2answers
186 views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
4
votes
2answers
13k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
4
votes
3answers
255 views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
4
votes
2answers
2k views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
4
votes
3answers
205 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
365 views

Download from dynamically changing urls via script

I want to write a script to download all Walter Lewin lectures from this page in wmv format: http://videolectures.net/mit801f99_physics_classical_mechanics/ However the links to the wmv file for ...
4
votes
1answer
5k views

How to use wget to download from hosting sites?

wget is a very useful tool to download stuff on the internet quickly, but can I use it to download from hosting sites, like FreakShare, IFile.it Depositfiles, Uploaded, Rapidshare? If so, how can I do ...
4
votes
2answers
2k views

How to copy someone's else folders from public.me.com with a wget-like tool?

How can I copy a folder from http://public.me.com/ (a service related to iDisk, or MobileMe) to my local filesystem with a Unix tool (like wget, a command-line non-interactive tool)? The problem is ...
4
votes
2answers
727 views

Does -q definitely turn off wget output logging?

I currently have a shell script running on a linux server which is using wget in oder to download a remote web page. This in turn is executed by a cron job which is scheduled to run at certain times. ...
4
votes
1answer
742 views

Why wget does not work via ssh tunnel? What does proxy prevent ssh-client to do?

To test things I'm trying to establish a local ssh tunnel from my laptop to a site via ssh-server, and download a page of the site (or view it in a browser). The tunnel is made like this: $ ssh -L ...
4
votes
1answer
3k views

Obtaining md5 during wget

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so ...
4
votes
1answer
197 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
4
votes
3answers
1k views

mounting dropbox (for exploring the files)

I want to simply be able to explore a dropbox filesystem, without much additional load on CPU, memory, storage (i.e., without saving copies of the remote files locally). It seems that the official ...
4
votes
3answers
272 views

How to tell if a file is being downloaded by wget?

I use wget to download files (most are zip files) automatically for me during the night. However, sometimes in the morning I find that a few files cannot be unzipped. I don't know why this is ...
4
votes
1answer
803 views

wget --spider: how to tell where broken links are coming from

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes ...
3
votes
1answer
309 views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
3
votes
3answers
1k views

Using wget to get file names from a text file

I have found a website that hosts a few files that I'm after, there's too many to download them all individually. The filenames take a fairly standard and reproduceable form i.e. 1_a, 1_b, 1_c etc Is ...
3
votes
1answer
323 views

How to tell wget where to fetch files passed via '--input-file'

I have a list of filenames (not urls) in a file, and I want to download them using --input-file option. How do I specify the url where those files will be fetched from?
3
votes
3answers
90 views

Using wget over ssh with a terminated session

I have a RaspberryPi. I want to download a big file with wget but the problem is that once I log into the Pi and use wget to download the file, it will abort the download when I close the session. ...
3
votes
2answers
5k views

“Short write” error message with wget

I got the following error when trying to pull down a 300 MB ISO. It appears to pull down 25 MB then stop. Any idea what's going on here? ~ # wget ...
3
votes
1answer
684 views

How to download all html files from a URL's directory with wget?

I'd like to use wget to download a tutorial in a website, but this tutorial is in a specific directory. Suppose the URL is the tutorial main page http://www.abc.com/aaa/bbb/ccc/index.html. But the ...
3
votes
2answers
216 views

How do I prevent wget from loading Apache directory listings in different orders?

I’m trying to recursively download a directory using this command: wget -r -l2 --no-parent -A "*.pdf" -nd -N -e robots=off http://… The script gets executed every 15 minutes, so I want to reduce ...