wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

30
votes
2answers
6k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them... Why are there two different methods?
26
votes
2answers
20k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
22
votes
2answers
6k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
12
votes
3answers
6k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
10
votes
5answers
12k views

How to download a file through an SSH server?

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C, The issue is, it is very slow to download a file direct from A, so I need download the ...
9
votes
1answer
6k views

How to download specific files from some url path with wget

If I don't want to have to download the files found in a specific url path manually, what options do I have? Using wildcards fail: $ wget 'http://www.shinken-monitoring.org/pub/debian/*deb' Warning: ...
8
votes
1answer
11k views

How to wget a file to a remote machine over SSH?

I'd like to basically pipe a wget command to a file on a remote server over SSH. How can I do this? I know I could simply ssh into the server and have it download the file, but I'd much rather use the ...
7
votes
6answers
1k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
7
votes
2answers
2k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
7
votes
2answers
5k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
6
votes
4answers
375 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
6
votes
3answers
3k views

Can one limit the max file size to download using wget?

I want to use wget (executed from a php script) to download image files, but don't want to download files over a certain size. Can I limit file size with wget? If not, what is a better way to do ...
6
votes
3answers
98 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
6
votes
1answer
457 views

How to download all images whose size>100kb in a webpage?

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very ...
6
votes
2answers
323 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
5
votes
3answers
1k views

How to wget Youtube horrible URL

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date: wget ...
5
votes
2answers
2k views

Download recursively with wget

I have a problem with the following wget command: wget -nd -r -l 10 http://web.archive.org/web/20110726051510/http://feedparser.org/docs/ It should download recursively all of the linked documents ...
5
votes
1answer
4k views

How to get text of a page using wget without html?

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c ...
5
votes
1answer
421 views

Download tool needed — with custom headers, resume, retry, custom filename output, and dynamic redirection

Simple problem -- I need a download manager, console, GUI, does not matter. The catch is in the features: custom header -- I have to pass a cookie in order to download (authorization) resume -- so ...
5
votes
2answers
4k views

“Malformed patch” error while applying the diff to wget source with patch

I would like to apply this patch so I copied code beginning at "Index: src/options.h" and ending with "+@item" and put it into a new file created in wget's source code folder. Then I did: $ patch -p0 ...
4
votes
2answers
365 views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
4
votes
2answers
3k views

wget recursive for only files that pass a regexp

I would like to use wget recursively on an ftp directory like this: ftp://ftp.ensembl.org/pub/current_fasta/ Which contains another 2 directory levels inside, the first with the species name, and ...
4
votes
3answers
3k views

wget - How to download recursively and only specific mime-types/extensions (i.e. text only)

How to download a full website, but ignoring all binary files. wget has this functionality using the -r flag but it downloads everything and some websites are just too much for a low-resources ...
4
votes
2answers
10k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
4
votes
3answers
101 views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
4
votes
2answers
993 views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
4
votes
3answers
89 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
317 views

Download from dynamically changing urls via script

I want to write a script to download all Walter Lewin lectures from this page in wmv format: http://videolectures.net/mit801f99_physics_classical_mechanics/ However the links to the wmv file for ...
4
votes
1answer
4k views

How to use wget to download from hosting sites?

wget is a very useful tool to download stuff on the internet quickly, but can I use it to download from hosting sites, like FreakShare, IFile.it Depositfiles, Uploaded, Rapidshare? If so, how can I do ...
4
votes
2answers
2k views

How to copy someone's else folders from public.me.com with a wget-like tool?

How can I copy a folder from http://public.me.com/ (a service related to iDisk, or MobileMe) to my local filesystem with a Unix tool (like wget, a command-line non-interactive tool)? The problem is ...
4
votes
2answers
627 views

Does -q definitely turn off wget output logging?

I currently have a shell script running on a linux server which is using wget in oder to download a remote web page. This in turn is executed by a cron job which is scheduled to run at certain times. ...
4
votes
1answer
629 views

Why wget does not work via ssh tunnel? What does proxy prevent ssh-client to do?

To test things I'm trying to establish a local ssh tunnel from my laptop to a site via ssh-server, and download a page of the site (or view it in a browser). The tunnel is made like this: $ ssh -L ...
4
votes
1answer
2k views

Obtaining md5 during wget

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so ...
4
votes
1answer
3k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...
4
votes
3answers
215 views

How to tell if a file is being downloaded by wget?

I use wget to download files (most are zip files) automatically for me during the night. However, sometimes in the morning I find that a few files cannot be unzipped. I don't know why this is ...
4
votes
1answer
676 views

wget --spider: how to tell where broken links are coming from

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes ...
4
votes
2answers
967 views

mounting dropbox (for exploring the files)

I want to simply be able to explore a dropbox filesystem, without much additional load on CPU, memory, storage (i.e., without saving copies of the remote files locally). It seems that the official ...
3
votes
1answer
151 views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
3
votes
2answers
2k views

Download list of files if they don't already exist

If I have a list of URLs separated by \n, are there any options I can pass to wget to download all the URLs and save them to the current directory, but only if the files don't already exist?
3
votes
3answers
760 views

Using wget to get file names from a text file

I have found a website that hosts a few files that I'm after, there's too many to download them all individually. The filenames take a fairly standard and reproduceable form i.e. 1_a, 1_b, 1_c etc Is ...
3
votes
1answer
306 views

How to tell wget where to fetch files passed via '--input-file'

I have a list of filenames (not urls) in a file, and I want to download them using --input-file option. How do I specify the url where those files will be fetched from?
3
votes
1answer
567 views

How to download all html files from a URL's directory with wget?

I'd like to use wget to download a tutorial in a website, but this tutorial is in a specific directory. Suppose the URL is the tutorial main page http://www.abc.com/aaa/bbb/ccc/index.html. But the ...
3
votes
2answers
178 views

How do I prevent wget from loading Apache directory listings in different orders?

I’m trying to recursively download a directory using this command: wget -r -l2 --no-parent -A "*.pdf" -nd -N -e robots=off http://… The script gets executed every 15 minutes, so I want to reduce ...
3
votes
2answers
4k views

“Short write” error message with wget

I got the following error when trying to pull down a 300 MB ISO. It appears to pull down 25 MB then stop. Any idea what's going on here? ~ # wget ...
3
votes
2answers
807 views

Convert links of downloaded website

I downloaded a website using: wget -c --mirror -p http://www.somewebsite.com for offline viewing and I just remembered that I forgot the --convert-links option! They are all on my hard drive right ...
3
votes
1answer
22 views

How to use wget --wait option when using input from a file?

I have a text file with a list of urls I want to download. I want wget to wait before downloading the next url. I'm using cat urls.txt | xargs -n1 -i wget --wait=30 {} However there is no wait ...
3
votes
2answers
65 views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
3
votes
1answer
70 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
3
votes
1answer
317 views

Extract directory from wget's stdout

I am trying to wget a tarball from github.com and, without creating a temporary file, extract a subdirectory from it: wget -qO- https://github.com/django-nonrel/django-nonrel/tarball/develop | tar ...
3
votes
2answers
915 views

Difference between wget versions

I have problem with wget behavior for 1.10 and later versions. I am using wget to download a report from my app. version 1.10.2: >wget-1.10.2.exe --http-user=trader --http-passwd=trader ...