Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

402
votes
3answers
553k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
230
votes
3answers
117k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
83
votes
3answers
62k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
82
votes
3answers
149k views

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to /var/cache/foobar/stackexchange-site-list....
52
votes
3answers
51k views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
52
votes
3answers
16k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
52
votes
5answers
120k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
51
votes
8answers
131k views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget "https://drive.google.com/folderview?id=0B-Zc9K0k9q-WWUlqMXAyTG40MjA&usp=sharing"...
47
votes
4answers
162k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
46
votes
3answers
102k views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
46
votes
3answers
57k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through man wget but the got lost in there, and ...
43
votes
1answer
118k views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null 2>&...
33
votes
2answers
31k views

Resume failed download using Linux command line tool

How do I resume a partially downloaded file using a Linux commandline tool? I downloaded a large file partially, i.e. 400 MB out of 900 MB due to power interruption, but when I start downloading ...
33
votes
3answers
68k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
29
votes
2answers
29k views

Download recursively with wget

I have a problem with the following wget command: wget -nd -r -l 10 http://web.archive.org/web/20110726051510/http://feedparser.org/docs/ It should download recursively all of the linked documents ...
28
votes
3answers
68k views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
28
votes
7answers
94k views

How to download a file through an SSH server?

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C, The issue is, it is very slow to download a file direct from A, so I need download the ...
24
votes
3answers
33k views

Format of cookies when using wget?

What's the Netscape format of wget's cookies.txt? I need to mirror a website that requires login. I use a Chrome extension that returns cookies in that format, I save them in cookies.txt, import with ...
21
votes
1answer
2k views

What is the meaning of “+” in wget progress bar?

I'm downloading with wget using a slow link. Sometimes the progress bar looks like this: 15% [+++++++++++++++===> ] 10,582,848 --.-K/s in 82s What is the meanig of "+"?
21
votes
1answer
36k views

How to download specific files from some url path with wget

If I don't want to have to download the files found in a specific url path manually, what options do I have? Using wildcards fail: $ wget 'http://www.shinken-monitoring.org/pub/debian/*deb' Warning: ...
19
votes
1answer
28k views

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like to see all ...
18
votes
4answers
23k views

wget - How to download recursively and only specific mime-types/extensions (i.e. text only)

How to download a full website, but ignoring all binary files. wget has this functionality using the -r flag but it downloads everything and some websites are just too much for a low-resources ...
17
votes
3answers
34k views

How to get text of a page using wget without html?

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c ...
16
votes
3answers
35k views

How to download an RPM package and install it in one line?

I was wondering whether using wget it was possible to download an RPM and then pipe it through sudo rpm -i to install it, in a single line? I realize I could just run: wget -c <URL> sudo rpm -i ...
16
votes
2answers
22k views

Does -q definitely turn off wget output logging?

I currently have a shell script running on a linux server which is using wget in oder to download a remote web page. This in turn is executed by a cron job which is scheduled to run at certain times. ...
16
votes
2answers
24k views

Using wget, What is the right command to get gzipped version instead of the actual HTML

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I don't know if wget ...
15
votes
6answers
15k views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
15
votes
4answers
43k views

Downloading files using wget

I am trying to download files from this website. The URL is: http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file When I use this command: wget http://www.ncbi.nlm.nih.gov/geo/...
13
votes
2answers
1k views

Why did my wget not die after ssh connection loss?

I sshed to my server and ran wget -r -np zzz.aaa/bbb/ccc and it started working. Then my Internet connection(at my home) got interrupted and I got worried assuming that wget has been hupped because ...
13
votes
1answer
27k views

How to wget a file to a remote machine over SSH?

I'd like to basically pipe a wget command to a file on a remote server over SSH. How can I do this? I know I could simply ssh into the server and have it download the file, but I'd much rather use the ...
13
votes
2answers
11k views

Ignore “other” domains when downloading with wget?

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC. I am using the following wget command to get the files I want: wget -I ABC -r -e ...
13
votes
3answers
13k views

How can I limit downloaded file size in wget?

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size. Can I limit file size with wget? If not, what is a better way?
13
votes
4answers
23k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
12
votes
2answers
2k views

Wget returning binary instead of html?

I am using wget to download a static html page. The W3C Validator tells me the page is encoded in UTF-8. Yet when I cat the file after download, I get a bunch of binary nonsense. I'm on Ubuntu, and ...
12
votes
7answers
15k views

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like puf < urls.txt does not work either. The operating system installed on the server is ...
12
votes
2answers
12k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
12
votes
2answers
39k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
12
votes
3answers
8k views

mounting dropbox (for exploring the files)

I want to simply be able to explore a dropbox filesystem, without much additional load on CPU, memory, storage (i.e., without saving copies of the remote files locally). It seems that the official ...
11
votes
2answers
946 views

What is the “2001 article” for blocking “automated retrieval programs” referred to by the wget man page?

The wget man page states this, under the section for the --random-wait parameter: Some web sites may perform log analysis to identify retrieval programs such as Wget by looking for ...
11
votes
5answers
13k views

How you stop 'wget' after it gets a 404?

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease: $ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' It fetches the first 10 ...
11
votes
3answers
25k views

How to use wget to download from hosting sites?

wget is a very useful tool to download stuff on the internet quickly, but can I use it to download from hosting sites, like FreakShare, IFile.it Depositfiles, Uploaded, Rapidshare? If so, how can I do ...
11
votes
1answer
14k views

Wget, abort retrying after failure or timeout

I'm calling a url with wget: /usr/bin/wget --read-timeout=7200 https://site_url/s Wget performs a GET request every 15 minutes in this case, despite the timeout being set, why does this happen? ...
10
votes
2answers
20k views

“Malformed patch” error while applying the diff to wget source with patch

I would like to apply this patch so I copied code beginning at "Index: src/options.h" and ending with "+@item" and put it into a new file created in wget's source code folder. Then I did: $ patch -p0 ...
10
votes
2answers
25k views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
10
votes
5answers
34k views

Write output of wget or curl to a custom filename based on the url

For example I have a link http://www.abc.com/123/def/ghi/jkl.mno. I want to download it using wget or curl and get the name of output file as def_ghi_jkl.mno, where the part def_ghi is taken from the ...
10
votes
1answer
11k views

How to work around missing 'last-modified' headers?

I'm running wget like this: wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main I get a bunch of these messages: Last-modified header missing -- time-...
9
votes
7answers
8k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
9
votes
5answers
46k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget https://github.com/...
9
votes
2answers
11k views

Download list of files if they don't already exist

If I have a list of URLs separated by \n, are there any options I can pass to wget to download all the URLs and save them to the current directory, but only if the files don't already exist?
9
votes
4answers
18k views

Is there a way to disable wget from getting files from parent directories to given depth?

wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider: www.foo.com/bar1/bar2/bar3/index.html I would like to ...