wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

7
votes
3answers
2k views

mounting dropbox (for exploring the files)

I want to simply be able to explore a dropbox filesystem, without much additional load on CPU, memory, storage (i.e., without saving copies of the remote files locally). It seems that the official ...
0
votes
3answers
306 views

How to also download duplicate images?

I have a list of 900 URLs. Each page contains one image. Some images are duplicates (with same URL). I want to download 900 images, including duplicates. I was able to download all pages and the ...
0
votes
1answer
1k views

make install - how to use wget, not curl

I do not have root access on my Ubuntu but want to install git. I was following this tutorial but when running make or make install it says curl is not installed. Can I use wget instead if this is a ...
5
votes
3answers
5k views

wget - How to download recursively and only specific mime-types/extensions (i.e. text only)

How to download a full website, but ignoring all binary files. wget has this functionality using the -r flag but it downloads everything and some websites are just too much for a low-resources ...
4
votes
2answers
448 views

Download from dynamically changing urls via script

I want to write a script to download all Walter Lewin lectures from this page in wmv format: http://videolectures.net/mit801f99_physics_classical_mechanics/ However the links to the wmv file for ...
7
votes
6answers
2k views

Is there a way to use curl interactively? Or is there an interactive curl/wget shell?

Imagine something like this: $ curlsh http://www.example.org > GET /foo/bar/bam ...output here... > POST /thing/pool ... ... result here.... is there a tool that lets me do that?
1
vote
1answer
860 views

How can I update certificates on my system to fix wget's error: “Unable to establish SSL connection.”?

Sometimes I notice that I am unable to connect via SSL with curl and wget to websites that have certificates that should be valid. One example might be https://squareup.com/ which has a Verisign ...
2
votes
1answer
563 views

Predict filename before downloading from a URL, in shell script

I have a shell script that downloads files from a list using wget, and resumes automatically if there is any non-critical error (because of unstable WIFI during storms). The problem is, I want to ...
40
votes
3answers
10k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
3
votes
1answer
435 views

Extract directory from wget's stdout

I am trying to wget a tarball from github.com and, without creating a temporary file, extract a subdirectory from it: wget -qO- https://github.com/django-nonrel/django-nonrel/tarball/develop | tar ...
5
votes
1answer
553 views

Download tool needed — with custom headers, resume, retry, custom filename output, and dynamic redirection

Simple problem -- I need a download manager, console, GUI, does not matter. The catch is in the features: custom header -- I have to pass a cookie in order to download (authorization) resume -- so ...
0
votes
3answers
2k views

wget is returning an html page instead of original file

In my case the problem is with SourceForge.net. Actually I was trying to download these files: http://sourceforge.net/projects/wxpython/files/wxPython/2.9.4.0/wxPython-src-2.9.4.0.tar.bz2/download ...
0
votes
1answer
377 views

wget images with certain path

im looking to only grab images from a document with a certain path for example domain.com/uploads/287167/file_name.jpg only grab those with the uploads/number/filename im currently doing it this ...
0
votes
2answers
796 views

wget - Considers download done after 1 second even though it isn't

I am trying to use wget to download the Raspbian image for the Raspberry Pi (http://downloads.raspberrypi.org/images/raspbian/2012-07-15-wheezy-raspbian/2012-07-15-wheezy-raspbian.zip). It starts ...
2
votes
3answers
812 views

Possible to get “old” time stamp behaviour?

When downloading a file or e.g. unzipping one, the time stamp is not set to when the file appeared on my harddisk, which is what I want. See this example ~$ wget ...
0
votes
2answers
792 views

How do I get wget to download a cgi file behind robots.txt?

As an example - http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22. I found a way to get through the robots.txt restrictions, but even then, it just downloads a binary file ...
2
votes
2answers
166 views

Print like Firefox

Made a CLI script to print Wikipedia articles: WIKI='https://en.wikipedia.org/w/index.php?title=' TITLE=$1 PRINTABLE='&printable=yes' SRC=$WIKI$TITLE$PRINTABLE HTML=$1".html" PS=$1".ps" wget ...
5
votes
2answers
6k views

How to use wget to download from hosting sites?

wget is a very useful tool to download stuff on the internet quickly, but can I use it to download from hosting sites, like FreakShare, IFile.it Depositfiles, Uploaded, Rapidshare? If so, how can I do ...
8
votes
1answer
6k views

How to get text of a page using wget without html?

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c ...
4
votes
1answer
934 views

wget --spider: how to tell where broken links are coming from

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes ...
1
vote
1answer
1k views

How to know whether wget has completed download successfully?

I attempted to download openSUSE 12.1 through wget. It stopped after 64% showing: 05:54:48 (120.30 KB/s) - `openSUSE-12.1-DVD-i586.iso' saved 4429185024/4429185024] It appears that the download is ...
2
votes
1answer
1k views

Will wget -k still convert links to relative paths if wget is stopped prematurely?

I am running wget on a site that (I have come to realize) has two languages - and duplicate pages for each language. I've already got the content in English, so I'd like to stop wget from downloading ...
1
vote
1answer
475 views

How to download openSUSE 12.1 throught `wget -c`?

http://software.opensuse.org/121/en Usually I right click and open the download link in a new tab. Copy the url on the command line with wget -c. In this case I just can't right click on the ...
3
votes
2answers
6k views

“Short write” error message with wget

I got the following error when trying to pull down a 300 MB ISO. It appears to pull down 25 MB then stop. Any idea what's going on here? ~ # wget ...
23
votes
4answers
10k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
13
votes
5answers
20k views

How to download a file through an SSH server?

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C, The issue is, it is very slow to download a file direct from A, so I need download the ...
2
votes
1answer
191 views

Is there any option in wget to get the referers or parent of a link that are crawled by wget

Is there any to get or find the referrers of a link that were crawled by wget? Suppose that some link returns a 400 error. wget only shows the url on which the 400 error occurrs, not the referrer of ...
2
votes
1answer
3k views

Format of cookies when using wget?

What's the format of the "natescapes" cookies.txt when I need to mirror a website that requires login. I use some chrome extension that returns cookies in that format, I save them in cookies.txt, ...
5
votes
2answers
3k views

Download list of files if they don't already exist

If I have a list of URLs separated by \n, are there any options I can pass to wget to download all the URLs and save them to the current directory, but only if the files don't already exist?
2
votes
1answer
1k views

how to specify the forwarding port when using multiple tsocks services?

According to the answer of this question, I have my /etc/tsocks.conf containing these lines: path { server = localhost server_port = 1081 reaches = <ip-address-of-server-b>/32 } path { server = ...
3
votes
2answers
621 views

How to check if `wget` supports a certain feature programatically?

Is there a way to programatically detect if the currently installed wget supports a given feature, for example, IRI support? I’m asking because I’d like to add these lines to my .wgetrc: # Use UTF-8 ...
0
votes
1answer
3k views

How to use wget or other tool to download a file, for which the link is hidden(activates on-click only)

I want to to download some subtitle files, stored in rar files form http://subs.sab.bz/ The site provides rss feeds for its new releases. Unfortunately, the link provided will open a download page, ...
2
votes
1answer
3k views

How do I create a local copy of a complete website section from OSX using wget?

This question follows from: How do I create a local copy of a complete website section from OSX using curl? After discovering OSX's native curl wouldn't do this task I downloaded wget from here: ...
1
vote
1answer
239 views

What is wget idiom for curl -o?

So I use code in some function like if [ ! -e $DOWNLOADS/$1 ]; then echo_run curl -L http://$4/$5/$1 -o $DOWNLOADS/$1 fi I wonder how to achieve the same thing without curl?
2
votes
3answers
535 views

lower bound for download speed in wget

I am trying to download some files by using wget but after an hour my download speed decrease. And if I cancel downloading and restart it, download speed will increase for an hour. Is it possible to ...
5
votes
2answers
5k views

“Malformed patch” error while applying the diff to wget source with patch

I would like to apply this patch so I copied code beginning at "Index: src/options.h" and ending with "+@item" and put it into a new file created in wget's source code folder. Then I did: $ patch -p0 ...
1
vote
3answers
1k views

routing wget --recursive to stdout (as in -O -) rather than a file

I want to "scan" a website content, recursively, and pipe it to another program. When I run: wget -r -O - $SITE I get WARNING: combining -O with -r or -p will mean that all downloaded content ...
1
vote
1answer
148 views

Alert when a process exits

I often download files using wget and cURL. Is there anyway I can make my Linux box give a visual alert when either of those processes exit (i.e: the download completes) PS : I am using Ubuntu ...
2
votes
1answer
553 views

Troubleshooting `$ wget -A.pdf -r site.com`

I am trying to get homework papers with the command but for some reason, it does not return them. Any idea why? $ wget -A.pdf -r -nd https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/ ...
3
votes
3answers
4k views

Is there a way to disable wget from getting files from parent directories to given depth?

wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider: www.foo.com/bar1/bar2/bar3/index.html I would like to ...
0
votes
1answer
582 views

Retrieve URL content and follow redirects

I want to visit an URL in CentOS and then automatically follow the redirects. So, for example: Visit http://domain.com This will redirect my by a `header()` to another webpage, so automatically ...
8
votes
2answers
4k views

Download recursively with wget

I have a problem with the following wget command: wget -nd -r -l 10 http://web.archive.org/web/20110726051510/http://feedparser.org/docs/ It should download recursively all of the linked documents ...
4
votes
2answers
2k views

How to copy someone's else folders from public.me.com with a wget-like tool?

How can I copy a folder from http://public.me.com/ (a service related to iDisk, or MobileMe) to my local filesystem with a Unix tool (like wget, a command-line non-interactive tool)? The problem is ...
6
votes
1answer
523 views

How to download all images whose size>100kb in a webpage?

I want to download images contained in some web pages, but only those that are larger than 100 KB. So far I've been downloading all the images and then deleting the small ones, but it's not very ...
0
votes
1answer
160 views

Is it possible to add some sort of hook to wget to pre-process HTML returned?

I want to mirror one of my websites. Unfortunately, the site is in its preview mode, and a GET param must be appended to all URLs in the page for it to continue in its preview mode. This GET param ...
4
votes
2answers
12k views

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to ...
4
votes
3answers
17k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
3
votes
2answers
898 views

Wget - Mirroring a full website with requisites on differents hosts

I am trying to make a full static copy of a wordpress website with wget to be browsed without any network connection (all links and images must be converted). The different requisites for the pages ...
3
votes
1answer
432 views

Download only recently modified files with wget over FTP

When fetching recursively with wget from an FTP server, how can I download only files modified after a given date? The use case is that I need to sync with several servers nightly, but I only need ...
4
votes
2answers
888 views

Does -q definitely turn off wget output logging?

I currently have a shell script running on a linux server which is using wget in oder to download a remote web page. This in turn is executed by a cron job which is scheduled to run at certain times. ...