wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

4
votes
2answers
5k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
3
votes
3answers
993 views

Download one html page with all needed graphics and linked pdf/zip files

I want to download a website from an URL, to view it locally, more exactly: Download one single html page (no other linked html pages) and everything needed to display it (css, images, etc.) Also ...
3
votes
5answers
2k views

Bash: Fastest way of determining dimensions of image from URL

I'm trying to figure out a really fast method in bash of determining an images dimensions. I know I could wget the image and then use imagemagick to determine the height and width of the image. I'm ...
5
votes
2answers
2k views

Ignore “other” domains when downloading with wget?

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC. I am using the following wget command to get the files I want: wget -I ABC -r -e ...
1
vote
2answers
687 views

Wget fails to download PNG files from Blogger

I am trying to mirror a blogger site so that I can have an exact copy of it on my filesystem to view. I have tried issuing the following command on Linux: wget -r -k -x -e robots=off --wait 1 ...
3
votes
0answers
156 views

WGET - Authenticating root AND all children folders

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below. There's a website that I'm trying to mirror. It is something ...
6
votes
3answers
116 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
0
votes
1answer
90 views

Mirroring site with an offset

I followed answers here http://webapps.stackexchange.com/questions/12311/how-to-download-all-english-books-from-gutenberg which are using wget I want to download using wget from a particular starting ...
2
votes
1answer
1k views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
2
votes
2answers
2k views

wget grabbing empty files that actually exist

I have a script that's meant to download a certain number of files from a remote server. It only needs to do this every 24 hours, since they are JSON sources for a database on my server. The files are ...
1
vote
4answers
4k views

Write output of wget or curl to a custom filename based on the url

For example I have a link http://www.abc.com/123/def/ghi/jkl.mno. I want to download it using wget or curl and get the name of output file as def_ghi_jkl.mno, where the part def_ghi is taken from the ...
2
votes
1answer
1k views

Get all images from website

I'm trying to wget all the images pictures from a website. I thought I knew how to use it, but I guess I don't. The images I'm trying to get come from here. The command I'm using is: wget -prA.png ...
1
vote
1answer
1k views

How to figure out where a link gets redirected

I currently working on a script that updates a file. The file has several versions. The important ones have static links. These get redirected to the actual file which then gets downloaded using wget. ...
9
votes
2answers
5k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
11
votes
2answers
12k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through mam wget but the got lost in there, and ...
0
votes
1answer
270 views

Download documents referenced by a web page with wget

I want to download all the documents pointed to by a web page with wget. When I run wget, it only downloads the webpage, not all the contents that the web page links to. wget --content-disposition ...
10
votes
2answers
4k views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
2
votes
2answers
559 views

Problem downloading Java JDK with WGET on Ubuntu

I'm attempting to install the Java JDK through the command line using WGET on ubuntu (no desktop). Here is the picture I took. The file that was downloaded was not the tar ball of the JDK: How can I ...
0
votes
2answers
724 views

how do I get the response ouput from the url .download it in file,not download the code

My file like this with some number or id 404000847307897 404000857827460 404001044194601 404000835511595 404000338418501 404000338654003 404000290144612 404000344175621 404000290144612 ...
3
votes
1answer
1k views

How to download all html files from a URL's directory with wget?

I'd like to use wget to download a tutorial in a website, but this tutorial is in a specific directory. Suppose the URL is the tutorial main page http://www.abc.com/aaa/bbb/ccc/index.html. But the ...
1
vote
2answers
83 views

How to download several urls into one file

I have URLs like this. http://203.90.117.109/wds_cdma/wds.aspx?min=404000290205346 http://203.90.117.109/wds_cdma/wds.aspx?min=404000832812637 ...
-2
votes
3answers
1k views

What is the meaning of wget -O -

I am trying to install dropbox onto my Debian machine, and I saw the instruction cd ~ && wget -O - "some website leading to the download of a tar file" | tar xzf - But what I did was ...
3
votes
3answers
2k views

Using wget to get file names from a text file

I have found a website that hosts a few files that I'm after, there's too many to download them all individually. The filenames take a fairly standard and reproduceable form i.e. 1_a, 1_b, 1_c etc Is ...
0
votes
1answer
429 views

403 http code appears when I go to a certain website

I wrote a PHP script to a particular site using a home computer for debugging. Script worked properly, when I tested it on a home computer. The script has stopped working when I deploy the script on ...
1
vote
1answer
1k views

How can I use wget to download large files? [closed]

I want to get the ISO for a particular version of Linux, say 'Mint 14 “Nadia” KDE '. I tried using wget in the terminal to download the ISO of this distro: $ wget ...
2
votes
1answer
833 views

Why is wget skipping files while trying to mirror a site

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html wget --recursive --no-clobber --page-requisites --html-extension --convert-links ...
3
votes
2answers
261 views

Wget with different filename, but same extension

I need wget to save to a different folder, different filename, and same extension, but I do not have the filename before downloading. For example, let's say I have: ...
-2
votes
1answer
640 views

Downloading the Google search results page using Linux commands

When I try using wget command, the page which gets downloaded contains only the lefthand side part. I want the right-hand side part of the search-results page, which is called Knowledge Graph. Can ...
4
votes
1answer
1k views

Why wget does not work via ssh tunnel? What does proxy prevent ssh-client to do?

To test things I'm trying to establish a local ssh tunnel from my laptop to a site via ssh-server, and download a page of the site (or view it in a browser). The tunnel is made like this: $ ssh -L ...
1
vote
1answer
527 views

How to make local mirror of a website with a confirmation popup?

I need to get a local copy of a website which requires confirming one's age before displaying content. No complex forms, just a checkbox and a button, "yes, I'm adult". I tried to use wget but with no ...
1
vote
1answer
772 views

wget recursive with files without extension

I'm not very good with Linux so I have maybe a basic question. I've search around for the "how to" and I couldn't find anything. I'm trying to get a site with wget. The problem is that it: Have ...
0
votes
0answers
155 views

How do i WGET an image from a webaddress and store it on a remote ftp location as a cronjob on dd-wrt

From a DD-WRT router I want to create a cron-job that wget's an image from a remote location. This image is to be passed on to a remote FTP server for storage. The process is to be repeated every ...
6
votes
3answers
8k views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
3
votes
2answers
350 views

How do I prevent wget from loading Apache directory listings in different orders?

I’m trying to recursively download a directory using this command: wget -r -l2 --no-parent -A "*.pdf" -nd -N -e robots=off http://… The script gets executed every 15 minutes, so I want to reduce ...
0
votes
1answer
1k views

wget hangs AFTER download complete, on exit_group

I am running CentOS release 6.3 (Final). This issue occurs for every VM on my server. When connected over SSH, wget hangs after download certain URLs. I am running the following command: wget ...
2
votes
2answers
762 views

How to optimize wget download script

Site contents <html> <body> <a href="http://www.sagar.com/" title="sagar">Sagar</a> <a href="http://www.sagarcom/1/" title="sagar1">Sagar Link1</a> ...
2
votes
1answer
1k views

How to download pdf files referenced by pages one or two levels inside of the seed

I have a list of URLs, and they are all like this: http://www.contratos.gov.co/consultas/detalleProceso.do?numConstancia=10-10-85225 ; inside those pages there are .pdf files being referenced. I need ...
0
votes
2answers
2k views

How to disable gzip compression with wget?

There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and ...
2
votes
1answer
383 views

Method to perform DNS lookup step very quickly during wget

When I download any document from internet using wget. It shows maximum amount of time consumed in getaddrinfo(). I know getaddrinfo() used for DNS lookup. Tell me any method, so that this time is ...
3
votes
1answer
694 views

Recursive wget eventually starts failing with “Unable to resolve host” / “Temporary failure in name resolution”

When recursively wget-ting a fairly large site, after downloading 18 MB or so, wget starts complaining "Unable to resolve host" and/or "Temporary failure in name resolution", even though I can look up ...
2
votes
1answer
773 views

DNS cache for wget

I want wget to use a global DNS cache, not just in memory for the current instance. The wget manual mentions --no-dns-cache Turn off caching of DNS lookups. Normally, Wget remembers the IP ...
1
vote
0answers
143 views

Wget, downloading outdated files

I am over writing a particular file (called "demo") on my server and wget-ting it. However, sometimes wget retrieves an outdated file, despite my checking on the net that the file has been updated. ...
2
votes
1answer
1k views

Recursively downloading all files from a website's child directory using wget

I am trying to recursively download the contents of a particular directory (via http from a web server) using wget. The command I am using is: wget -r -l 0 --user=josh --ask-password ...
4
votes
3answers
550 views

How to tell if a file is being downloaded by wget?

I use wget to download files (most are zip files) automatically for me during the night. However, sometimes in the morning I find that a few files cannot be unzipped. I don't know why this is ...
3
votes
1answer
2k views

wget command still downloads folder after exclusion

I am trying to learn wget better. However when I run my code it still downloads the forum section. Where am I going wrong? wget --recursive --no-clobber --page-requisites --html-extension ...
12
votes
3answers
19k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
79
votes
2answers
76k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
0
votes
3answers
126 views

Wget images issue from api_key

We have a vendor that sends us xml feeds of new items with the appropriate path and image name. I created script that will strip the web address path for the image, but if I try to obtain these with ...
0
votes
1answer
709 views

Bash While loops

I have a loop set up like the following #!/bin/bash cat $1|while read line;do wget --limit-rate=50k http://url/$line.json -O got-$line; if [ "$?" -ne "0" ]; then rm got-$line; fi ...
3
votes
1answer
772 views

Download content from a certain <div> using wget

I'm trying to download all the images from a certain ImageBam gallery. I tried doing this: wget -P pics -H -nd -r -A '.jpg,.jpeg,.png,.gif,' -erobots=off ...