wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

2
votes
1answer
49 views

Multiple user credentials in wget

Using wget, is it possible to supply more than one user/password pair when mirroring a website that has multiple password-protected areas with different credentials? I would like to mirror an entire ...
2
votes
1answer
193 views

Wget: convert-links and avoid redownloading already fetched files?

I am downloading data spread among multiple files that don't change once published. Therefore --timestamping is not good enough, because it is constantly checking whatever resource is changed, which ...
1
vote
1answer
85 views

How to run Firefox in script-mode [duplicate]

I have a small problem. I want to save web pages using Firefoxs "SAVE" button and then click the "Next" button to the next page and save it. In total I need to store 1000 pages. Apparently I can't ...
0
votes
1answer
63 views

A small script about wget

I want to download an image file called mch_aktuell_gross_20141229_79514688.gif (basically a forecast gif) from www.meteoradar.ch Here is my script: #!/bin/bash wget \ --recursive \ ...
2
votes
3answers
102 views

Pipe files downloaded with wget into a function

I'm downloading a bunch of images with wget and I want to rm the files if they are empty with a function to which I can pass the file. My question is how do I pipe the images to a function like this: ...
3
votes
0answers
356 views

Multiple wget calls in a single session?

I have the following code to retrieve a page from a website that needs authentication (it uses Sungard Higher Education authentication): wget --delete-after --save-cookies cookies.txt ...
0
votes
1answer
846 views

wget 403 forbidden error on AWS

I'm not the most talented developer out there, and the question I'm asking may be basic, but I need some help:) I used wget to get source content from Seeking Alpha on my AWS server, and have been ...
0
votes
1answer
27 views

Excluding some php-pages but not others with wget?

I want to use wget to recursively download a website. The site contains articles served by a php-script, eg. www.example.com/article.php?id=1021&ch=5 - these are what I want, all separate article ...
1
vote
1answer
72 views

wget recursion problems

Is there any way to get wget to recursively download from one starting URL (including every prerequisites) and include other paths for which the same will be done? To make this clearer: The file ...
2
votes
0answers
400 views

wget resume download from dropbox gives can't write error

I want to resume download from dropbox using wget. I was downloading a file wget -O SQLyog.exe "https://www.dropbox.com/s/dfv2z0le0c2bo7z/SQLyog-9.3.3-0Trial.exe?dl=1" .......... Length: 15775640 ...
0
votes
1answer
827 views

What options does wget --report-speed take?

When I do this command: wget --report-speed=type they only type it accepts is bits. It won't have numbers, kilobits / kilobytes or bytes. The help page (wget --help) says: --report-speed=TYPE ...
1
vote
1answer
593 views

“wget --restrict-file-names=windows” seems to fail converting links for NTFS

I've downloaded a website from the net using the wget command-line tool with --restrict-file-names=windows so that once the download is complete I could copy the downloaded files to a mounted Windows ...
0
votes
1answer
39 views

Configure wget to ignore a specified domain, or treat it differently?

I'm using wget to mirror an old website with all of its --page-requisites. Unfortunately, one of the domains which used to serve some of these requisites - is now offline. Every time wget tries to get ...
1
vote
1answer
2k views

How do I put a time stamp with: wget --output-file=[FILENAME] [URL]

So I've been trying to put a timestamp on some images I'm downloading from a webcam, but so far it's been saving the images with the names camera3_date' here's my line that I enter: wget ...
5
votes
2answers
585 views

Bash Script - Wget Graphical Download Progress

I'm trying to port this code from wget + zenity to wget + kdialog. The code with zenity: #!/bin/bash # Start wget | zenity # Note the & at the end of the pipe, this allows the script to continue ...
0
votes
1answer
907 views

wget: 401 Unauthorized

I want to download an entire website that is protected by NTLM but it fails with: HTTP-Anforderung gesendet, warte auf Antwort... 401 Unauthorized Unbekanntes Authentifizierungsschema. ...
3
votes
1answer
4k views

wget is unable to resolve host address 80% of the time

I have a wget 'http://xxxx.net/somepage.asp?params@abc' in a bash file. When I run it, four out of five times it says: Resolving xxxx.net (xxxx.net)... failed: No address associated with hostname. ...
1
vote
1answer
119 views

wget doesn't convert URLs properly when --convert-links is specified

According to the man pages, when I specify --convert-links to wget: After the download is complete, convert the links in the document to make them suitable for local viewing [...] Note that only ...
1
vote
4answers
436 views

Constructing URLs with a list of words and downloading from the URLs with command line

Suppose I have a list of words in a text file e.g: abstinence ; abuse ; academic ; accommodate ; accommodation ; accommodationist ; accomplice ; accomplish ; accord ; account ; accountable ; ...
1
vote
2answers
117 views

Download all pages linked on a page in a certain region

I would like to download my users activity page of this Unix&Linux with wget and all pages that are linked in the activity list. I tried wget -m -l 2 which should mirror the site recursively ...
0
votes
1answer
131 views

Mirror list of files

I've a list of URLs: https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.3.0/angular.min.js https://cdnjs.cloudflare.com/ajax/libs/bootstrap-select/1.6.2/js/bootstrap-select.min.css ...
3
votes
1answer
5k views

Resume failed download using Linux command line tool

How to resume partially downloaded file using linux commandline tool. I downloaded large file partially i.e. 400MB out of 900MB due to power interruption, but when I start downloading again it ...
1
vote
1answer
607 views

List of links to images in website with wget

I have a question, as much as I think I cannot think of the way to put all links of pictures in a list, I asked this question before but it was marked as duplicate to post where OP asked to download ...
1
vote
3answers
5k views

wget downloading tar.gz file as html?

I need to download this file http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz using wget. I use the command wget ...
0
votes
1answer
25 views

allpackages.txt.gz has two different file-sizes?

I saw http://linux.koolsolutions.com/2008/12/19/counting-number-of-packages-under-debian-linux/ and I wanted to count the number of packages in testing. I tried as shared therein :- ...
0
votes
2answers
333 views

Get the file list in an HTML directory listing

I can get all files on the bash patches site by downloading them in a sequence: SEQ=$(seq -f "%03g" 1 30) for i in $SEQ; do wget http://ftp.gnu.org/gnu/bash/bash-4.3-patches/bash43-$i; done But ...
2
votes
0answers
363 views

Why can't I transfer a file via wget (FTP) using exec() function in PHP?

Put simply, I can't work out why the code below won't carry out a file transfer? I have been assured by the sysadmin staff on the remote server that there are no permissions/firewall issues that could ...
1
vote
2answers
140 views

Can i capture and send GET requests with wget?

So, right now i am using Charles to do this and it works fine. But can i use wget to capture and send GET requests? I want to use wget and the command line because of the scriptability and ...
0
votes
1answer
472 views

curl wget - 403 Forbidden

I am trying to download a site as you see bellow with curl or wget but either one seems to work. I tried different options like "curl -k" seemed promising, but did not work. Every one is allowed to ...
1
vote
0answers
99 views

How to abort wget after getting N times file not found?

I try to grab pictures from few galleries of some site. I do not know how many files are there, but I know the files are continuous named. So I use a regex like that: wget ...
1
vote
1answer
471 views

Wget crawling without download file

I am just trying to count how many URLs match with the pattern x--*.html. In this case below all files that match are being downloaded. $ wget --random-wait --mirror --no-parent -A x--*.html ...
0
votes
2answers
838 views

After downloading an .asp or .php file, can it be converted to a .html file? [closed]

I wonder if downloading files via wget and having them named .asp or .php means that I download the webpage? Aren't these executable files? If so, after downloading an .asp or .php file, can it be ...
3
votes
4answers
473 views

Possible to download these webpages as a book?

I would like to download a SQL tutorial here http://www.w3schools.com/sql/default.asp, as a book with all linked SQL related sections. Here is my command wget -r -np -nH -p -k ...
2
votes
3answers
92 views

Count of several keywords on a site

I currently have the following command wget -O - -q http_url_goes_here | grep -Eci '(cart|shop|buy|order|store|shipping|products|price)' It gives me a count of ALL occurences of each of the ...
0
votes
0answers
478 views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
2
votes
2answers
5k views

wget download using list of urls

I'm trying to download several files with wget. The relevant URLs are stored in the file url.txt. When using the command wget --content-disposition -i url.txt it doesn't find the .txt file. Where do ...
18
votes
1answer
987 views

What is the meaning of “+” in wget progress bar?

I'm downloading with wget using a slow link. Sometimes the progress bar looks like this: 15% [+++++++++++++++===> ] 10,582,848 --.-K/s in 82s What is the meanig of "+"?
1
vote
1answer
402 views

wget internal buffer size

I'm constantly using wget to download many files over 1GB each. I'm getting concerned about how this may be straining my HDD. Can wget's internal buffer size be modified so that it writes to my HDD ...
4
votes
6answers
2k views

Save multiple URL targets to text files

I have a list of URLs and I want to save each of their targets in a separate text file. Here's an example of the input file containing the URLs: ~$: head -3 url.txt ...
2
votes
3answers
494 views

How can I use wget to create a list of URLs from an index.html?

So wget has an ability to recursively download files, however it does it one file at a time. I would like to pass in a directory URL, and for each URL it encounters in the recursion for it to spawn ...
0
votes
1answer
94 views

wget rotated backup problem

I'm calling wget 1.11.4 as follows $ wget -nd -S -N --backups=4 -r --no-parent --reject="index.html*" -e robots=off <url> to repeatedly download all files from a server. My problem is, ...
4
votes
1answer
131 views

wget files by pattern only from specified directories recursively

I need to download on an hourly basis (sometimes more frequently), files which are being written in segments of 24 hours. The files I am interested in are in specific subdirectories which I am trying ...
6
votes
5answers
18k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
9
votes
3answers
11k views

downloading files using wget

I am trying to download an http link from here. The URL is http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file When I use wget ...
7
votes
3answers
4k views

How you stop 'wget' after it gets a 404?

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease: $ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' It fetches the first 10 ...
3
votes
2answers
8k views

wget to get all the files in a directory only returns index.html

I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes ...
1
vote
2answers
1k views

How to get wget to use the direct URL file name as output file name for a shortened URL

I'm trying to download all the books from a URL list in bulk. The list is in a text file in http://ligman.me/W3TuLr. I'm trying to use wget on linux for this. All these URLs seem to be created using a ...
1
vote
1answer
95 views

How to burn an iso while downloading it

I'm in a real hurry so I thought there is a way to do that but I don't know how. I thought about piping wget -c cdimage.ubuntu.com/daily-live/current/utopic-desktop-amd64.iso | BurningApp but I don't ...
1
vote
0answers
14 views

Why does wget return a HTTP 403 error to download openSSH pet? [duplicate]

I am trying to learn how I would get OpenSSH using the command line. I found a *.pet link on this webpage: http://puppylinux.org/wikka/OpenSSH which says: Installation Install an appropriate ...
24
votes
4answers
25k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...