wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

3
votes
3answers
65 views

how to remove accidentally created '~' and '..'

I'm using wget to download some webpages and I used the -P option without knowing that it wouldn't expend .. or ~. So I found I've created them after running the program. Now the problem is, how do I ...
0
votes
1answer
26 views

Why doesn't wget -r get all FTP subdirectories?

When I did the command : wget -r ftp://user:paswword@ftp.xxxx.ovh.net/ It's missing any sub-sub-directories. Does recursive FTP have a limit?
5
votes
3answers
4k views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
0
votes
1answer
61 views

Download Forum Attachments

Is there a way to download forum jpeg image attachments, possibly using Wget or Curl or some other tool? I would like to download jpeg attachments from specific pages on forums. I'm not interested in ...
1
vote
2answers
4k views

wget to get all the files in a directory only returns index.html

I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes ...
0
votes
2answers
50 views

Downloading images from csv file to a folder based on its id

I have a csv-file that looks like this (short example, there are actually about 20 images per id). ...
0
votes
0answers
40 views

wget not working but apt-get does

Although this question relates to Raspbian running on my Raspberry Pi, it feels like it is a more general Linux query.... The Pi is sat on a network which includes a Microsoft TMG server. I have ...
1
vote
1answer
34 views

wget -k doesn't convert all links

I am trying to mirror a complete website to another domain. I am trying to use wget -mk http://example.com However the -k option leaves some URLs original while other are converted such as: ...
0
votes
2answers
555 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
2
votes
1answer
54 views

How do I securely download files in shell scripting?

I am writing some shell scripts to automatically download software packages off my server and set them up on customer servers. At some point within the script, the download command is wget ...
0
votes
0answers
9 views

How to keep directory structure with aria2?

I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure. Or maybe I should try another program?
2
votes
2answers
78 views

How to use wget to download certain files from a certain directory when the directory itself does not have an index.html?

There are a number of questions similar to this on StackExchange but none address this issue. More specifically I want to download all the pdf files in the 2007 directory at ...
4
votes
3answers
9k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
43
votes
3answers
12k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
0
votes
1answer
30 views

wget and preventing files from downloading on a recursive wget

I can use a command like wget -m -A.jpg http://www.mysite.tld/ but that will download every single file, then if it is not a jpg, it will delete the file, like this: `HTTP request sent, awaiting ...
-2
votes
2answers
48 views

Wget - Download pics with particular url string

Is there a Wget command that will allow me to download all jpegs that include the following url: http://www.sample.com/images/imag/ So all jpegs under that particular url will be downloaded, such as ...
1
vote
1answer
17 views

Multiple user credentials in wget

Using wget, is it possible to supply more than one user/password pair when mirroring a website that has multiple password-protected areas with different credentials? I would like to mirror an entire ...
2
votes
0answers
32 views

Wget: convert-links and avoid redownloading already fetched files?

I am downloading data spread among multiple files that don't change once published. Therefore --timestamping is not good enough, because it is constantly checking whatever resource is changed, which ...
0
votes
0answers
19 views

wget --exclude-directories not working for all

I am making a script for downloading a single file from a website. So I am trying to exclude the remaining directories. Here is it: #!/bin/bash grep -O '<a href=".*>'*html wget \ ...
1
vote
1answer
62 views

How to run Firefox in script-mode [duplicate]

I have a small problem. I want to save web pages using Firefoxs "SAVE" button and then click the "Next" button to the next page and save it. In total I need to store 1000 pages. Apparently I can't ...
0
votes
1answer
37 views

A small script about wget

I want to download an image file called mch_aktuell_gross_20141229_79514688.gif (basically a forecast gif) from www.meteoradar.ch Here is my script: #!/bin/bash wget \ --recursive \ ...
2
votes
3answers
50 views

Pipe files downloaded with wget into a function

I'm downloading a bunch of images with wget and I want to rm the files if they are empty with a function to which I can pass the file. My question is how do I pipe the images to a function like this: ...
2
votes
1answer
491 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
3
votes
0answers
91 views

Multiple wget calls in a single session?

I have the following code to retrieve a page from a website that needs authentication (it uses Sungard Higher Education authentication): wget --delete-after --save-cookies cookies.txt ...
1
vote
1answer
39 views

wget recursion problems

Is there any way to get wget to recursively download from one starting URL (including every prerequisites) and include other paths for which the same will be done? To make this clearer: The file ...
0
votes
1answer
17 views

Excluding some php-pages but not others with wget?

I want to use wget to recursively download a website. The site contains articles served by a php-script, eg. www.example.com/article.php?id=1021&ch=5 - these are what I want, all separate article ...
0
votes
1answer
108 views

wget 403 forbidden error on AWS

I'm not the most talented developer out there, and the question I'm asking may be basic, but I need some help:) I used wget to get source content from Seeking Alpha on my AWS server, and have been ...
7
votes
2answers
7k views

How to use wget to download from hosting sites?

wget is a very useful tool to download stuff on the internet quickly, but can I use it to download from hosting sites, like FreakShare, IFile.it Depositfiles, Uploaded, Rapidshare? If so, how can I do ...
1
vote
0answers
82 views

wget resume download from dropbox gives can't write error

I want to resume download from dropbox using wget. I was downloading a file wget -O SQLyog.exe "https://www.dropbox.com/s/dfv2z0le0c2bo7z/SQLyog-9.3.3-0Trial.exe?dl=1" .......... Length: 15775640 ...
1
vote
1answer
291 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
7
votes
1answer
4k views

Obtaining md5 during wget

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so ...
12
votes
3answers
14k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
-2
votes
2answers
772 views

What is the meaning of wget -O -

I am trying to install dropbox onto my Debian machine, and I saw the instruction cd ~ && wget -O - "some website leading to the download of a tar file" | tar xzf - But what I did was ...
0
votes
1answer
73 views

What options does wget --report-speed take?

When I do this command: wget --report-speed=type they only type it accepts is bits. It won't have numbers, kilobits / kilobytes or bytes. The help page (wget --help) says: --report-speed=TYPE ...
4
votes
1answer
218 views

Bash Script - Wget Graphical Download Progress

I'm trying to port this code from wget + zenity to wget + kdialog. The code with zenity: #!/bin/bash # Start wget | zenity # Note the & at the end of the pipe, this allows the script to continue ...
1
vote
1answer
92 views

“wget --restrict-file-names=windows” seems to fail converting links for NTFS

I've downloaded a website from the net using the wget command-line tool with --restrict-file-names=windows so that once the download is complete I could copy the downloaded files to a mounted Windows ...
0
votes
1answer
23 views

Configure wget to ignore a specified domain, or treat it differently?

I'm using wget to mirror an old website with all of its --page-requisites. Unfortunately, one of the domains which used to serve some of these requisites - is now offline. Every time wget tries to get ...
1
vote
1answer
207 views

How do I put a time stamp with: wget --output-file=[FILENAME] [URL]

So I've been trying to put a timestamp on some images I'm downloading from a webcam, but so far it's been saving the images with the names camera3_date' here's my line that I enter: wget ...
0
votes
0answers
85 views

wget stops downloading: Cannot write to `-' (success)

Please have in mind that I don't know anything about this, nor any of its vocabulary, I am a simple web coder, trying to figure my way on the universe! I'm trying to link dropbox on my server, I am ...
0
votes
0answers
12 views

wget mirror site with images stored on other different domains [duplicate]

I'm trying to wget mirror of site atompunk: # wget -l 2 -c -m "http://atompunk.tumblr.com" For me, it take only index. I'm interesting in pictures which is stored on the other domains... How? ...
0
votes
1answer
104 views

wget: 401 Unauthorized

I want to download an entire website that is protected by NTLM but it fails with: HTTP-Anforderung gesendet, warte auf Antwort... 401 Unauthorized Unbekanntes Authentifizierungsschema. ...
1
vote
1answer
613 views

wget is unable to resolve host address 80% of the time

I have a wget 'http://xxxx.net/somepage.asp?params@abc' in a bash file. When I run it, four out of five times it says: Resolving xxxx.net (xxxx.net)... failed: No address associated with hostname. ...
1
vote
3answers
186 views

Constructing URLs with a list of words and downloading from the URLs with command line

Suppose I have a list of words in a text file e.g: abstinence ; abuse ; academic ; accommodate ; accommodation ; accommodationist ; accomplice ; accomplish ; accord ; account ; accountable ; ...
1
vote
1answer
44 views

wget doesn't convert URLs properly when --convert-links is specified

According to the man pages, when I specify --convert-links to wget: After the download is complete, convert the links in the document to make them suitable for local viewing [...] Note that only ...
1
vote
2answers
58 views

Download all pages linked on a page in a certain region

I would like to download my users activity page of this Unix&Linux with wget and all pages that are linked in the activity list. I tried wget -m -l 2 which should mirror the site recursively ...
3
votes
4answers
224 views

Possible to download these webpages as a book?

I would like to download a SQL tutorial here http://www.w3schools.com/sql/default.asp, as a book with all linked SQL related sections. Here is my command wget -r -np -nH -p -k ...
0
votes
1answer
48 views

Mirror list of files

I've a list of URLs: https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.3.0/angular.min.js https://cdnjs.cloudflare.com/ajax/libs/bootstrap-select/1.6.2/js/bootstrap-select.min.css ...
-2
votes
1answer
157 views

Resume failed download using Linux command line tool

How to resume partially downloaded file using linux commandline tool. I downloaded large file partially i.e. 400MB out of 900MB due to power interruption, but when I start downloading again it ...
3
votes
2answers
1k views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
3
votes
2answers
7k views

“Short write” error message with wget

I got the following error when trying to pull down a 300 MB ISO. It appears to pull down 25 MB then stop. Any idea what's going on here? ~ # wget ...