wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

2
votes
1answer
593 views

How to wget website with same page and content names?

I'm trying mirror a website where it has page: http://www.site.com/news and contents: http://www.site.com/news/36-news/news-one http://www.site.com/news/37-news/news-two ...
5
votes
2answers
4k views

wget recursive for only files that pass a regexp

I would like to use wget recursively on an ftp directory like this: ftp://ftp.ensembl.org/pub/current_fasta/ Which contains another 2 directory levels inside, the first with the species name, and ...
5
votes
3answers
1k views

How to wget Youtube horrible URL

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date: wget ...
1
vote
3answers
587 views

wget batch linux file

I am trying to download images from a manufacturer's site using wget. When I run the command from the shell it works fine. But when I run it from a file I am getting a placeholder file. Here is the ...
7
votes
3answers
4k views

How can I limit downloaded file size in wget?

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size. Can I limit file size with wget? If not, what is a better way?
27
votes
2answers
8k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
6
votes
1answer
4k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...
0
votes
1answer
569 views

Force wget to download only from HTTPS?

wget https://skype.com -O - > /dev/null isn't good. I just want to create a short bash script to check that an HTTPS cert is valid or no. i think wget doesn't connects if the cert is invalid. i ...
7
votes
2answers
419 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
2
votes
2answers
6k views

How to make wget save the web page contents to a file?

I'm trying to use wget to save the text of a web page. I run: wget "http://www.finance.yahoo.com/q/op?s=GOOG" > goog.txt to try and save the web page to goog.txt, but instead wget tells me: ...
1
vote
2answers
3k views

Identifying open and closed server IP Address/ports

I use a fairly simple WGET command to verify if the IP Address and port are open or closed for a particular server: wget -a /tmp/ports.log -t1 -T1 10.178.30.45:443 The issue I have is that there are ...
3
votes
1answer
345 views

How to tell wget where to fetch files passed via '--input-file'

I have a list of filenames (not urls) in a file, and I want to download them using --input-file option. How do I specify the url where those files will be fetched from?
1
vote
3answers
4k views

Need to execute javascript as a cron job… possible?

I need to load a page once per hour and execute the javascript on that page. I currently load the page manually with Firefox, but I would like to automate this by running a cron job on the server, ...
1
vote
1answer
334 views

wget not able to log into ftp

I am trying to use wget for downloading files from a ftp repository. The FTP repository has a login and password. I have to go through a proxy server which does not have a login and password. When I ...
2
votes
2answers
496 views

Configure wget to not clobber good data with empty data

I currently have a setup where I have an emergency server which on a scheduled basis grabs files from my main webserver using wget. If all goes well I want wget to clobber the old cached file. ...
-1
votes
3answers
657 views

How can i “wget” the content of this site

If I run: wget 'http://beforeitsnews.com/stories/in/0000000000000013' -O - I can't see the news...why? How can i wget or curl this site, so that i can see the news on it? I can't even browse it ...
4
votes
2answers
4k views

How to work around missing 'last-modified' headers?

I'm running wget like this: wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main I get a bunch of these messages: Last-modified header missing -- ...
9
votes
1answer
13k views

How to wget a file to a remote machine over SSH?

I'd like to basically pipe a wget command to a file on a remote server over SSH. How can I do this? I know I could simply ssh into the server and have it download the file, but I'd much rather use the ...
0
votes
2answers
904 views

Why curl -O -C on mac fails to download

The following script works on Ubuntu, but not on Mac. Why? How to edit it so that it would work there too? #!/bin/sh v=1.1 test_file="Test10.java" jar_file="dp4j-$v-jar-with-dependencies.jar" curl ...
1
vote
4answers
1k views

Wget command that returns [a list of] all webpages on a webserver

Let's say we have a website www.example.com with the following pages: www.example.com/page1 www.example.com/unknown www.example.com/unknown Is there a Wget command that would produce the following ...
2
votes
1answer
913 views

Download and install latest deb package from github via terminal

I would like to download and install the latest .deb-package from github (https://github.com/elbersb/otr-verwaltung/downloads to be exact). How can I download the latest package (e.g. ...
3
votes
2answers
938 views

Convert links of downloaded website

I downloaded a website using: wget -c --mirror -p http://www.somewebsite.com for offline viewing and I just remembered that I forgot the --convert-links option! They are all on my hard drive right ...
3
votes
3answers
4k views

Why am I failing to mirror a web site (using wget)?

I have tried using wget --mirror http://tshepang.net/, but it only retrieves one page, "tshepang.net/index.html". Is this a bug in wget? Here's the output, from using the --debug option: DEBUG ...
1
vote
1answer
339 views

How to do a recursive download with gwget (wget frontend)?

I usually run wget with the -np -r options, but it doesn't include a preview of the queue size, so I'm checking out the wget frontend gwget (on debian). I can't get the recursive option to work; does ...
3
votes
2answers
957 views

Difference between wget versions

I have problem with wget behavior for 1.10 and later versions. I am using wget to download a report from my app. version 1.10.2: >wget-1.10.2.exe --http-user=trader --http-passwd=trader ...