Tagged Questions

wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

1
vote
3answers
585 views

wget batch linux file

I am trying to download images from a manufacturer's site using wget. When I run the command from the shell it works fine. But when I run it from a file I am getting a placeholder file. Here is the ...
7
votes
3answers
4k views

How can I limit downloaded file size in wget?

I want to use wget (from a php script) to download image files, but don't want to download files over a certain size. Can I limit file size with wget? If not, what is a better way?
27
votes
2answers
8k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
6
votes
1answer
4k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...
0
votes
1answer
567 views

Force wget to download only from HTTPS?

wget https://skype.com -O - > /dev/null isn't good. I just want to create a short bash script to check that an HTTPS cert is valid or no. i think wget doesn't connects if the cert is invalid. i ...
7
votes
2answers
417 views

Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks ...
2
votes
2answers
6k views

How to make wget save the web page contents to a file?

I'm trying to use wget to save the text of a web page. I run: wget "http://www.finance.yahoo.com/q/op?s=GOOG" > goog.txt to try and save the web page to goog.txt, but instead wget tells me: ...
1
vote
2answers
3k views

Identifying open and closed server IP Address/ports

I use a fairly simple WGET command to verify if the IP Address and port are open or closed for a particular server: wget -a /tmp/ports.log -t1 -T1 10.178.30.45:443 The issue I have is that there are ...
3
votes
1answer
345 views

How to tell wget where to fetch files passed via '--input-file'

I have a list of filenames (not urls) in a file, and I want to download them using --input-file option. How do I specify the url where those files will be fetched from?
1
vote
3answers
4k views

Need to execute javascript as a cron job… possible?

I need to load a page once per hour and execute the javascript on that page. I currently load the page manually with Firefox, but I would like to automate this by running a cron job on the server, ...
1
vote
1answer
333 views

wget not able to log into ftp

I am trying to use wget for downloading files from a ftp repository. The FTP repository has a login and password. I have to go through a proxy server which does not have a login and password. When I ...
2
votes
2answers
496 views

Configure wget to not clobber good data with empty data

I currently have a setup where I have an emergency server which on a scheduled basis grabs files from my main webserver using wget. If all goes well I want wget to clobber the old cached file. ...
-1
votes
3answers
656 views

How can i “wget” the content of this site

If I run: wget 'http://beforeitsnews.com/stories/in/0000000000000013' -O - I can't see the news...why? How can i wget or curl this site, so that i can see the news on it? I can't even browse it ...
4
votes
2answers
4k views

How to work around missing 'last-modified' headers?

I'm running wget like this: wget --mirror --adjust-extension --convert-links --no-cookies http://tshepang.net -o log-main I get a bunch of these messages: Last-modified header missing -- ...
9
votes
1answer
13k views

How to wget a file to a remote machine over SSH?

I'd like to basically pipe a wget command to a file on a remote server over SSH. How can I do this? I know I could simply ssh into the server and have it download the file, but I'd much rather use the ...
0
votes
2answers
898 views

Why curl -O -C on mac fails to download

The following script works on Ubuntu, but not on Mac. Why? How to edit it so that it would work there too? #!/bin/sh v=1.1 test_file="Test10.java" jar_file="dp4j-$v-jar-with-dependencies.jar" curl ...
1
vote
4answers
1k views

Wget command that returns [a list of] all webpages on a webserver

Let's say we have a website www.example.com with the following pages: www.example.com/page1 www.example.com/unknown www.example.com/unknown Is there a Wget command that would produce the following ...
2
votes
1answer
912 views

Download and install latest deb package from github via terminal

I would like to download and install the latest .deb-package from github (https://github.com/elbersb/otr-verwaltung/downloads to be exact). How can I download the latest package (e.g. ...
3
votes
2answers
938 views

Convert links of downloaded website

I downloaded a website using: wget -c --mirror -p http://www.somewebsite.com for offline viewing and I just remembered that I forgot the --convert-links option! They are all on my hard drive right ...
3
votes
3answers
4k views

Why am I failing to mirror a web site (using wget)?

I have tried using wget --mirror http://tshepang.net/, but it only retrieves one page, "tshepang.net/index.html". Is this a bug in wget? Here's the output, from using the --debug option: DEBUG ...
1
vote
1answer
339 views

How to do a recursive download with gwget (wget frontend)?

I usually run wget with the -np -r options, but it doesn't include a preview of the queue size, so I'm checking out the wget frontend gwget (on debian). I can't get the recursive option to work; does ...
3
votes
2answers
956 views

Difference between wget versions

I have problem with wget behavior for 1.10 and later versions. I am using wget to download a report from my app. version 1.10.2: >wget-1.10.2.exe --http-user=trader --http-passwd=trader ...