wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

3
votes
1answer
20 views

How to use wget --wait option when using input from a file?

I have a text file with a list of urls I want to download. I want wget to wait before downloading the next url. I'm using cat urls.txt | xargs -n1 -i wget --wait=30 {} However there is no wait ...
1
vote
2answers
50 views

wget - corrupted directories

I am using wget to create an HTML mirror of a php templated site on localhost. I am encountering a problem with some directories being mirrored as extensionless documents. What could be causing this ...
2
votes
4answers
58 views

Find + Grep Recursive Search

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form : /dir1/ /dir1/page1.txt /dir1/page2.txt ... The mirrorring is very ...
1
vote
3answers
59 views

`wait` doesn't wait for `wget --background`

I have gone through several SE posts on the unix wait command and none of the posts resolve my problem. I have a weather forecast model to run. It has first to prepare some folders, then download data ...
4
votes
3answers
100 views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
1
vote
1answer
43 views

Using wget to target files within a date range

I'm using wget to download some log files from a remote server. Is it possible to use wget to target files using date ranges (based on the last modified file header) ? For example: wget ...
1
vote
1answer
38 views

wget duplicate files

I'm executing the command: wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com ...
3
votes
2answers
42 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
2
votes
2answers
102 views

wget to get list of files

im using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://site.com/00001.csv etc, is there a way to terminate/end the script when I hit ...
1
vote
1answer
92 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
4
votes
3answers
87 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
3
votes
2answers
61 views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
0
votes
1answer
45 views

Custom command wget without response

I have an arduino with an ethernet shield that is controlling a relay, which control the lights. I usually just use the phone to send a http get request from a widget in tasker, but I would like to be ...
-4
votes
2answers
44 views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
1
vote
1answer
20 views

Auth via wget works, but cannot navigate inside webpage after it

I can get a cookie, ok: wget -q --tries=5 --retry-connrefused --timeout=29 -U 'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0' --no-check-certificate --keep-session-cookies ...
6
votes
4answers
373 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
1
vote
1answer
58 views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
0
votes
0answers
27 views

How To Install PHP on CentOS by download packages?

I'm using CentOS 6.5 and I need to install PHP 4.3.11 i can't use "yum install php" because it'll install the latest version of PHP. So i use these command: wget ...
2
votes
3answers
167 views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
3
votes
1answer
68 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
1
vote
1answer
107 views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
2
votes
0answers
38 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
196 views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
85 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
1
vote
2answers
44 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
589 views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
268 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...
3
votes
1answer
72 views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
0
votes
2answers
39 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
3
votes
1answer
116 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
2
votes
2answers
89 views

Is it possible to execute the result of wget as a command?

I have 10 VPS machines and all of them have to run this one script that I need to change often, but I don't want to change it at 10 places, every time I need to make a change. I put the script as a ...
1
vote
1answer
703 views

How to uninstall BOOST 1.49 and install BOOST 1.54 in ubuntu?

I have a boost 1.49 in my ubuntu system. And I am trying to uninstall boost 1.49 and completely remove it from my system. And after that install the Boost 1.54.0 I did like this to uninstall 1.49 ...
2
votes
1answer
56 views

How to use diff while doing wget?

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added. Is there any ...
3
votes
1answer
105 views

Wget download file ( do not overwrite ) auto update script

I want to write auto update script for my embedded device, which can check and download newer version of my program and extract the files on the device. The download center is hosted on remote web ...
1
vote
2answers
201 views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
1
vote
0answers
56 views

Using wget, download a website translated by google translate

I want to download(recursively) http://e-maxx.ru/algo , after it has been translated to english using Google translate.
2
votes
1answer
183 views

wget downloaded file names with absolute path for use with apt-get

I'm looking for a way of updating a machine that is offline. For that purpose I found this explanation using --print-uris from apt-get. I have successfully installed new packages (and dependencies) ...
1
vote
4answers
73 views

play mp3 file as soon as its downloaded

How do you combine these two different commands into one script: $ wget http://mymusic.store.com/musicfoo.mp3 #file gets saved as musicfoo.mp3 $ mplayer musicfoo.mp3 wget attempts to save the file ...
0
votes
2answers
41 views

Not able to download the content of the web site

I'm using Ubuntu 13.04. I opened this website in my browser. The site starts downloading and displaying previews of a book on the same page. I'd like to download the book displayed on the page to my ...
3
votes
1answer
148 views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
1
vote
1answer
545 views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
1
vote
3answers
138 views

Download one html page with all needed graphics and linked pdf/zip files

I want to download a website from an URL, to view it locally, more exactly: Download one single html page (no other linked html pages) and everything needed to display it (css, images, etc.) Also ...
2
votes
5answers
520 views

Bash: Fastest way of determining dimensions of image from URL

I'm trying to figure out a really fast method in bash of determining an images dimensions. I know I could wget the image and then use imagemagick to determine the height and width of the image. I'm ...
1
vote
2answers
223 views

Ignore “other” domains when downloading with wget?

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC. I am using the following wget command to get the files I want: wget -I ABC -r -e ...
1
vote
2answers
160 views

Wget fails to download PNG files from Blogger

I am trying to mirror a blogger site so that I can have an exact copy of it on my filesystem to view. I have tried issuing the following command on Linux: wget -r -k -x -e robots=off --wait 1 ...
3
votes
0answers
64 views

WGET - Authenticating root AND all children folders

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below. There's a website that I'm trying to mirror. It is something ...
6
votes
3answers
98 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
0
votes
1answer
48 views

Mirroring site with an offset

I followed answers here http://webapps.stackexchange.com/questions/12311/how-to-download-all-english-books-from-gutenberg which are using wget I want to download using wget from a particular starting ...
2
votes
1answer
346 views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
2
votes
2answers
283 views

wget grabbing empty files that actually exist

I have a script that's meant to download a certain number of files from a remote server. It only needs to do this every 24 hours, since they are JSON sources for a database on my server. The files are ...