wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

1
vote
1answer
125 views

wget duplicate files

I'm executing the command: wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com ...
3
votes
2answers
294 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
3
votes
2answers
698 views

wget to get list of files

im using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://site.com/00001.csv etc, is there a way to terminate/end the script when I hit ...
1
vote
1answer
438 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
4
votes
3answers
362 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
3
votes
2answers
4k views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
0
votes
1answer
171 views

Custom command wget without response

I have an arduino with an ethernet shield that is controlling a relay, which control the lights. I usually just use the phone to send a http get request from a widget in tasker, but I would like to be ...
-4
votes
2answers
664 views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
1
vote
1answer
50 views

Auth via wget works, but cannot navigate inside webpage after it

I can get a cookie, ok: wget -q --tries=5 --retry-connrefused --timeout=29 -U 'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0' --no-check-certificate --keep-session-cookies ...
6
votes
4answers
595 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
1
vote
1answer
1k views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
0
votes
0answers
85 views

How To Install PHP on CentOS by download packages?

I'm using CentOS 6.5 and I need to install PHP 4.3.11 i can't use "yum install php" because it'll install the latest version of PHP. So i use these command: wget ...
2
votes
3answers
2k views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
3
votes
2answers
770 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
1
vote
1answer
2k views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
2
votes
1answer
270 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
2k views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
124 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
1
vote
2answers
187 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
1k views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
471 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...
3
votes
1answer
903 views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
0
votes
2answers
158 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
4
votes
1answer
305 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
2
votes
2answers
395 views

Is it possible to execute the result of wget as a command?

I have 10 VPS machines and all of them have to run this one script that I need to change often, but I don't want to change it at 10 places, every time I need to make a change. I put the script as a ...
2
votes
1answer
2k views

How to uninstall BOOST 1.49 and install BOOST 1.54 in ubuntu?

I have a boost 1.49 in my ubuntu system. And I am trying to uninstall boost 1.49 and completely remove it from my system. And after that install the Boost 1.54.0 I did like this to uninstall 1.49 ...
2
votes
1answer
101 views

How to use diff while doing wget?

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added. Is there any ...
3
votes
1answer
163 views

Wget download file ( do not overwrite ) auto update script

I want to write auto update script for my embedded device, which can check and download newer version of my program and extract the files on the device. The download center is hosted on remote web ...
1
vote
2answers
2k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
1
vote
0answers
132 views

Using wget, download a website translated by google translate

I want to download(recursively) http://e-maxx.ru/algo , after it has been translated to english using Google translate.
2
votes
1answer
327 views

wget downloaded file names with absolute path for use with apt-get

I'm looking for a way of updating a machine that is offline. For that purpose I found this explanation using --print-uris from apt-get. I have successfully installed new packages (and dependencies) ...
1
vote
4answers
129 views

play mp3 file as soon as its downloaded

How do you combine these two different commands into one script: $ wget http://mymusic.store.com/musicfoo.mp3 #file gets saved as musicfoo.mp3 $ mplayer musicfoo.mp3 wget attempts to save the file ...
0
votes
2answers
63 views

Not able to download the content of the web site

I'm using Ubuntu 13.04. I opened this website in my browser. The site starts downloading and displaying previews of a book on the same page. I'd like to download the book displayed on the page to my ...
3
votes
1answer
2k views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
2
votes
2answers
2k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
1
vote
3answers
481 views

Download one html page with all needed graphics and linked pdf/zip files

I want to download a website from an URL, to view it locally, more exactly: Download one single html page (no other linked html pages) and everything needed to display it (css, images, etc.) Also ...
3
votes
5answers
1k views

Bash: Fastest way of determining dimensions of image from URL

I'm trying to figure out a really fast method in bash of determining an images dimensions. I know I could wget the image and then use imagemagick to determine the height and width of the image. I'm ...
1
vote
2answers
902 views

Ignore “other” domains when downloading with wget?

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC. I am using the following wget command to get the files I want: wget -I ABC -r -e ...
1
vote
2answers
412 views

Wget fails to download PNG files from Blogger

I am trying to mirror a blogger site so that I can have an exact copy of it on my filesystem to view. I have tried issuing the following command on Linux: wget -r -k -x -e robots=off --wait 1 ...
3
votes
0answers
110 views

WGET - Authenticating root AND all children folders

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below. There's a website that I'm trying to mirror. It is something ...
6
votes
3answers
110 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
0
votes
1answer
59 views

Mirroring site with an offset

I followed answers here http://webapps.stackexchange.com/questions/12311/how-to-download-all-english-books-from-gutenberg which are using wget I want to download using wget from a particular starting ...
2
votes
1answer
745 views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
2
votes
2answers
822 views

wget grabbing empty files that actually exist

I have a script that's meant to download a certain number of files from a remote server. It only needs to do this every 24 hours, since they are JSON sources for a database on my server. The files are ...
1
vote
4answers
712 views

Write output of wget or curl to a custom filename based on the url

For example I have a link http://www.abc.com/123/def/ghi/jkl.mno. I want to download it using wget or curl and get the name of output file as def_ghi_jkl.mno, where the part def_ghi is taken from the ...
2
votes
1answer
919 views

Get all images from website

I'm trying to wget all the images pictures from a website. I thought I knew how to use it, but I guess I don't. The images I'm trying to get come from here. The command I'm using is: wget -prA.png ...
1
vote
1answer
589 views

How to figure out where a link gets redirected

I currently working on a script that updates a file. The file has several versions. The important ones have static links. These get redirected to the actual file which then gets downloaded using wget. ...
7
votes
2answers
3k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
3
votes
1answer
5k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through mam wget but the got lost in there, and ...
0
votes
1answer
157 views

Download documents referenced by a web page with wget

I want to download all the documents pointed to by a web page with wget. When I run wget, it only downloads the webpage, not all the contents that the web page links to. wget --content-disposition ...