wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

2
votes
3answers
868 views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
3
votes
1answer
473 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
1
vote
1answer
1k views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
2
votes
1answer
179 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
2k views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
119 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
1
vote
2answers
119 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
1k views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
412 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...
3
votes
1answer
526 views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
0
votes
2answers
105 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
4
votes
1answer
236 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
2
votes
2answers
220 views

Is it possible to execute the result of wget as a command?

I have 10 VPS machines and all of them have to run this one script that I need to change often, but I don't want to change it at 10 places, every time I need to make a change. I put the script as a ...
2
votes
1answer
2k views

How to uninstall BOOST 1.49 and install BOOST 1.54 in ubuntu?

I have a boost 1.49 in my ubuntu system. And I am trying to uninstall boost 1.49 and completely remove it from my system. And after that install the Boost 1.54.0 I did like this to uninstall 1.49 ...
2
votes
1answer
85 views

How to use diff while doing wget?

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added. Is there any ...
3
votes
1answer
152 views

Wget download file ( do not overwrite ) auto update script

I want to write auto update script for my embedded device, which can check and download newer version of my program and extract the files on the device. The download center is hosted on remote web ...
1
vote
2answers
888 views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
1
vote
0answers
118 views

Using wget, download a website translated by google translate

I want to download(recursively) http://e-maxx.ru/algo , after it has been translated to english using Google translate.
2
votes
1answer
285 views

wget downloaded file names with absolute path for use with apt-get

I'm looking for a way of updating a machine that is offline. For that purpose I found this explanation using --print-uris from apt-get. I have successfully installed new packages (and dependencies) ...
1
vote
4answers
114 views

play mp3 file as soon as its downloaded

How do you combine these two different commands into one script: $ wget http://mymusic.store.com/musicfoo.mp3 #file gets saved as musicfoo.mp3 $ mplayer musicfoo.mp3 wget attempts to save the file ...
0
votes
2answers
57 views

Not able to download the content of the web site

I'm using Ubuntu 13.04. I opened this website in my browser. The site starts downloading and displaying previews of a book on the same page. I'd like to download the book displayed on the page to my ...
3
votes
1answer
478 views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
2
votes
2answers
2k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
1
vote
3answers
298 views

Download one html page with all needed graphics and linked pdf/zip files

I want to download a website from an URL, to view it locally, more exactly: Download one single html page (no other linked html pages) and everything needed to display it (css, images, etc.) Also ...
2
votes
5answers
827 views

Bash: Fastest way of determining dimensions of image from URL

I'm trying to figure out a really fast method in bash of determining an images dimensions. I know I could wget the image and then use imagemagick to determine the height and width of the image. I'm ...
1
vote
2answers
649 views

Ignore “other” domains when downloading with wget?

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC. I am using the following wget command to get the files I want: wget -I ABC -r -e ...
1
vote
2answers
328 views

Wget fails to download PNG files from Blogger

I am trying to mirror a blogger site so that I can have an exact copy of it on my filesystem to view. I have tried issuing the following command on Linux: wget -r -k -x -e robots=off --wait 1 ...
3
votes
0answers
96 views

WGET - Authenticating root AND all children folders

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below. There's a website that I'm trying to mirror. It is something ...
6
votes
3answers
108 views

generate links of pages with incrementing variable

consider the links http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 ...
0
votes
1answer
56 views

Mirroring site with an offset

I followed answers here http://webapps.stackexchange.com/questions/12311/how-to-download-all-english-books-from-gutenberg which are using wget I want to download using wget from a particular starting ...
2
votes
1answer
635 views

download pdf files from using curl

How to download hundereds of .pdf files from http://www.ncbi.nlm.nih.gov/pmc/articles using a loop, for example for the following document ids: PMC3386155 PMC3625956 PMC3477654 PMC3531051 ...
2
votes
2answers
651 views

wget grabbing empty files that actually exist

I have a script that's meant to download a certain number of files from a remote server. It only needs to do this every 24 hours, since they are JSON sources for a database on my server. The files are ...
1
vote
4answers
498 views

Write output of wget or curl to a custom filename based on the url

For example I have a link http://www.abc.com/123/def/ghi/jkl.mno. I want to download it using wget or curl and get the name of output file as def_ghi_jkl.mno, where the part def_ghi is taken from the ...
2
votes
1answer
733 views

Get all images from website

I'm trying to wget all the images pictures from a website. I thought I knew how to use it, but I guess I don't. The images I'm trying to get come from here. The command I'm using is: wget -prA.png ...
1
vote
1answer
405 views

How to figure out where a link gets redirected

I currently working on a script that updates a file. The file has several versions. The important ones have static links. These get redirected to the actual file which then gets downloaded using wget. ...
7
votes
2answers
3k views

Alternatives to wget

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds. I have lost my password to that server and I need to grab everything that is stored there. I can go page by ...
3
votes
1answer
3k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through mam wget but the got lost in there, and ...
0
votes
1answer
128 views

Download documents referenced by a web page with wget

I want to download all the documents pointed to by a web page with wget. When I run wget, it only downloads the webpage, not all the contents that the web page links to. wget --content-disposition ...
5
votes
2answers
887 views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
2
votes
2answers
322 views

Problem downloading Java JDK with WGET on Ubuntu

I'm attempting to install the Java JDK through the command line using WGET on ubuntu (no desktop). Here is the picture I took. The file that was downloaded was not the tar ball of the JDK: How can I ...
0
votes
2answers
451 views

how do I get the response ouput from the url .download it in file,not download the code

My file like this with some number or id 404000847307897 404000857827460 404001044194601 404000835511595 404000338418501 404000338654003 404000290144612 404000344175621 404000290144612 ...
3
votes
1answer
720 views

How to download all html files from a URL's directory with wget?

I'd like to use wget to download a tutorial in a website, but this tutorial is in a specific directory. Suppose the URL is the tutorial main page http://www.abc.com/aaa/bbb/ccc/index.html. But the ...
1
vote
2answers
69 views

How to download several urls into one file

I have URLs like this. http://203.90.117.109/wds_cdma/wds.aspx?min=404000290205346 http://203.90.117.109/wds_cdma/wds.aspx?min=404000832812637 ...
-3
votes
2answers
423 views

What is the meaning of wget -O -

I am trying to install dropbox onto my Debian machine, and I saw the instruction cd ~ && wget -O - "some website leading to the download of a tar file" | tar xzf - But what I did was ...
3
votes
3answers
1k views

Using wget to get file names from a text file

I have found a website that hosts a few files that I'm after, there's too many to download them all individually. The filenames take a fairly standard and reproduceable form i.e. 1_a, 1_b, 1_c etc Is ...
0
votes
1answer
251 views

403 http code appears when I go to a certain website

I wrote a PHP script to a particular site using a home computer for debugging. Script worked properly, when I tested it on a home computer. The script has stopped working when I deploy the script on ...
1
vote
1answer
556 views

How can I use wget to download large files? [closed]

I want to get the ISO for a particular version of Linux, say 'Mint 14 “Nadia” KDE '. I tried using wget in the terminal to download the ISO of this distro: $ wget ...
2
votes
1answer
430 views

Why is wget skipping files while trying to mirror a site

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html wget --recursive --no-clobber --page-requisites --html-extension --convert-links ...
3
votes
2answers
158 views

Wget with different filename, but same extension

I need wget to save to a different folder, different filename, and same extension, but I do not have the filename before downloading. For example, let's say I have: ...
-1
votes
1answer
413 views

Downloading the Google search results page using Linux commands

When I try using wget command, the page which gets downloaded contains only the lefthand side part. I want the right-hand side part of the search-results page, which is called Knowledge Graph. Can ...