wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

1
vote
1answer
192 views

Wget or curl a self-signed certificate from server

From my browser, I can browse to a machine in my companies local intranet and (after marking the certificate as trusted) export that certificate to a file. I'm trying to automate this process into a ...
4
votes
2answers
546 views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
0
votes
1answer
94 views

How to get list of urls from a URL recursively with filtering

I want to get list of URLs that contains vimeo.com from a web site recursively by a command , so that I can pipe it to vimeo_downloader.sh. I prefer to use wget, but also I'm happy with other ...
2
votes
1answer
192 views

Trying to get a dynamic page with wget without success

I am using this command to get a page and all its images and javascripts on iTunes with wget. All I want is this page and all its images and scripts. wget -kKErpNF --no-check-certificate ...
1
vote
1answer
1k views

wget a file, logging the output and showing the output on prompt

In a bash script, I need to download a file from the web, for doing it i use wget command. I would log the output of wget command, and at "the same time" having the output prompting on terminal. I ...
1
vote
1answer
234 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
1
vote
2answers
448 views

Resolve filename from a remote URL without downloading a file

I am creating a script, which should download latest version of an application from repository and deploy the app. The main issue: there are several repositories and I need to check, which of them ...
2
votes
1answer
277 views

How to use wget --wait option when using input from a file?

I have a text file with a list of urls I want to download. I want wget to wait before downloading the next url. I'm using cat urls.txt | xargs -n1 -i wget --wait=30 {} However there is no wait ...
1
vote
2answers
113 views

wget - corrupted directories

I am using wget to create an HTML mirror of a php templated site on localhost. I am encountering a problem with some directories being mirrored as extensionless documents. What could be causing this ...
2
votes
4answers
136 views

Find + Grep Recursive Search

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form : /dir1/ /dir1/page1.txt /dir1/page2.txt ... The mirrorring is very ...
1
vote
3answers
561 views

`wait` doesn't wait for `wget --background`

I have gone through several SE posts on the unix wait command and none of the posts resolve my problem. I have a weather forecast model to run. It has first to prepare some folders, then download data ...
5
votes
3answers
916 views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
2
votes
1answer
222 views

Using wget to target files within a date range

I'm using wget to download some log files from a remote server. Is it possible to use wget to target files using date ranges (based on the last modified file header) ? For example: wget ...
1
vote
1answer
149 views

wget duplicate files

I'm executing the command: wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com ...
3
votes
2answers
362 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
3
votes
2answers
913 views

wget to get list of files

im using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://site.com/00001.csv etc, is there a way to terminate/end the script when I hit ...
1
vote
1answer
520 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
4
votes
3answers
441 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
4
votes
2answers
6k views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
0
votes
1answer
182 views

Custom command wget without response

I have an arduino with an ethernet shield that is controlling a relay, which control the lights. I usually just use the phone to send a http get request from a widget in tasker, but I would like to be ...
-4
votes
2answers
794 views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
1
vote
1answer
53 views

Auth via wget works, but cannot navigate inside webpage after it

I can get a cookie, ok: wget -q --tries=5 --retry-connrefused --timeout=29 -U 'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0' --no-check-certificate --keep-session-cookies ...
6
votes
4answers
646 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
1
vote
1answer
2k views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
0
votes
0answers
90 views

How To Install PHP on CentOS by download packages?

I'm using CentOS 6.5 and I need to install PHP 4.3.11 i can't use "yum install php" because it'll install the latest version of PHP. So i use these command: wget ...
3
votes
3answers
2k views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
3
votes
2answers
947 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
1
vote
1answer
2k views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
2
votes
1answer
329 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
3k views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
130 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
1
vote
2answers
222 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
1k views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
504 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...
3
votes
1answer
1k views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
0
votes
2answers
247 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
4
votes
1answer
346 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
2
votes
2answers
536 views

Is it possible to execute the result of wget as a command?

I have 10 VPS machines and all of them have to run this one script that I need to change often, but I don't want to change it at 10 places, every time I need to make a change. I put the script as a ...
2
votes
1answer
3k views

How to uninstall BOOST 1.49 and install BOOST 1.54 in ubuntu?

I have a boost 1.49 in my ubuntu system. And I am trying to uninstall boost 1.49 and completely remove it from my system. And after that install the Boost 1.54.0 I did like this to uninstall 1.49 ...
2
votes
1answer
115 views

How to use diff while doing wget?

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added. Is there any ...
3
votes
1answer
170 views

Wget download file ( do not overwrite ) auto update script

I want to write auto update script for my embedded device, which can check and download newer version of my program and extract the files on the device. The download center is hosted on remote web ...
1
vote
2answers
2k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
1
vote
0answers
134 views

Using wget, download a website translated by google translate

I want to download(recursively) http://e-maxx.ru/algo , after it has been translated to english using Google translate.
2
votes
1answer
342 views

wget downloaded file names with absolute path for use with apt-get

I'm looking for a way of updating a machine that is offline. For that purpose I found this explanation using --print-uris from apt-get. I have successfully installed new packages (and dependencies) ...
1
vote
4answers
143 views

play mp3 file as soon as its downloaded

How do you combine these two different commands into one script: $ wget http://mymusic.store.com/musicfoo.mp3 #file gets saved as musicfoo.mp3 $ mplayer musicfoo.mp3 wget attempts to save the file ...
0
votes
2answers
68 views

Not able to download the content of the web site

I'm using Ubuntu 13.04. I opened this website in my browser. The site starts downloading and displaying previews of a book on the same page. I'd like to download the book displayed on the page to my ...
4
votes
1answer
4k views

What does 2>&1 in this command mean? [duplicate]

I understand that this command attempts to write to nowhere or a null device but what does 2>&1 mean here? wget -q -O - http://yourwebsite.com/wp-cron.php?doing_wp_cron > /dev/null ...
2
votes
2answers
3k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
2
votes
3answers
572 views

Download one html page with all needed graphics and linked pdf/zip files

I want to download a website from an URL, to view it locally, more exactly: Download one single html page (no other linked html pages) and everything needed to display it (css, images, etc.) Also ...
3
votes
5answers
1k views

Bash: Fastest way of determining dimensions of image from URL

I'm trying to figure out a really fast method in bash of determining an images dimensions. I know I could wget the image and then use imagemagick to determine the height and width of the image. I'm ...