wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

0
votes
1answer
38 views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
6
votes
2answers
266 views

downloading files using wget

I am trying to download an http link from here. The URL is http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file When I use wget ...
5
votes
3answers
313 views

How you stop 'wget' after it gets a 404?

If you use brace expansion with wget, you can fetch sequentially-numbered images with ease: $ wget 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg' It fetches the first 10 ...
1
vote
2answers
52 views

wget to get all the files in a directory only returns index.html

I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes ...
1
vote
2answers
32 views

How to get wget to use the direct URL file name as output file name for a shortened URL

I'm trying to download all the books from a URL list in bulk. The list is in a text file in http://ligman.me/W3TuLr. I'm trying to use wget on linux for this. All these URLs seem to be created using a ...
1
vote
1answer
57 views

How to burn an iso while downloading it

I'm in a real hurry so I thought there is a way to do that but I don't know how. I thought about piping wget -c cdimage.ubuntu.com/daily-live/current/utopic-desktop-amd64.iso | BurningApp but I don't ...
1
vote
0answers
14 views

Why does wget return a HTTP 403 error to download openSSH pet? [duplicate]

I am trying to learn how I would get OpenSSH using the command line. I found a *.pet link on this webpage: http://puppylinux.org/wikka/OpenSSH which says: Installation Install an appropriate ...
15
votes
3answers
696 views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
1
vote
1answer
33 views

SSL no certificate

I'm trying to make a bash script who will check the SSL status of my web site client and return the error. But for every IP, wget https://$ip:$port gives me the following error: ERROR: The ...
0
votes
1answer
49 views

wget changing image name from within page

The image is generated once every n minutes, and I want to get each of these to my local directory. I had no success in getting the png-image using the command: wget -r -l1 --no-parent -A.png ...
0
votes
1answer
45 views

Download all source files for a webpage

I want to download the source files for a webpage which is a database search engine. Using curl I'm only able to download the main html page. I would also like to download all the javascript files, ...
1
vote
0answers
34 views

Using wget to dowload images in facebook conversations

This look a bit harder than usual. I'm a new wget user and i was thinking that maybe it's possible to somehow login into facebook and download images from a specific conversation. I mean, i'd like ...
2
votes
1answer
52 views

make wget refer to a local copy without redundantly downloading files

I want to archive a message board,I do that by using wget with the parameters: --page-requisites, --span-hosts, --convert-links and --no-clobber. The problem is that using --convert-links disables ...
2
votes
2answers
56 views

Tell wget not to spider URL matching a pattern?

I want to test how my site would be behave when being spidered. However, I want to exclude all URLs containing the word "page". I tried: $ wget -r -R "*page*" --spider --no-check-certificate -w 1 ...
3
votes
3answers
85 views

Using wget over ssh with a terminated session

I have a RaspberryPi. I want to download a big file with wget but the problem is that once I log into the Pi and use wget to download the file, it will abort the download when I close the session. ...
1
vote
2answers
257 views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: ‍ wget ...
1
vote
1answer
30 views

Wget or curl a self-signed certificate from server

From my browser, I can browse to a machine in my companies local intranet and (after marking the certificate as trusted) export that certificate to a file. I'm trying to automate this process into a ...
4
votes
2answers
182 views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
0
votes
1answer
59 views

How to get list of urls from a URL recursively with filtering

I want to get list of URLs that contains vimeo.com from a web site recursively by a command , so that I can pipe it to vimeo_downloader.sh. I prefer to use wget, but also I'm happy with other ...
2
votes
1answer
50 views

Trying to get a dynamic page with wget without success

I am using this command to get a page and all its images and javascripts on iTunes with wget. All I want is this page and all its images and scripts. wget -kKErpNF --no-check-certificate ...
1
vote
1answer
189 views

wget a file, logging the output and showing the output on prompt

In a bash script, I need to download a file from the web, for doing it i use wget command. I would log the output of wget command, and at "the same time" having the output prompting on terminal. I ...
1
vote
1answer
108 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
1
vote
2answers
98 views

Resolve filename from a remote URL without downloading a file

I am creating a script, which should download latest version of an application from repository and deploy the app. The main issue: there are several repositories and I need to check, which of them ...
2
votes
1answer
83 views

How to use wget --wait option when using input from a file?

I have a text file with a list of urls I want to download. I want wget to wait before downloading the next url. I'm using cat urls.txt | xargs -n1 -i wget --wait=30 {} However there is no wait ...
1
vote
2answers
76 views

wget - corrupted directories

I am using wget to create an HTML mirror of a php templated site on localhost. I am encountering a problem with some directories being mirrored as extensionless documents. What could be causing this ...
2
votes
4answers
99 views

Find + Grep Recursive Search

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form : /dir1/ /dir1/page1.txt /dir1/page2.txt ... The mirrorring is very ...
1
vote
3answers
195 views

`wait` doesn't wait for `wget --background`

I have gone through several SE posts on the unix wait command and none of the posts resolve my problem. I have a weather forecast model to run. It has first to prepare some folders, then download data ...
4
votes
3answers
241 views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
1
vote
1answer
117 views

Using wget to target files within a date range

I'm using wget to download some log files from a remote server. Is it possible to use wget to target files using date ranges (based on the last modified file header) ? For example: wget ...
1
vote
1answer
70 views

wget duplicate files

I'm executing the command: wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com ...
3
votes
2answers
135 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
2
votes
2answers
191 views

wget to get list of files

im using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://site.com/00001.csv etc, is there a way to terminate/end the script when I hit ...
1
vote
1answer
243 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
4
votes
3answers
195 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
3
votes
2answers
1k views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
0
votes
1answer
89 views

Custom command wget without response

I have an arduino with an ethernet shield that is controlling a relay, which control the lights. I usually just use the phone to send a http get request from a widget in tasker, but I would like to be ...
-4
votes
2answers
156 views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
1
vote
1answer
37 views

Auth via wget works, but cannot navigate inside webpage after it

I can get a cookie, ok: wget -q --tries=5 --retry-connrefused --timeout=29 -U 'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0' --no-check-certificate --keep-session-cookies ...
6
votes
4answers
489 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
1
vote
1answer
340 views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
0
votes
0answers
59 views

How To Install PHP on CentOS by download packages?

I'm using CentOS 6.5 and I need to install PHP 4.3.11 i can't use "yum install php" because it'll install the latest version of PHP. So i use these command: wget ...
2
votes
3answers
647 views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
3
votes
1answer
317 views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
1
vote
1answer
689 views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
2
votes
0answers
123 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
1k views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
110 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
1
vote
2answers
87 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
881 views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
374 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...