wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

1
vote
1answer
77 views

How to burn an iso while downloading it

I'm in a real hurry so I thought there is a way to do that but I don't know how. I thought about piping wget -c cdimage.ubuntu.com/daily-live/current/utopic-desktop-amd64.iso | BurningApp but I don't ...
1
vote
0answers
14 views

Why does wget return a HTTP 403 error to download openSSH pet? [duplicate]

I am trying to learn how I would get OpenSSH using the command line. I found a *.pet link on this webpage: http://puppylinux.org/wikka/OpenSSH which says: Installation Install an appropriate ...
23
votes
3answers
16k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
1
vote
1answer
193 views

SSL no certificate

I'm trying to make a bash script who will check the SSL status of my web site client and return the error. But for every IP, wget https://$ip:$port gives me the following error: ERROR: The ...
0
votes
1answer
133 views

wget changing image name from within page

The image is generated once every n minutes, and I want to get each of these to my local directory. I had no success in getting the png-image using the command: wget -r -l1 --no-parent -A.png ...
0
votes
1answer
2k views

Download all source files for a webpage

I want to download the source files for a webpage which is a database search engine. Using curl I'm only able to download the main html page. I would also like to download all the javascript files, ...
1
vote
0answers
298 views

Using wget to dowload images in facebook conversations

This look a bit harder than usual. I'm a new wget user and i was thinking that maybe it's possible to somehow login into facebook and download images from a specific conversation. I mean, i'd like ...
3
votes
1answer
507 views

make wget refer to a local copy without redundantly downloading files

I want to archive a message board,I do that by using wget with the parameters: --page-requisites, --span-hosts, --convert-links and --no-clobber. The problem is that using --convert-links disables ...
4
votes
2answers
1k views

Tell wget not to spider URL matching a pattern?

I want to test how my site would be behave when being spidered. However, I want to exclude all URLs containing the word "page". I tried: $ wget -r -R "*page*" --spider --no-check-certificate -w 1 ...
3
votes
3answers
681 views

Using wget over ssh with a terminated session

I have a RaspberryPi. I want to download a big file with wget but the problem is that once I log into the Pi and use wget to download the file, it will abort the download when I close the session. ...
12
votes
6answers
16k views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget ...
1
vote
1answer
523 views

Wget or curl a self-signed certificate from server

From my browser, I can browse to a machine in my companies local intranet and (after marking the certificate as trusted) export that certificate to a file. I'm trying to automate this process into a ...
6
votes
2answers
3k views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
0
votes
1answer
202 views

How to get list of urls from a URL recursively with filtering

I want to get list of URLs that contains vimeo.com from a web site recursively by a command , so that I can pipe it to vimeo_downloader.sh. I prefer to use wget, but also I'm happy with other ...
2
votes
1answer
529 views

Trying to get a dynamic page with wget without success

I am using this command to get a page and all its images and javascripts on iTunes with wget. All I want is this page and all its images and scripts. wget -kKErpNF --no-check-certificate ...
1
vote
1answer
3k views

wget a file, logging the output and showing the output on prompt

In a bash script, I need to download a file from the web, for doing it i use wget command. I would log the output of wget command, and at "the same time" having the output prompting on terminal. I ...
2
votes
1answer
418 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
2
votes
2answers
813 views

Resolve filename from a remote URL without downloading a file

I am creating a script, which should download latest version of an application from repository and deploy the app. The main issue: there are several repositories and I need to check, which of them ...
2
votes
1answer
1k views

How to use wget --wait option when using input from a file?

I have a text file with a list of urls I want to download. I want wget to wait before downloading the next url. I'm using cat urls.txt | xargs -n1 -i wget --wait=30 {} However there is no wait ...
1
vote
2answers
187 views

wget - corrupted directories

I am using wget to create an HTML mirror of a php templated site on localhost. I am encountering a problem with some directories being mirrored as extensionless documents. What could be causing this ...
2
votes
4answers
169 views

Find + Grep Recursive Search

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form : /dir1/ /dir1/page1.txt /dir1/page2.txt ... The mirrorring is very ...
1
vote
3answers
1k views

`wait` doesn't wait for `wget --background`

I have gone through several SE posts on the unix wait command and none of the posts resolve my problem. I have a weather forecast model to run. It has first to prepare some folders, then download data ...
5
votes
3answers
2k views

How can I pass argument stored on a variable to WGET

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code: useragent='--user-agent="Mozilla/5.0 (Windows NT ...
2
votes
1answer
350 views

Using wget to target files within a date range

I'm using wget to download some log files from a remote server. Is it possible to use wget to target files using date ranges (based on the last modified file header) ? For example: wget ...
1
vote
1answer
461 views

wget duplicate files

I'm executing the command: wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com ...
4
votes
2answers
927 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
3
votes
2answers
2k views

wget to get list of files

im using wget in a bash script to get a list of files from a website, the files are sequential and in the format http://site.com/00001.csv etc, is there a way to terminate/end the script when I hit ...
3
votes
1answer
916 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
5
votes
3answers
885 views

download only first few bytes of a source page

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a ...
12
votes
2answers
19k views

wget with wildcards in http downloads

I need to download a file using wget, however I don't know exactly what the file name will be. https://foo/bar.1234.tar.gz According to the man page, wget lets you turn off and on globbing when ...
0
votes
1answer
286 views

Custom command wget without response

I have an arduino with an ethernet shield that is controlling a relay, which control the lights. I usually just use the phone to send a http get request from a widget in tasker, but I would like to be ...
-4
votes
2answers
2k views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
1
vote
1answer
76 views

Auth via wget works, but cannot navigate inside webpage after it

I can get a cookie, ok: wget -q --tries=5 --retry-connrefused --timeout=29 -U 'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0' --no-check-certificate --keep-session-cookies ...
6
votes
4answers
980 views

Using sed to add a URL to the start of every line

I have the URL (see below) of a certain web page that lists many different versions of a software package. URL=http://ftp.gnu.org/gnu/wget/ The following one-liner gets me the latest version tar ...
1
vote
1answer
5k views

Why does curl command take urls with parameters as input

I know both wget and curl. Being curious I read a little about their differences. I recently tried to use curl with an url who has parameters and it turns out curl strips down the parameters from ...
4
votes
3answers
5k views

Download multiple URLs at once

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for ...
4
votes
2answers
2k views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this ...
3
votes
1answer
3k views

How to download a file on a webpage that is not a direct link but a javascript click.submit()?

Examples are http://developer.amd.com/tools-and-sdks/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/ The files are listed in a table that need javascript to run on ...
3
votes
1answer
788 views

Resume mirroring via wget

I want to mirror all files for example KDEs to local directory. I used $ wget -m -P ./kde http://alien.slackbook.org/ktown/current/4.12.1/x86_64/kde/ But after a while it broke. To continue I ...
0
votes
2answers
6k views

curl command output in a variable [duplicate]

I am connecting to mySQL database using a shell script. After connecting to the mySQL database, I execute a query. The query gives me 300,000 URLs as the result. For each of the 300,000 URLs I need ...
0
votes
2answers
180 views

Simplest wget example Scientific Linux

I'm on Sci Linux and know nothing about commands. How can I use the wget command? Can someone provide a simple example where it actually downloads something? How do you specify where the download ...
3
votes
2answers
695 views

Why is “wget -r -e robots=off http://ccachicago.org” not acting recursively?

I am trying to recursively download http://ccachicago.org, and am getting exactly one file, the root index.html, downloaded. I've looked at Download recursively with wget and started using the ...
1
vote
1answer
2k views

can't wget rpm oracle on centos linux

Forgotten how I did this last time. Tried the following 2 methods that I thought had worked for me in the past: method #1 $ wget \ ...
2
votes
4answers
675 views

Using wget and grep to download HTML pages and filter by keywords

KEYWORDS=("dolphins" "fish" "dogs" "cats" "iguanas") IFS=$'\n' find . -type f -size +1c ! -exec grep -qF "${KEYWORDS[*]}" {} \; -exec truncate -s 0 -f {} \; If a file is over 1 byte, search for ...
4
votes
2answers
2k views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
0
votes
2answers
1k views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
4
votes
1answer
616 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
2
votes
2answers
1k views

Is it possible to execute the result of wget as a command?

I have 10 VPS machines and all of them have to run this one script that I need to change often, but I don't want to change it at 10 places, every time I need to make a change. I put the script as a ...
3
votes
1answer
5k views

How to uninstall BOOST 1.49 and install BOOST 1.54 in ubuntu?

I have a boost 1.49 in my ubuntu system. And I am trying to uninstall boost 1.49 and completely remove it from my system. And after that install the Boost 1.54.0 I did like this to uninstall 1.49 ...
2
votes
1answer
229 views

How to use diff while doing wget?

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added. Is there any ...