wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

0
votes
1answer
650 views

bash cannot find mvn command after install

I tried to install Maven on CentOS 7, but afterwards typing mvn -version in the terminal results in bash: mvn: command not found.... How can I fix this so that I can call maven from the CentOS 7 ...
0
votes
2answers
1k views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
2
votes
0answers
269 views

wget resume download from dropbox gives can't write error

I want to resume download from dropbox using wget. I was downloading a file wget -O SQLyog.exe "https://www.dropbox.com/s/dfv2z0le0c2bo7z/SQLyog-9.3.3-0Trial.exe?dl=1" .......... Length: 15775640 ...
8
votes
3answers
4k views

downloading files using wget

I am trying to download an http link from here. The URL is http://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE48191&format=file When I use wget ...
2
votes
1answer
85 views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
-1
votes
0answers
12 views

How to scrape a website with collapsed boxes in order

I want to get certain parts of a website and my current script is: echo https://www... | wget -O- -i- | hxnormalize -l 240 -x $1 2>/dev/null | hxnormalize -x | hxselect -i "div.margin-left" | lynx ...
1
vote
1answer
25 views

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably simply and from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like ...
0
votes
1answer
62 views

Re-download Corrupted Part of ISO

I have download an ISO file from one of the public FTP server. It took a lot of time to download the file because of my Internet Connection Issues. I used the wget command to download the ISO in ...
1
vote
1answer
161 views

How do i WGET an image from a webaddress and store it on a remote ftp location as a cronjob on dd-wrt

From a DD-WRT router I want to create a cron-job that wget's an image from a remote location. This image is to be passed on to a remote FTP server for storage. The process is to be repeated every ...
3
votes
3answers
9k views

“Short write” error message with wget

I got the following error when trying to pull down a 300 MB ISO. It appears to pull down 25 MB then stop. Any idea what's going on here? ~ # wget ...
1
vote
1answer
17 views

wget a url from my public outdoor wx cam

I am trying to grab an image from my weathercam using wget and cron. For now I just need to see how to get an image downloaded and timestamp the filename. ...
10
votes
5answers
14k views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget ...
1
vote
1answer
303 views

“wget --restrict-file-names=windows” seems to fail converting links for NTFS

I've downloaded a website from the net using the wget command-line tool with --restrict-file-names=windows so that once the download is complete I could copy the downloaded files to a mounted Windows ...
1
vote
2answers
35 views

How do I get size of (deb) file from download link/URL?

I use --print-uris of apt-get to get url (download-link) of debs I want to download and install on my Debian (Trisquel) GNU/Linux. --print-uris Instead of fetching the files to install ...
1
vote
0answers
39 views

How to download thousands of files quickly?

I need to download thousands of files (approximately 2-12 MB each) - Currently I am using the following command wget --reject "index.html*" -A *.gz -N -r -l1 -nd --no-check-certificate --user ...
1
vote
4answers
300 views

Constructing URLs with a list of words and downloading from the URLs with command line

Suppose I have a list of words in a text file e.g: abstinence ; abuse ; academic ; accommodate ; accommodation ; accommodationist ; accomplice ; accomplish ; accord ; account ; accountable ; ...
1
vote
2answers
32 views

Put two Cron jobs into crontab using wget

I am trying to put these two Cron Jobs: 0 3 * * * ! sudo -u asterisk /var/lib/asterisk/bin/module_admin --repos extended,standard,unsupported upgradeall 30 3 * * * ! sudo -u asterisk ...
16
votes
6answers
31k views

How to download a file through an SSH server?

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C, The issue is, it is very slow to download a file direct from A, so I need download the ...
1
vote
0answers
22 views

Download a webpage by putting everything it needs in a dedicated directory?

To download a webpage ( its html file and other files needed by it), wget seems to mirror the directory structure of the files of the webpage on the website server When in a web browser (e.g. Firefox ...
5
votes
2answers
223 views

Download a list of webpages in a file and rename them to avoid overwriting?

$ wget -r -np -nH --cut-dirs=2 -p -k "http://www.example.com/1" -O test.html Cannot specify both -k and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details. ...
3
votes
2answers
4k views

Format of cookies when using wget?

What's the format of the "natescapes" cookies.txt when I need to mirror a website that requires login. I use some chrome extension that returns cookies in that format, I save them in cookies.txt, ...
1
vote
2answers
30 views

Equal command between wget and cURL (with options)?

I usually use wget to download remote file, the command I use: wget --timeout=5 --tries=1 --referer=http://www.example.com 'http://www.example.com/abcdefg123.jpg' -O image.jpg As you can see, there ...
4
votes
2answers
867 views

Changing default permissions for wget?

I'm using a PHP script to call shell_exec and execute wget to download some files to /var/www/dir/. (Internal tool, so security isn't much of an issue) The directory has 777 permissions. But when I ...
4
votes
4answers
13k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
12
votes
2answers
13k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through mam wget but the got lost in there, and ...
2
votes
2answers
278 views

How to use wget to download certain files from a certain directory when the directory itself does not have an index.html?

There are a number of questions similar to this on StackExchange but none address this issue. More specifically I want to download all the pdf files in the 2007 directory at ...
0
votes
1answer
43 views

Whats the best way to copy a website onto your harddrive?

I like the website www.globalresearch.ca and I have made copies of it. This command works for me wget -r -l14 -t2 -T60 -E -k --no-check-certificate --restrict-file-names="windows,nocontrol" ...
2
votes
1answer
105 views

Wget: convert-links and avoid redownloading already fetched files?

I am downloading data spread among multiple files that don't change once published. Therefore --timestamping is not good enough, because it is constantly checking whatever resource is changed, which ...
1
vote
2answers
5k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
2
votes
2answers
135 views

Why wget cannot download page as root?

This is odd, I try to download a html file using wget wget --user=xxx --password=yyy http://192.168.1.1/ When I execute it as myself, it works, when I execute it as root it fails with message: ...
4
votes
5answers
216 views

Download HTTPS website available only through username and password with wget?

I'm trying to recursively download website which is normally available only when you login. I have valid username and password, but the problem is that I need to login through web interface, so using ...
3
votes
1answer
19 views

wget from ftp: supress default page creation

I'm trying to download subfolder from ftp. There are only data files. I'm using: wget -nH -np --cut-dirs=5 -r http://sss.com/1/2/3/4/5 Unfortunately wget creates one additional file: index.html ...
-1
votes
1answer
48 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
32
votes
3answers
9k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
2
votes
1answer
397 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
2
votes
1answer
60 views

wget on the fly to extract specific info from file without downloading the entire file

I have some files (which are in .bam format(binary)) to download from a location. I can download them using wget and then want to extract some specific information from them only. For that I use ...
0
votes
0answers
101 views

Redirect with axel in linux

I am trying to download a file from googlevideo. This download link may expire in a few minutes but I will post everything I have. When I use axel with the following command, I get $ axel -n 10 ...
0
votes
2answers
2k views

How to disable gzip compression with wget?

There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and ...
5
votes
3answers
599 views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
0
votes
1answer
89 views

wget: force no default certificates

I'm having trouble with verification of custom server SSL certificates and wget. wget -O- --ca-certificate=myservercert.pem https://www.google.com This should fail but does not, as wget does ...
1
vote
1answer
36 views

Using date -1day with wget

I want to use wget with a cron to pull a file everyday however I want it to pull the previous days file from the remote server and the file is controlled by date variables in the URL as such: ...
2
votes
1answer
54 views

Using wget, how to download to a specific location, without creating folders, and always overwrite original files

How do I use wget to download files to a specific location, without creating directories, and overwrite the original every time. I've tried using the -r -P and nc options in combination but this ...
2
votes
1answer
35 views

How to download a lot of documents on the terminal from a server?

I have to write a bash script (.sh) with less than 300 characters which downloads all of the genomes 22 in this server ftp://ftp.ncbi.nih.gov/genomes/Homo_sapiens/ (the ones with fa.gz extention) and ...
2
votes
1answer
65 views

Download a file to a server?

After ssh to a server, I want to download two files in https://iain.cx/src/ktmux/#download to the server. So for the first file ktmux_helper, I right click to find its link on my local computer, and ...
2
votes
2answers
6k views

wget to get all the files in a directory only returns index.html

I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes ...
0
votes
2answers
108 views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
4
votes
2answers
2k views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
2
votes
1answer
37 views

Multiple user credentials in wget

Using wget, is it possible to supply more than one user/password pair when mirroring a website that has multiple password-protected areas with different credentials? I would like to mirror an entire ...
5
votes
2answers
391 views

Bash Script - Wget Graphical Download Progress

I'm trying to port this code from wget + zenity to wget + kdialog. The code with zenity: #!/bin/bash # Start wget | zenity # Note the & at the end of the pipe, this allows the script to continue ...
7
votes
2answers
7k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...