wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

1
vote
1answer
18 views

What is the curl equivalent of this wget command?

I'm trying to switch over part of a script from wget to curl. I have this command: wget --output-document=dl.html --quiet --timeout=10 --tries=1 $URL What's the curl equivalent of this?
1
vote
0answers
13 views

How to update an off-line web-site using wget in order to adjust extensions and convert links only?

I would like to update an off-line web-site downloaded using wget GNU command. BUT I want to take in consideration the following conditions : If a file does not already exist locally, update it. ...
1
vote
0answers
24 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
0
votes
3answers
28 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
-1
votes
2answers
73 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
2
votes
1answer
29 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
53 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension ...
0
votes
0answers
13 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
0
votes
1answer
96 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
22 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
3
votes
2answers
72 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of ...
-1
votes
0answers
7 views

Using wget to download all images from deadendthrills.com

I used to download all images from deadendthrills.com using the following command in order to use them in my wallpaper changer: wget -r -np --reject "index.html*" -e robots=off ...
1
vote
3answers
78 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k ...
-1
votes
2answers
27 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
0
votes
1answer
39 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
0
votes
1answer
54 views
+50

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
0
votes
2answers
68 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
48 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
58 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
1
vote
1answer
66 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
21 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...
3
votes
2answers
55 views

Can I trust wget's exit status even if it was restarted?

I launched a wget job with the following command: urls=(https://example.com/file1 https://example.com/file2 https://example.com/file3 https://example.com/file4) for url in ${urls[@]}; do ...
0
votes
1answer
18 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url ...
1
vote
0answers
68 views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
0
votes
1answer
94 views

Why does downloading bash script from pastebin not work? [duplicate]

I'm trying to upload and download a bash script from pastebin, upload was straightforward enough with copy and paste, download didn't go as easy. There are two scripts I'm trying to download, these ...
-1
votes
1answer
57 views

WGET encountering 403 when trying to access filter - nginx [closed]

I'm trying to download some images from an asset folder of a site, that folder happens to be forbidden. I've tried various suggestions online including a referrer flag for google.com and setting the ...
1
vote
0answers
56 views

How to use wget to process a html file before downloading a file [closed]

I'm trying to write a bash script to download a certain file. In order to download the file, you need to be logged in, first to access he file, secondly because the file is generated with you ...
0
votes
1answer
39 views

Skip series in batch download with wget in a bash script

How can i skip a folder in a bash script using wget to batch download files, if the last file checked does not exist? Here is the sample code: #!/bin/bash # Script to download Reports @ ...
0
votes
1answer
53 views

wget mirror site + resources from other domain

I'm trying to download a website so it can be browsed offline, so I want the html, images, css, javascript, etc. The problem is that the images are not all hosted on the same domain as the website, ...
5
votes
1answer
98 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called ...
3
votes
1answer
89 views

using wget to download all audio files (over 100,000 pages on wikia)

I am trying to download all audio files in Wookiepedia, the Star Wars wiki. My first thought is something like this wget -r -A -nd .mp3 .ogg http://starwars.wikia.com/wiki/ This should download ...
1
vote
2answers
59 views

downloading images from webpage with different links

I am trying to download all the images from a webpage. The images are included as follows: <img src="https://alwaysSamePart.com/SomeRandomLinkThatVaries.jpg"> How should I parametrize my wget ...
3
votes
1answer
78 views

wget: Follow custom URL attributes

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a ...
-2
votes
2answers
57 views

How to use this script [closed]

#!\bin\bash for i in `seq 1 1000`; do wget -q -O quotes http://www.quotedb.com/quote/quote.php?action=random_quote; cat quotes>>final.txt; done I got this script from one website but I don't ...
0
votes
0answers
54 views

wget changes downloaded file size

I'm using wget to download Excel file with xlsx extension. The thing is that when I want to deal with the file using openpyxl, I get the following error: "BadZipfile: File is not a zip file" But when ...
1
vote
1answer
53 views

Linux server able to wget but not able to ping

A proxy server was added using export http://link:port/. I try to ping google.com and there is no response, the DNS is able to resolve an IP. wget of google.com works. routes: Destination ...
2
votes
1answer
25 views

Tools for parsing automatically-generated HTTP indexes?

Web servers commonly generate directory indexes listing the files within a directory and additional metadata. There is variation in the content and formatting of these pages, but generally have a ...
2
votes
0answers
54 views

Problem using wget to create static archive of website

I have been using the wget command to create static archives of a number of websites that I no longer want to keep updating. This command has worked well for a number of sites: wget --recursive ...
-3
votes
1answer
42 views

Why wget is not willing to download recursively?

The command $ wget -r http://www.comp.brad.ac.uk/research/GIP/tutorials/index.html only downloads index.html and robots.txt for me, even though there are links in it to further pages in the same ...
1
vote
1answer
38 views

wget for recursive download of dxf files

There is a web server with many dxf type files which can only be downloaded one at a time. I thought it should be a good idea to use wget to download all of those files at the same time. ...
3
votes
2answers
416 views

How to monitor WGET download progress after closing SSH session

I have started downloading my ISO's etc directly to my fileserver using wget. After I close the ssh session, how can I check back on the download process? Scenario: I start the download, then shut ...
2
votes
0answers
96 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent ...
0
votes
1answer
43 views

Letting HTML file act as URL with wget

If you want to get straight to the question, please skip all the way down to the bolded and italicized question past the code. Intro: I am experiencing problems with wget and html files. I am using ...
0
votes
1answer
46 views

How to extract all links from a given url in a decent time

I'm trying to extract the links from a specified url and redirect them to a file. At first I tried using --spider , this is my code: wget --spider --force-html -m http://file/path 2>&1 | grep ...
0
votes
0answers
998 views

wget failing with 404 File not found

I have two instances of jboss running on the same server with two different ports. ( 8280 & 10580). I am facing issue while downloading a file using wget on Port 10580 , but the file is getting ...
1
vote
0answers
45 views

How to get all files of a certain type from a virtual web path?

I try to use wget to download certain files from a website. But probably the virtual locations of the files do not represent their actual location. For example, if the link to the file is ...
2
votes
1answer
240 views

What is a flag to download hidden files by wget?

I downloaded a large directory (with subdirectories) from http by wget but hidden files (.filename) were left out. How can I download it? EDITED for @dhag I used command: wget --mirror --no-parent ...
0
votes
1answer
28 views

Using a list to exclude results from wget command download

Say I have a file called download.txt. In this list is the following: File1.file File2.file File3.file Is there a way to use wget command to use that list to download all files from a directory ...
0
votes
1answer
26 views

URL dependency in make

I need a dependency in a Makefile that fires if a URL has been updated since the file that depends on it. The situation is a bit complicated: URL 1 points to a download page. I want the rule to be ...
1
vote
1answer
29 views

Calling wget from make based on a pattern

I'm using make to grab a set of files that are updated daily on a Web site. What I'd like to do is place the files into a folder named with the date, and then use make to do other processing on the ...