wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

1
vote
0answers
9 views

Shell script to grab one or many Wistia videos

What would be the best way to come up with a shell script that grabs one or many Wistia videos? There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: ...
2
votes
1answer
27 views

Can I use wget to download all files recursively, but not their actual content?

Sometimes, I wish to only get the directory structure of a website, but the files themselves are not important. I only want their name. Sort of like a mirror where every entry is just an empty dummy ...
1
vote
1answer
30 views

wget - how to reject string from downloading html pages

I am using the following wget command and it downloads the required files I need except for one thing... wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 ...
0
votes
0answers
35 views

wgetting links that are written in script form - not plain html

I use wget to download all the html pages in an online gallery - in order to grab all the links to the full size images as follows: wget --wait=3 --user-agent=Webtography --no-cookies --timestamping -...
2
votes
1answer
40 views

Using /dev/tcp with proxy

Is it possible to use /dev/tcp with a proxy? I only found wget is possible to use with proxy, but wget doesn't help me in my case.
0
votes
0answers
16 views

wget: avoid using server's last modified time [duplicate]

When I download a file using wget, the last modified time of the newly created file will follow that of the server. As a result, I cannot sort the files according to the date / sequence that I have ...
1
vote
1answer
60 views

Using WGET to pull all file names from an index

I work for a university and I want to pull all the file names of all the PDF catalogs we have and make a text file. These PDFs are all located in an Intranet index. WGET works fine with the intranet ...
1
vote
1answer
29 views

What is the curl equivalent of this wget command?

I'm trying to switch over part of a script from wget to curl. I have this command: wget --output-document=dl.html --quiet --timeout=10 --tries=1 $URL What's the curl equivalent of this?
1
vote
0answers
15 views

How to update an off-line web-site using wget in order to adjust extensions and convert links only?

I would like to update an off-line web-site downloaded using wget GNU command. BUT I want to take in consideration the following conditions : If a file does not already exist locally, update it. ...
1
vote
0answers
35 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
0
votes
3answers
32 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
-1
votes
2answers
80 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
2
votes
1answer
34 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
55 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension --convert-...
0
votes
0answers
29 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
0
votes
1answer
164 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
24 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
3
votes
2answers
90 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of translate....
1
vote
3answers
89 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k http://imgur....
-1
votes
2answers
32 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
0
votes
1answer
42 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
1
vote
1answer
102 views

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
0
votes
2answers
84 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
65 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
63 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
1
vote
1answer
94 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
24 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...
3
votes
2answers
56 views

Can I trust wget's exit status even if it was restarted?

I launched a wget job with the following command: urls=(https://example.com/file1 https://example.com/file2 https://example.com/file3 https://example.com/file4) for url in ${urls[@]}; do ...
0
votes
1answer
24 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url --no-check-...
1
vote
0answers
87 views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
0
votes
1answer
124 views

Why does downloading bash script from pastebin not work? [duplicate]

I'm trying to upload and download a bash script from pastebin, upload was straightforward enough with copy and paste, download didn't go as easy. There are two scripts I'm trying to download, these ...
-1
votes
1answer
81 views

WGET encountering 403 when trying to access filter - nginx [closed]

I'm trying to download some images from an asset folder of a site, that folder happens to be forbidden. I've tried various suggestions online including a referrer flag for google.com and setting the ...
1
vote
0answers
59 views

How to use wget to process a html file before downloading a file [closed]

I'm trying to write a bash script to download a certain file. In order to download the file, you need to be logged in, first to access he file, secondly because the file is generated with you ...
0
votes
1answer
40 views

Skip series in batch download with wget in a bash script

How can i skip a folder in a bash script using wget to batch download files, if the last file checked does not exist? Here is the sample code: #!/bin/bash # Script to download Reports @ brazil.crl....
0
votes
1answer
77 views

wget mirror site + resources from other domain

I'm trying to download a website so it can be browsed offline, so I want the html, images, css, javascript, etc. The problem is that the images are not all hosted on the same domain as the website, ...
5
votes
1answer
104 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called mirror.de/�%...
3
votes
1answer
101 views

using wget to download all audio files (over 100,000 pages on wikia)

I am trying to download all audio files in Wookiepedia, the Star Wars wiki. My first thought is something like this wget -r -A -nd .mp3 .ogg http://starwars.wikia.com/wiki/ This should download ...
1
vote
2answers
70 views

downloading images from webpage with different links

I am trying to download all the images from a webpage. The images are included as follows: <img src="https://alwaysSamePart.com/SomeRandomLinkThatVaries.jpg"> How should I parametrize my wget ...
3
votes
1answer
90 views

wget: Follow custom URL attributes

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a ...
-2
votes
2answers
60 views

How to use this script [closed]

#!\bin\bash for i in `seq 1 1000`; do wget -q -O quotes http://www.quotedb.com/quote/quote.php?action=random_quote; cat quotes>>final.txt; done I got this script from one website but I don't ...
0
votes
0answers
57 views

wget changes downloaded file size

I'm using wget to download Excel file with xlsx extension. The thing is that when I want to deal with the file using openpyxl, I get the following error: "BadZipfile: File is not a zip file" But when ...
1
vote
1answer
55 views

Linux server able to wget but not able to ping

A proxy server was added using export http://link:port/. I try to ping google.com and there is no response, the DNS is able to resolve an IP. wget of google.com works. routes: Destination ...
2
votes
1answer
26 views

Tools for parsing automatically-generated HTTP indexes?

Web servers commonly generate directory indexes listing the files within a directory and additional metadata. There is variation in the content and formatting of these pages, but generally have a ...
2
votes
0answers
60 views

Problem using wget to create static archive of website

I have been using the wget command to create static archives of a number of websites that I no longer want to keep updating. This command has worked well for a number of sites: wget --recursive --no-...
-3
votes
1answer
48 views

Why wget is not willing to download recursively?

The command $ wget -r http://www.comp.brad.ac.uk/research/GIP/tutorials/index.html only downloads index.html and robots.txt for me, even though there are links in it to further pages in the same ...
1
vote
1answer
41 views

wget for recursive download of dxf files

There is a web server with many dxf type files which can only be downloaded one at a time. I thought it should be a good idea to use wget to download all of those files at the same time. Unfortunately,...
3
votes
2answers
512 views

How to monitor WGET download progress after closing SSH session

I have started downloading my ISO's etc directly to my fileserver using wget. After I close the ssh session, how can I check back on the download process? Scenario: I start the download, then shut ...
2
votes
0answers
114 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
0
votes
1answer
45 views

Letting HTML file act as URL with wget

If you want to get straight to the question, please skip all the way down to the bolded and italicized question past the code. Intro: I am experiencing problems with wget and html files. I am using ...
0
votes
1answer
51 views

How to extract all links from a given url in a decent time

I'm trying to extract the links from a specified url and redirect them to a file. At first I tried using --spider , this is my code: wget --spider --force-html -m http://file/path 2>&1 | grep ...