wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

0
votes
1answer
14 views

Owncloud does not show files, that are uploaded via SSH

After spending 10 Hours configuring owncloud on Fedora 24 (because of NO documentation and proper tutorial), I finally got OwnCloud up and running perfectly. My owncloud configuration files are in /...
0
votes
1answer
37 views

download a torrent file from kickass torrent from linux cli

this one has been bugging me for a while and I cannot find any answer. In fact I usually get google results for cli torrent clients rather than what I want. So rtorrent is notorious about crashing ...
0
votes
0answers
20 views

How to compile wget or curl with https support statically on armv5tejl?

I have an IP camera with BusyBox. It has armv5tejl architecture When I want to use wget to download something from https://.... When I try to use it with https I get an error: wget: not an http or ...
1
vote
1answer
42 views

wget a website to csv

is there a way to wget a website and put it's tabular content in .csv? or maybe a cURL request a webpage, grab it's tabular content represented in numbers that consists of HTML to .csv?
2
votes
3answers
64 views

Accessing each line in .txt file, download and rename links

Let's say I have a .txt file where I have a list of image links that I want to download. example: image.jpg image2.jpg image3.jpg I use: cat images.txt | xargs wget and it works just fine What I ...
0
votes
2answers
37 views

recursively download from a website

I am trying to get images from a website url "www.example.com/products" in this products folder lots of subfolders there I need to download the products folder. In the www.example.com/products, www....
1
vote
1answer
33 views

wget not downloading files recursively

I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line: wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/...
2
votes
1answer
32 views

Shell script to grab one or many Wistia videos

What would be the best way to come up with a shell script that grabs one or many Wistia videos? There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: ...
2
votes
1answer
33 views

Can I use wget to download all files recursively, but not their actual content?

Sometimes, I wish to only get the directory structure of a website, but the files themselves are not important. I only want their name. Sort of like a mirror where every entry is just an empty dummy ...
1
vote
1answer
36 views

wget - how to reject string from downloading html pages

I am using the following wget command and it downloads the required files I need except for one thing... wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 ...
0
votes
0answers
38 views

wgetting links that are written in script form - not plain html

I use wget to download all the html pages in an online gallery - in order to grab all the links to the full size images as follows: wget --wait=3 --user-agent=Webtography --no-cookies --timestamping -...
2
votes
1answer
43 views

Using /dev/tcp with proxy

Is it possible to use /dev/tcp with a proxy? I only found wget is possible to use with proxy, but wget doesn't help me in my case.
0
votes
0answers
16 views

wget: avoid using server's last modified time [duplicate]

When I download a file using wget, the last modified time of the newly created file will follow that of the server. As a result, I cannot sort the files according to the date / sequence that I have ...
1
vote
1answer
74 views

Using WGET to pull all file names from an index

I work for a university and I want to pull all the file names of all the PDF catalogs we have and make a text file. These PDFs are all located in an Intranet index. WGET works fine with the intranet ...
1
vote
1answer
55 views

What is the curl equivalent of this wget command?

I'm trying to switch over part of a script from wget to curl. I have this command: wget --output-document=dl.html --quiet --timeout=10 --tries=1 $URL What's the curl equivalent of this?
1
vote
0answers
16 views

How to update an off-line web-site using wget in order to adjust extensions and convert links only?

I would like to update an off-line web-site downloaded using wget GNU command. BUT I want to take in consideration the following conditions : If a file does not already exist locally, update it. ...
1
vote
0answers
42 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
0
votes
3answers
33 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
-1
votes
2answers
85 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
2
votes
1answer
41 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
59 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension --convert-...
0
votes
0answers
55 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
0
votes
1answer
274 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
37 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
3
votes
2answers
115 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of translate....
1
vote
3answers
114 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k http://imgur....
-1
votes
2answers
35 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
0
votes
1answer
48 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
1
vote
1answer
106 views

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
0
votes
2answers
120 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
76 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
77 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
1
vote
1answer
126 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
25 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...
3
votes
2answers
58 views

Can I trust wget's exit status even if it was restarted?

I launched a wget job with the following command: urls=(https://example.com/file1 https://example.com/file2 https://example.com/file3 https://example.com/file4) for url in ${urls[@]}; do ...
0
votes
1answer
30 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url --no-check-...
1
vote
0answers
108 views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
0
votes
1answer
187 views

Why does downloading bash script from pastebin not work? [duplicate]

I'm trying to upload and download a bash script from pastebin, upload was straightforward enough with copy and paste, download didn't go as easy. There are two scripts I'm trying to download, these ...
-1
votes
1answer
131 views

WGET encountering 403 when trying to access filter - nginx [closed]

I'm trying to download some images from an asset folder of a site, that folder happens to be forbidden. I've tried various suggestions online including a referrer flag for google.com and setting the ...
1
vote
0answers
67 views

How to use wget to process a html file before downloading a file [closed]

I'm trying to write a bash script to download a certain file. In order to download the file, you need to be logged in, first to access he file, secondly because the file is generated with you ...
0
votes
1answer
41 views

Skip series in batch download with wget in a bash script

How can i skip a folder in a bash script using wget to batch download files, if the last file checked does not exist? Here is the sample code: #!/bin/bash # Script to download Reports @ brazil.crl....
0
votes
1answer
127 views

wget mirror site + resources from other domain

I'm trying to download a website so it can be browsed offline, so I want the html, images, css, javascript, etc. The problem is that the images are not all hosted on the same domain as the website, ...
5
votes
1answer
113 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called mirror.de/�%...
3
votes
1answer
115 views

using wget to download all audio files (over 100,000 pages on wikia)

I am trying to download all audio files in Wookiepedia, the Star Wars wiki. My first thought is something like this wget -r -A -nd .mp3 .ogg http://starwars.wikia.com/wiki/ This should download ...
1
vote
2answers
83 views

downloading images from webpage with different links

I am trying to download all the images from a webpage. The images are included as follows: <img src="https://alwaysSamePart.com/SomeRandomLinkThatVaries.jpg"> How should I parametrize my wget ...
3
votes
1answer
101 views

wget: Follow custom URL attributes

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a ...
-2
votes
2answers
60 views

How to use this script [closed]

#!\bin\bash for i in `seq 1 1000`; do wget -q -O quotes http://www.quotedb.com/quote/quote.php?action=random_quote; cat quotes>>final.txt; done I got this script from one website but I don't ...
0
votes
0answers
61 views

wget changes downloaded file size

I'm using wget to download Excel file with xlsx extension. The thing is that when I want to deal with the file using openpyxl, I get the following error: "BadZipfile: File is not a zip file" But when ...
1
vote
1answer
63 views

Linux server able to wget but not able to ping

A proxy server was added using export http://link:port/. I try to ping google.com and there is no response, the DNS is able to resolve an IP. wget of google.com works. routes: Destination ...
2
votes
1answer
30 views

Tools for parsing automatically-generated HTTP indexes?

Web servers commonly generate directory indexes listing the files within a directory and additional metadata. There is variation in the content and formatting of these pages, but generally have a ...