wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

1
vote
1answer
12 views

Shell script to grab one or many Wistia videos

What would be the best way to come up with a shell script that grabs one or many Wistia videos? There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: ...
2
votes
1answer
27 views

Can I use wget to download all files recursively, but not their actual content?

Sometimes, I wish to only get the directory structure of a website, but the files themselves are not important. I only want their name. Sort of like a mirror where every entry is just an empty dummy ...
1
vote
1answer
30 views

wget - how to reject string from downloading html pages

I am using the following wget command and it downloads the required files I need except for one thing... wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 ...
4
votes
2answers
3k views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this (...
5
votes
4answers
8k views

Is there a way to disable wget from getting files from parent directories to given depth?

wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider: www.foo.com/bar1/bar2/bar3/index.html I would like to ...
0
votes
2answers
1k views

how to wget a github file

For example, there is a file here that I want to download via CLI: https://raw.githubusercontent.com/raspberrypi/linux/rpi-4.1.y/arch/arm/configs/bcmrpi_defconfig How to I download the actual file, ...
0
votes
0answers
35 views

wgetting links that are written in script form - not plain html

I use wget to download all the html pages in an online gallery - in order to grab all the links to the full size images as follows: wget --wait=3 --user-agent=Webtography --no-cookies --timestamping -...
2
votes
1answer
40 views

Using /dev/tcp with proxy

Is it possible to use /dev/tcp with a proxy? I only found wget is possible to use with proxy, but wget doesn't help me in my case.
3
votes
3answers
2k views

Possible to get “old” time stamp behaviour?

When downloading a file or e.g. unzipping one, the time stamp is not set to when the file appeared on my harddisk, which is what I want. See this example ~$ wget https://www.google.com/images/srpr/...
0
votes
0answers
16 views

wget: avoid using server's last modified time [duplicate]

When I download a file using wget, the last modified time of the newly created file will follow that of the server. As a result, I cannot sort the files according to the date / sequence that I have ...
50
votes
4answers
27k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
1
vote
1answer
61 views

Using WGET to pull all file names from an index

I work for a university and I want to pull all the file names of all the PDF catalogs we have and make a text file. These PDFs are all located in an Intranet index. WGET works fine with the intranet ...
4
votes
2answers
1k views

wget --spider: how to tell where broken links are coming from

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes ...
1
vote
1answer
31 views

What is the curl equivalent of this wget command?

I'm trying to switch over part of a script from wget to curl. I have this command: wget --output-document=dl.html --quiet --timeout=10 --tries=1 $URL What's the curl equivalent of this?
1
vote
1answer
102 views

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
1
vote
0answers
15 views

How to update an off-line web-site using wget in order to adjust extensions and convert links only?

I would like to update an off-line web-site downloaded using wget GNU command. BUT I want to take in consideration the following conditions : If a file does not already exist locally, update it. ...
1
vote
0answers
35 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
-1
votes
2answers
82 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
0
votes
3answers
32 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
18
votes
3answers
35k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
2
votes
1answer
34 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
55 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension --convert-...
0
votes
0answers
30 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
3
votes
2answers
92 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of translate....
1
vote
2answers
2k views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
0
votes
1answer
173 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
24 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
0
votes
1answer
68 views

A small script about wget

I want to download an image file called mch_aktuell_gross_20141229_79514688.gif (basically a forecast gif) from www.meteoradar.ch Here is my script: #!/bin/bash wget \ --recursive \ -...
5
votes
1answer
936 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ -...
32
votes
3answers
56k views

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to /var/cache/foobar/stackexchange-site-list....
-1
votes
2answers
32 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
1
vote
3answers
89 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k http://imgur....
0
votes
1answer
43 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
3
votes
2answers
7k views

wget download using list of urls

I'm trying to download several files with wget. The relevant URLs are stored in the file url.txt. When using the command wget --content-disposition -i url.txt it doesn't find the .txt file. Where do ...
0
votes
2answers
87 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
67 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
63 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
-3
votes
3answers
4k views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
6
votes
3answers
8k views

Format of cookies when using wget?

What's the Netscape format of wget's cookies.txt? I need to mirror a website that requires login. I use a Chrome extension that returns cookies in that format, I save them in cookies.txt, import with ...
1
vote
1answer
99 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
24 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...
0
votes
3answers
1k views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
3
votes
2answers
57 views

Can I trust wget's exit status even if it was restarted?

I launched a wget job with the following command: urls=(https://example.com/file1 https://example.com/file2 https://example.com/file3 https://example.com/file4) for url in ${urls[@]}; do ...
0
votes
1answer
24 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url --no-check-...
3
votes
2answers
1k views

Wget TLS 1.2 error

According to the manual, I should be able to specify TLS version 1.2 when using wget. When I try, it fails. wget https://site --no-check-certificate --secure-protocol=TLSv1_2 wget: --secure-...
1
vote
0answers
88 views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
0
votes
1answer
130 views

Why does downloading bash script from pastebin not work? [duplicate]

I'm trying to upload and download a bash script from pastebin, upload was straightforward enough with copy and paste, download didn't go as easy. There are two scripts I'm trying to download, these ...
-1
votes
1answer
86 views

WGET encountering 403 when trying to access filter - nginx [closed]

I'm trying to download some images from an asset folder of a site, that folder happens to be forbidden. I've tried various suggestions online including a referrer flag for google.com and setting the ...
1
vote
0answers
59 views

How to use wget to process a html file before downloading a file [closed]

I'm trying to write a bash script to download a certain file. In order to download the file, you need to be logged in, first to access he file, secondly because the file is generated with you ...
0
votes
1answer
40 views

Skip series in batch download with wget in a bash script

How can i skip a folder in a bash script using wget to batch download files, if the last file checked does not exist? Here is the sample code: #!/bin/bash # Script to download Reports @ brazil.crl....