wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

-1
votes
2answers
73 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
0
votes
3answers
26 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
17
votes
3answers
34k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
2
votes
1answer
28 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
53 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension ...
0
votes
0answers
12 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
3
votes
2answers
72 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of ...
1
vote
2answers
2k views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
0
votes
1answer
87 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
21 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
-1
votes
0answers
7 views

Using wget to download all images from deadendthrills.com

I used to download all images from deadendthrills.com using the following command in order to use them in my wallpaper changer: wget -r -np --reject "index.html*" -e robots=off ...
0
votes
2answers
955 views

how to wget a github file

For example, there is a file here that I want to download via CLI: https://raw.githubusercontent.com/raspberrypi/linux/rpi-4.1.y/arch/arm/configs/bcmrpi_defconfig How to I download the actual file, ...
0
votes
1answer
68 views

A small script about wget

I want to download an image file called mch_aktuell_gross_20141229_79514688.gif (basically a forecast gif) from www.meteoradar.ch Here is my script: #!/bin/bash wget \ --recursive \ ...
5
votes
1answer
915 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ ...
29
votes
3answers
52k views

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to ...
-1
votes
2answers
25 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
1
vote
3answers
76 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k ...
0
votes
1answer
39 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
0
votes
1answer
34 views

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
3
votes
2answers
7k views

wget download using list of urls

I'm trying to download several files with wget. The relevant URLs are stored in the file url.txt. When using the command wget --content-disposition -i url.txt it doesn't find the .txt file. Where do ...
0
votes
2answers
66 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
47 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
58 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
-4
votes
3answers
4k views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
6
votes
3answers
7k views

Format of cookies when using wget?

What's the Netscape format of wget's cookies.txt? I need to mirror a website that requires login. I use a Chrome extension that returns cookies in that format, I save them in cookies.txt, import with ...
1
vote
1answer
64 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
21 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...
0
votes
3answers
1k views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
3
votes
2answers
55 views

Can I trust wget's exit status even if it was restarted?

I launched a wget job with the following command: urls=(https://example.com/file1 https://example.com/file2 https://example.com/file3 https://example.com/file4) for url in ${urls[@]}; do ...
0
votes
1answer
17 views

wget TLSv1 behaves the same as TLSv1_2

I used wireshark to compare curl --tlsv1.0 vs --tlsv1.2 The server handshake obediently uses TLSv1.0 for servers that support it (the point is to check to validate they don't): wget url ...
3
votes
2answers
1k views

Wget TLS 1.2 error

According to the manual, I should be able to specify TLS version 1.2 when using wget. When I try, it fails. wget https://site --no-check-certificate --secure-protocol=TLSv1_2 wget: ...
1
vote
0answers
65 views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
0
votes
1answer
89 views

Why does downloading bash script from pastebin not work? [duplicate]

I'm trying to upload and download a bash script from pastebin, upload was straightforward enough with copy and paste, download didn't go as easy. There are two scripts I'm trying to download, these ...
-1
votes
1answer
54 views

WGET encountering 403 when trying to access filter - nginx [closed]

I'm trying to download some images from an asset folder of a site, that folder happens to be forbidden. I've tried various suggestions online including a referrer flag for google.com and setting the ...
1
vote
0answers
54 views

How to use wget to process a html file before downloading a file [closed]

I'm trying to write a bash script to download a certain file. In order to download the file, you need to be logged in, first to access he file, secondly because the file is generated with you ...
0
votes
1answer
39 views

Skip series in batch download with wget in a bash script

How can i skip a folder in a bash script using wget to batch download files, if the last file checked does not exist? Here is the sample code: #!/bin/bash # Script to download Reports @ ...
5
votes
1answer
96 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called ...
6
votes
2answers
7k views

Is there a command line method by which I can check whether a downloaded file is complete or broken?

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.
0
votes
1answer
315 views

wget not working but apt-get does

Although this question relates to Raspbian running on my Raspberry Pi, it feels like it is a more general Linux query.... The Pi is sat on a network which includes a Microsoft TMG server. I have ...
0
votes
1answer
52 views

wget mirror site + resources from other domain

I'm trying to download a website so it can be browsed offline, so I want the html, images, css, javascript, etc. The problem is that the images are not all hosted on the same domain as the website, ...
3
votes
1answer
86 views

using wget to download all audio files (over 100,000 pages on wikia)

I am trying to download all audio files in Wookiepedia, the Star Wars wiki. My first thought is something like this wget -r -A -nd .mp3 .ogg http://starwars.wikia.com/wiki/ This should download ...
1
vote
2answers
58 views

downloading images from webpage with different links

I am trying to download all the images from a webpage. The images are included as follows: <img src="https://alwaysSamePart.com/SomeRandomLinkThatVaries.jpg"> How should I parametrize my wget ...
7
votes
5answers
21k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
0
votes
1answer
51 views

wget from Source Forge

I know this has been asked here: How do I download from SourceForge with wget? But I can't get it to work with my specific tool. Can anyone help me download from this sourceforge link: ...
3
votes
1answer
78 views

wget: Follow custom URL attributes

wget is a great tool to make a quick snapshot of a small site. As far as I know (I really hope I just couldn't find it in wget --help), wget can follow well-known HTML URL attributes only, like <a ...
2
votes
2answers
1k views

How to use wget to download certain files from a certain directory when the directory itself does not have an index.html?

There are a number of questions similar to this on StackExchange but none address this issue. I want to download all the pdf files in the 2007 directory at ...
10
votes
2answers
12k views

How to get text of a page using wget without html?

If I try wget on a webpage, I am getting the page as html. Is it possible to retrieve only text of a file without associated html ? (This is required for me since some of the HTML pages contains c ...
-2
votes
2answers
57 views

How to use this script [closed]

#!\bin\bash for i in `seq 1 1000`; do wget -q -O quotes http://www.quotedb.com/quote/quote.php?action=random_quote; cat quotes>>final.txt; done I got this script from one website but I don't ...
0
votes
0answers
53 views

wget changes downloaded file size

I'm using wget to download Excel file with xlsx extension. The thing is that when I want to deal with the file using openpyxl, I get the following error: "BadZipfile: File is not a zip file" But when ...
16
votes
3answers
23k views

How do I download from SourceForge with wget?

This question explains how to do it with curl. The accepted answer also points out that If wget is available, that would be far simpler. I looked through mam wget but the got lost in there, and ...