wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

0
votes
1answer
14 views

Owncloud does not show files, that are uploaded via SSH

After spending 10 Hours configuring owncloud on Fedora 24 (because of NO documentation and proper tutorial), I finally got OwnCloud up and running perfectly. My owncloud configuration files are in /...
0
votes
1answer
38 views

download a torrent file from kickass torrent from linux cli

this one has been bugging me for a while and I cannot find any answer. In fact I usually get google results for cli torrent clients rather than what I want. So rtorrent is notorious about crashing ...
27
votes
5answers
35k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
0
votes
2answers
1k views

how to wget a github file

For example, there is a file here that I want to download via CLI: https://raw.githubusercontent.com/raspberrypi/linux/rpi-4.1.y/arch/arm/configs/bcmrpi_defconfig How to I download the actual file, ...
3
votes
1answer
130 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
0
votes
0answers
20 views

How to compile wget or curl with https support statically on armv5tejl?

I have an IP camera with BusyBox. It has armv5tejl architecture When I want to use wget to download something from https://.... When I try to use it with https I get an error: wget: not an http or ...
1
vote
1answer
42 views

wget a website to csv

is there a way to wget a website and put it's tabular content in .csv? or maybe a cURL request a webpage, grab it's tabular content represented in numbers that consists of HTML to .csv?
2
votes
3answers
65 views

Accessing each line in .txt file, download and rename links

Let's say I have a .txt file where I have a list of image links that I want to download. example: image.jpg image2.jpg image3.jpg I use: cat images.txt | xargs wget and it works just fine What I ...
0
votes
2answers
38 views

recursively download from a website

I am trying to get images from a website url "www.example.com/products" in this products folder lots of subfolders there I need to download the products folder. In the www.example.com/products, www....
1
vote
1answer
33 views

wget not downloading files recursively

I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line: wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/...
2
votes
1answer
32 views

Shell script to grab one or many Wistia videos

What would be the best way to come up with a shell script that grabs one or many Wistia videos? There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: ...
2
votes
1answer
34 views

Can I use wget to download all files recursively, but not their actual content?

Sometimes, I wish to only get the directory structure of a website, but the files themselves are not important. I only want their name. Sort of like a mirror where every entry is just an empty dummy ...
1
vote
1answer
37 views

wget - how to reject string from downloading html pages

I am using the following wget command and it downloads the required files I need except for one thing... wget -U "Mozilla/5.0" --wait=3 --load-cookies cookies.txt --timestamping --recursive --level=2 ...
4
votes
2answers
3k views

How to make wget download recursive combining --accept with --exclude-directories?

I'm trying to download some directories from an Apache server, but I need to ignore some directories that have huge files I don't care about The dir structure in the server is somewhat like this (...
5
votes
4answers
8k views

Is there a way to disable wget from getting files from parent directories to given depth?

wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider: www.foo.com/bar1/bar2/bar3/index.html I would like to ...
0
votes
0answers
38 views

wgetting links that are written in script form - not plain html

I use wget to download all the html pages in an online gallery - in order to grab all the links to the full size images as follows: wget --wait=3 --user-agent=Webtography --no-cookies --timestamping -...
2
votes
1answer
43 views

Using /dev/tcp with proxy

Is it possible to use /dev/tcp with a proxy? I only found wget is possible to use with proxy, but wget doesn't help me in my case.
3
votes
3answers
2k views

Possible to get “old” time stamp behaviour?

When downloading a file or e.g. unzipping one, the time stamp is not set to when the file appeared on my harddisk, which is what I want. See this example ~$ wget https://www.google.com/images/srpr/...
0
votes
0answers
16 views

wget: avoid using server's last modified time [duplicate]

When I download a file using wget, the last modified time of the newly created file will follow that of the server. As a result, I cannot sort the files according to the date / sequence that I have ...
52
votes
4answers
28k views

Throttle the download speed of wget or curl while downloading

Is it possible to throttle (limit) the download speed of wget or curl ? Is it possible to change the throttle value while it is downloading ?
1
vote
1answer
74 views

Using WGET to pull all file names from an index

I work for a university and I want to pull all the file names of all the PDF catalogs we have and make a text file. These PDFs are all located in an Intranet index. WGET works fine with the intranet ...
4
votes
2answers
1k views

wget --spider: how to tell where broken links are coming from

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes ...
1
vote
1answer
55 views

What is the curl equivalent of this wget command?

I'm trying to switch over part of a script from wget to curl. I have this command: wget --output-document=dl.html --quiet --timeout=10 --tries=1 $URL What's the curl equivalent of this?
1
vote
1answer
106 views

How to Wget with Subset Condition?

I want to download a subset of data recursively in the website that is within HTML class .container current main page in .container daughter pages of links also in .containers in daughter pages ...
1
vote
0answers
16 views

How to update an off-line web-site using wget in order to adjust extensions and convert links only?

I would like to update an off-line web-site downloaded using wget GNU command. BUT I want to take in consideration the following conditions : If a file does not already exist locally, update it. ...
1
vote
0answers
42 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
-1
votes
2answers
86 views

Have sed echo string after match

I need to have the string directly following/pbs.twimg.com\/profile_images\/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use ...
0
votes
3answers
33 views

wget’s --output-document= option failed when contain ~ character

I wright a documentation so I prefer use long options because it’s self-descripting. --output-document= is more readable than his short equivalent -O. Then, I have to indicate a file in the home ...
19
votes
3answers
36k views

How do I use wget with a list of URLs and their corresponding output files?

Suppose list_of_urls looks like this: http://www.url1.com/some.txt http://www.url2.com/video.mp4 I know how to use that with: wget -i list_of_urls But, what if my list_of_urls has this, and they ...
2
votes
1answer
41 views

Why is wget ignoring some files in ftp transfer?

I am trying to deploy a WordPress website from a test server to a production server. wget seems to be an efficient solution for transferring lots of files between 2 servers via FTP. I connect to the ...
0
votes
1answer
59 views

I'm unable to get wget to succesfully be recursive while mirroring a blog

I want to make an offline mirror of this blog: http://blogs.gamefilia.com/lord-areg I'm using: wget --recursive --level=0 --span-hosts --no-clobber --page-requisites --html-extension --convert-...
0
votes
0answers
56 views

wget --post-data only works from shell, doesn't work from cgi-bin

I have just recompiled my OPENWRT distribution. I took care of including: - wget in the custom configuration of busybox - the rpcd package - uhttpd and uhttpd-mod-ubus I am experiencing a strange ...
3
votes
2answers
116 views

Accessing Google-translate via wget

I don't want to call the whole Firefox/Chrome/Opera... to find out the meaning of a word with the Google translate, so I decided to write a shell script which uses wget to get the content of translate....
1
vote
2answers
2k views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
0
votes
1answer
280 views

Wget: Segmentation fault (core dumped)

Whenever I try to download anything with wget I get Segmentation fault (core dumped) with or without -c it doesn't matter. I have tried this in both Ubuntu mate on my Raspberry pi 3 and arch Linux on ...
0
votes
0answers
38 views

wget probelm with a tunnel connection using ssh

I know this is not a new question, but I have researched a bit and so far none of the answers I've seen gave me the answer. I have this situation: I´m working on a campus where the network is highly ...
0
votes
1answer
68 views

A small script about wget

I want to download an image file called mch_aktuell_gross_20141229_79514688.gif (basically a forecast gif) from www.meteoradar.ch Here is my script: #!/bin/bash wget \ --recursive \ -...
5
votes
1answer
953 views

Wget out of memory error kills process

I wanted to backup all my .vcf files from my carddav server (ownCloud). The script is very simple and is the following: $ wget -Avcf -r -np -l0 --no-check-certificate -e robots=off --user=user \ -...
35
votes
3answers
59k views

Download using wget to a different directory than current directory

I need to use wget to download a file to the directory /var/cache/foobar/ (so, as an example, if I download stackexchange-site-list.txt, it'd be downloaded to /var/cache/foobar/stackexchange-site-list....
-1
votes
2answers
35 views

Migrating website do another server using wget

I have a website located, say, at http://www.abc.com. I need to make it available at, say, http://www.cde.com. I tried to download the site using wget, but encountered two problems: There are many ...
1
vote
3answers
117 views

Wget a page on imgur

I am trying to save a page and its images from imgur.com. All I get is the pages, no images. Take this URL for example: http://imgur.com/gallery/l5NXz I have tried this: wget -p -k http://imgur....
0
votes
1answer
48 views

Rental server refuses SSH connection; won't allow authorise me to download my SSH key

I am attempting to connect to my rental server via ssh to install MemSQL for a project. I have written the following in ~/.ssh/config: HOST eHosts HostName e9.ehosts.com Port 2222 User orderof3 But ...
3
votes
2answers
8k views

wget download using list of urls

I'm trying to download several files with wget. The relevant URLs are stored in the file url.txt. When using the command wget --content-disposition -i url.txt it doesn't find the .txt file. Where do ...
0
votes
2answers
121 views

How can I convert all the html files I get into text files after a wget command?

How can I convert all the html files I get into plain text files after a wget command? I'm thinking of using lynx to convert html files into ".txt" files, getting rid of tags. I have this snippet ...
0
votes
0answers
76 views

Very slow download from FTP-site, must wait for control-connection to close between operations

I'm trying to recursively download some files and directories (some with sub-directories) from a FTP-site with wget, but it's turning-out to be a painfully slow process. It seems like I'm only able ...
1
vote
1answer
77 views

Display wget transfer in a more compact way (while keeping the error detection functionality)

I know that wget have some options to display or not the progress bars. I would like to display the wget transfer in a more short way, or.. a percent or something dynamic but not taking so much space ...
-3
votes
3answers
4k views

How do I use wget to download all links from my site and save to a text file?

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start ...
6
votes
3answers
8k views

Format of cookies when using wget?

What's the Netscape format of wget's cookies.txt? I need to mirror a website that requires login. I use a Chrome extension that returns cookies in that format, I save them in cookies.txt, import with ...
1
vote
1answer
126 views

bash scripting - wget or curl file with action based on the status 200 or not

I would like to write a script where I try to download a file with wget (or curl if is really not something that can be done with wget) and do 2 actions if the file exists and it's downloaded ...
-1
votes
1answer
25 views

linux + wget + print the files from URL address using login and password

I have the following URL address and need login/password in order to login https://gt.dsdckdhdb.com/ after we login we can see files there how with wget command we can see the files from the URL ...