wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

-1
votes
1answer
43 views

wget -P -O together does not work?

i did try command: #!/bin/bash folder_tmp=~/name-x-tmp/ #this is worked echo -n "url: " && read -e url #this is worked wget -P $folder_tmp -O 1.txt $url #this is not worked!! exit 1 ...
1
vote
1answer
201 views

Static compilation of wget

I downloaded the source code of wget and tried a static compilation. These are the steps that I followed : ./configure LDFLAGS=-static At the end the output shown is as follows : Version: ...
2
votes
1answer
34 views

Download only specific directories from FTP site using wget

I have read the manual page for wget and search Google for an answer, but haven't been able to get it. How would I use wget to download only folders that contain a specific term in their names from ...
12
votes
2answers
864 views

What is the “2001 article” for blocking “automated retrieval programs” referred to by the wget man page?

The wget man page states this, under the section for the --random-wait parameter: Some web sites may perform log analysis to identify retrieval programs such as Wget by looking for ...
2
votes
3answers
183 views

How to download files without full URL?

I was always wondering how to download the files through the Linux shell (I have wget, curl) that do not have a full URL of the file to be downloaded, but the full URL is passed e.g. to the browser ...
1
vote
2answers
45 views

Download arbitrary number of files using wget from a self-redirecting directory

I need to download a very large number of pdf files (would be tedious if not impossible to collect and collate all the links) from my university website using wget, with links like this: ...
3
votes
3answers
218 views

How do I download all files listed in an array?

I want to download all files in this GitHub directory to /usr/share/enlightenment/data/config. I have this script: L=('e.cfg' 'e_randr.cfg' 'exehist.cfg' 'module.battery.cfg' 'module.clock.cfg' ...
1
vote
1answer
300 views

How to download all images from a forum thread with wget

For example in a thread that is 30 pages long and the url ends in page=XX is there a way to make a loop that would increment the page each time and download the images only
1
vote
1answer
81 views

Wget - Retrieving Recursive URLs from list of URLs

I have a list of urls in a txt file. Each of these urls point to a page with another list of urls. What WGET command can I use to retrieve the urls from the pages that have been linked from the the ...
1
vote
2answers
177 views

How do I pipe the files downloaded with wget through tar?

I would like to pipe tarballs (.tar.gz files in this case) I download with GNU Wget (wget) through tar -xzf (to decompress them, if this is unclear) but I don't know how. I have tried: wget -q -O- ...
0
votes
2answers
37 views

Retain the ../../../ in wget command

../../../ is removed when i use it with the url in wget command. PLease see below: user $ wget http://n0t.meaning.anything:20000/../../../some/folder/file --2015-10-29 16:48:13-- ...
0
votes
2answers
25 views

try download a file from server if the pathname is known

I access a webpage by this url http://someRandomName.no.meaning:20025/hello.html The location of the hello.html that is displayed is /var/some/fulder/hello.html. (I mean in the server it is ...
0
votes
1answer
51 views

wget from Source Forge

I know this has been asked here: How do I download from SourceForge with wget? But I can't get it to work with my specific tool. Can anyone help me download from this sourceforge link: ...
2
votes
1answer
55 views

Download all of today's stackexchange with wget

Lets say that I want to recursively download all of today's posts on stackexchange as well as any wikipedia articles that they link to. wget -kprcH -l 4 -t 3 stackexchange.com is not a good ...
1
vote
0answers
71 views

How do I download all pdfs on webpage recursively keeping track of the site structure? [closed]

I'd like to download all the PDFs linked to this webpage and its subpages: ...
3
votes
2answers
1k views

Wget TLS 1.2 error

According to the manual, I should be able to specify TLS version 1.2 when using wget. When I try, it fails. wget https://site --no-check-certificate --secure-protocol=TLSv1_2 wget: ...
1
vote
2answers
61 views

Where is my downloaded file

I tried to install java runtime with the following commands su - cd /opt wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; ...
1
vote
1answer
95 views

Wget list of files with predictable URL patterns programmatically?

There are similar questions like this one, but nothing exactly the same. Using the new Apple TV screensaver JSON resource as an example, with predictable URLs, (only the numbers after the b and the - ...
2
votes
0answers
26 views

Make wget link-rewrite aware of all downloaded files

After downloading a tutorial via wget -r -l 1 -k -nd -p http://scikit-learn.org/stable/tutorial/index.html wget rewrites (-k) the link of all the documents on the index.html page to point to local ...
0
votes
1answer
128 views

Can I get wget -x to download the folder structure but not create the domain directory?

I'm currently doing this: wget -i download.txt -x It creates a root directory example.org/. What I'd like to do is only create subdirectories. Is there a way to compel wget into doing what I'd ...
0
votes
2answers
482 views

How to run the HTTP request without using CURL

I have ARM cpu based BusyBox v1.8.1 (Embedded Linux) with limited binaries. How can I http post or put without using curl? I have wget available: # wget BusyBox v1.8.1 (2015-04-06 16:22:12 IDT) ...
1
vote
1answer
77 views

Download yesterdays file from FTP server

How am I able to download files from a FTP server that have only been added yesterday? For all subfolders + keep the same structure.
1
vote
2answers
94 views

Downloading multiple similar URLs using braces

#!/bin/sh orig=$1 var=${orig%.jpg} var=`echo "$var" | grep -o "[0-9][0-9]$"` cutten=${orig%[0-9][0-9].jpg} sequence='{1,' for i in `seq 2 $(( $var - 1 ))`; do sequence=$sequence$i"," done ...
0
votes
1answer
209 views

curl and wget cut off url

I have a web cms that allows so called action IDs which are urls. When you hit those, they perform some code in the cms. So I wanted to perform an action periodically and used cron for that. PROBLEM: ...
1
vote
2answers
106 views

Getting a file with wget when the filename may change slightly

I have a program that takes data from five government sources and merges them into one large database for my company. I use wget to retrieve the files. However I have discovered that one of the ...
0
votes
2answers
945 views

how to wget a github file

For example, there is a file here that I want to download via CLI: https://raw.githubusercontent.com/raspberrypi/linux/rpi-4.1.y/arch/arm/configs/bcmrpi_defconfig How to I download the actual file, ...
0
votes
1answer
101 views

Continuing an interrupted `wget` session?

Is it possible to continue an interrupted wget session - eg. by parsing the log-file (created with -o or -a ), or after somehow have wget store additional information to disk (like it's list over ...
1
vote
1answer
99 views

wget fails to exclude directories

I want to download files from a site, but I want to exclude specific directories. I am using the following command: wget --user="xxxx" --password="xxx" -r -np -nH --cut-dirs=7 \ --reject ...
1
vote
1answer
1k views

Wget, abort retrying after failure or timeout

I'm calling a url with wget: /usr/bin/wget --read-timeout=7200 https://site_url/s Wget performs a GET request every 15 minutes in this case, despite the timeout being set, why does this happen? ...
0
votes
0answers
1k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
0
votes
1answer
69 views

Get total size of a list (containing HTTP links)

I have a file containing a list of files on the web, and I would like to know the total files' size (in human-readable format) and check their availability. Is there a command/solution to do so? P.S. ...
0
votes
2answers
138 views

wget, logging the output and the response

I calling an url using wget. This url gives me a response, its a Message id. I want to write the logs to a log file, with the message id as well. Also the log should be appended each time. I trying to ...
0
votes
2answers
197 views

Issue when running wget inside a bash script with an url stored in a variable

I'm having the following bash script: URL=`grep -E -m 1 -o "<ExportCatalogResult>(.*)</ExportCatalogResult>" costa_export.xml| sed -e ...
0
votes
1answer
186 views

WGET Continue without --no-check-certificate

I'm using wget to check if several boxes have access to other servers. The problem is sometimes the servers attempt to self-sign and the script hangs when this happens. It simply stops on the line ...
2
votes
1answer
4k views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
1
vote
1answer
167 views

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably simply and from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like ...
1
vote
1answer
37 views

wget a url from my public outdoor wx cam

I am trying to grab an image from my weathercam using wget and cron. For now I just need to see how to get an image downloaded and timestamp the filename. ...
1
vote
2answers
111 views

How do I get size of (deb) file from download link/URL?

I use --print-uris of apt-get to get url (download-link) of debs I want to download and install on my Debian (Trisquel) GNU/Linux. --print-uris Instead of fetching the files to install ...
1
vote
0answers
125 views

How to download thousands of files quickly?

I need to download thousands of files (approximately 2-12 MB each) - Currently I am using the following command wget --reject "index.html*" -A *.gz -N -r -l1 -nd --no-check-certificate --user ...
1
vote
2answers
100 views

Put two Cron jobs into crontab using wget

I am trying to put these two Cron Jobs: 0 3 * * * ! sudo -u asterisk /var/lib/asterisk/bin/module_admin --repos extended,standard,unsupported upgradeall 30 3 * * * ! sudo -u asterisk ...
1
vote
0answers
69 views

Download a webpage by putting everything it needs in a dedicated directory?

To download a webpage ( its html file and other files needed by it), wget seems to mirror the directory structure of the files of the webpage on the website server When in a web browser (e.g. Firefox ...
5
votes
2answers
439 views

Download a list of webpages in a file and rename them to avoid overwriting?

$ wget -r -np -nH --cut-dirs=2 -p -k "http://www.example.com/1" -O test.html Cannot specify both -k and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details. ...
1
vote
2answers
218 views

Equal command between wget and cURL (with options)?

I usually use wget to download remote file, the command I use: wget --timeout=5 --tries=1 --referer=http://www.example.com 'http://www.example.com/abcdefg123.jpg' -O image.jpg As you can see, there ...
0
votes
1answer
94 views

Whats the best way to copy a website onto your harddrive?

I like the website www.globalresearch.ca and I have made copies of it. This command works for me wget -r -l14 -t2 -T60 -E -k --no-check-certificate --restrict-file-names="windows,nocontrol" ...
3
votes
1answer
26 views

wget from ftp: supress default page creation

I'm trying to download subfolder from ftp. There are only data files. I'm using: wget -nH -np --cut-dirs=5 -r http://sss.com/1/2/3/4/5 Unfortunately wget creates one additional file: index.html ...
4
votes
5answers
3k views

Download HTTPS website available only through username and password with wget?

I'm trying to recursively download website which is normally available only when you login. I have valid username and password, but the problem is that I need to login through web interface, so using ...
-1
votes
1answer
239 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
2
votes
1answer
116 views

wget on the fly to extract specific info from file without downloading the entire file

I have some files (which are in .bam format(binary)) to download from a location. I can download them using wget and then want to extract some specific information from them only. For that I use ...
8
votes
3answers
3k views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
1
vote
1answer
91 views

Using date -1day with wget

I want to use wget with a cron to pull a file everyday however I want it to pull the previous days file from the remote server and the file is controlled by date variables in the URL as such: ...