wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

learn more… | top users | synonyms

1
vote
2answers
164 views

How do I pipe the files downloaded with wget through tar?

I would like to pipe tarballs (.tar.gz files in this case) I download with GNU Wget (wget) through tar -xzf (to decompress them, if this is unclear) but I don't know how. I have tried: wget -q -O- ...
0
votes
2answers
37 views

Retain the ../../../ in wget command

../../../ is removed when i use it with the url in wget command. PLease see below: user $ wget http://n0t.meaning.anything:20000/../../../some/folder/file --2015-10-29 16:48:13-- ...
0
votes
2answers
23 views

try download a file from server if the pathname is known

I access a webpage by this url http://someRandomName.no.meaning:20025/hello.html The location of the hello.html that is displayed is /var/some/fulder/hello.html. (I mean in the server it is ...
0
votes
1answer
48 views

wget from Source Forge

I know this has been asked here: How do I download from SourceForge with wget? But I can't get it to work with my specific tool. Can anyone help me download from this sourceforge link: ...
2
votes
1answer
54 views

Download all of today's stackexchange with wget

Lets say that I want to recursively download all of today's posts on stackexchange as well as any wikipedia articles that they link to. wget -kprcH -l 4 -t 3 stackexchange.com is not a good ...
1
vote
0answers
63 views

How do I download all pdfs on webpage recursively keeping track of the site structure? [closed]

I'd like to download all the PDFs linked to this webpage and its subpages: ...
3
votes
2answers
871 views

Wget TLS 1.2 error

According to the manual, I should be able to specify TLS version 1.2 when using wget. When I try, it fails. wget https://site --no-check-certificate --secure-protocol=TLSv1_2 wget: ...
1
vote
2answers
61 views

Where is my downloaded file

I tried to install java runtime with the following commands su - cd /opt wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; ...
1
vote
1answer
88 views

Wget list of files with predictable URL patterns programmatically?

There are similar questions like this one, but nothing exactly the same. Using the new Apple TV screensaver JSON resource as an example, with predictable URLs, (only the numbers after the b and the - ...
2
votes
0answers
24 views

Make wget link-rewrite aware of all downloaded files

After downloading a tutorial via wget -r -l 1 -k -nd -p http://scikit-learn.org/stable/tutorial/index.html wget rewrites (-k) the link of all the documents on the index.html page to point to local ...
0
votes
1answer
113 views

Can I get wget -x to download the folder structure but not create the domain directory?

I'm currently doing this: wget -i download.txt -x It creates a root directory example.org/. What I'd like to do is only create subdirectories. Is there a way to compel wget into doing what I'd ...
0
votes
2answers
408 views

How to run the HTTP request without using CURL

I have ARM cpu based BusyBox v1.8.1 (Embedded Linux) with limited binaries. How can I http post or put without using curl? I have wget available: # wget BusyBox v1.8.1 (2015-04-06 16:22:12 IDT) ...
1
vote
1answer
72 views

Download yesterdays file from FTP server

How am I able to download files from a FTP server that have only been added yesterday? For all subfolders + keep the same structure.
1
vote
2answers
90 views

Downloading multiple similar URLs using braces

#!/bin/sh orig=$1 var=${orig%.jpg} var=`echo "$var" | grep -o "[0-9][0-9]$"` cutten=${orig%[0-9][0-9].jpg} sequence='{1,' for i in `seq 2 $(( $var - 1 ))`; do sequence=$sequence$i"," done ...
0
votes
1answer
199 views

curl and wget cut off url

I have a web cms that allows so called action IDs which are urls. When you hit those, they perform some code in the cms. So I wanted to perform an action periodically and used cron for that. PROBLEM: ...
1
vote
2answers
105 views

Getting a file with wget when the filename may change slightly

I have a program that takes data from five government sources and merges them into one large database for my company. I use wget to retrieve the files. However I have discovered that one of the ...
0
votes
2answers
836 views

how to wget a github file

For example, there is a file here that I want to download via CLI: https://raw.githubusercontent.com/raspberrypi/linux/rpi-4.1.y/arch/arm/configs/bcmrpi_defconfig How to I download the actual file, ...
0
votes
1answer
94 views

Continuing an interrupted `wget` session?

Is it possible to continue an interrupted wget session - eg. by parsing the log-file (created with -o or -a ), or after somehow have wget store additional information to disk (like it's list over ...
1
vote
1answer
90 views

wget fails to exclude directories

I want to download files from a site, but I want to exclude specific directories. I am using the following command: wget --user="xxxx" --password="xxx" -r -np -nH --cut-dirs=7 \ --reject ...
1
vote
1answer
1k views

Wget, abort retrying after failure or timeout

I'm calling a url with wget: /usr/bin/wget --read-timeout=7200 https://site_url/s Wget performs a GET request every 15 minutes in this case, despite the timeout being set, why does this happen? ...
0
votes
0answers
1k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
0
votes
1answer
63 views

Get total size of a list (containing HTTP links)

I have a file containing a list of files on the web, and I would like to know the total files' size (in human-readable format) and check their availability. Is there a command/solution to do so? P.S. ...
0
votes
2answers
128 views

wget, logging the output and the response

I calling an url using wget. This url gives me a response, its a Message id. I want to write the logs to a log file, with the message id as well. Also the log should be appended each time. I trying to ...
0
votes
2answers
166 views

Issue when running wget inside a bash script with an url stored in a variable

I'm having the following bash script: URL=`grep -E -m 1 -o "<ExportCatalogResult>(.*)</ExportCatalogResult>" costa_export.xml| sed -e ...
0
votes
1answer
175 views

WGET Continue without --no-check-certificate

I'm using wget to check if several boxes have access to other servers. The problem is sometimes the servers attempt to self-sign and the script hangs when this happens. It simply stops on the line ...
2
votes
1answer
4k views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
1
vote
1answer
152 views

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably simply and from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like ...
1
vote
1answer
36 views

wget a url from my public outdoor wx cam

I am trying to grab an image from my weathercam using wget and cron. For now I just need to see how to get an image downloaded and timestamp the filename. ...
1
vote
2answers
107 views

How do I get size of (deb) file from download link/URL?

I use --print-uris of apt-get to get url (download-link) of debs I want to download and install on my Debian (Trisquel) GNU/Linux. --print-uris Instead of fetching the files to install ...
1
vote
0answers
117 views

How to download thousands of files quickly?

I need to download thousands of files (approximately 2-12 MB each) - Currently I am using the following command wget --reject "index.html*" -A *.gz -N -r -l1 -nd --no-check-certificate --user ...
1
vote
2answers
94 views

Put two Cron jobs into crontab using wget

I am trying to put these two Cron Jobs: 0 3 * * * ! sudo -u asterisk /var/lib/asterisk/bin/module_admin --repos extended,standard,unsupported upgradeall 30 3 * * * ! sudo -u asterisk ...
1
vote
0answers
68 views

Download a webpage by putting everything it needs in a dedicated directory?

To download a webpage ( its html file and other files needed by it), wget seems to mirror the directory structure of the files of the webpage on the website server When in a web browser (e.g. Firefox ...
5
votes
2answers
421 views

Download a list of webpages in a file and rename them to avoid overwriting?

$ wget -r -np -nH --cut-dirs=2 -p -k "http://www.example.com/1" -O test.html Cannot specify both -k and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details. ...
1
vote
2answers
203 views

Equal command between wget and cURL (with options)?

I usually use wget to download remote file, the command I use: wget --timeout=5 --tries=1 --referer=http://www.example.com 'http://www.example.com/abcdefg123.jpg' -O image.jpg As you can see, there ...
0
votes
1answer
93 views

Whats the best way to copy a website onto your harddrive?

I like the website www.globalresearch.ca and I have made copies of it. This command works for me wget -r -l14 -t2 -T60 -E -k --no-check-certificate --restrict-file-names="windows,nocontrol" ...
3
votes
1answer
25 views

wget from ftp: supress default page creation

I'm trying to download subfolder from ftp. There are only data files. I'm using: wget -nH -np --cut-dirs=5 -r http://sss.com/1/2/3/4/5 Unfortunately wget creates one additional file: index.html ...
4
votes
5answers
3k views

Download HTTPS website available only through username and password with wget?

I'm trying to recursively download website which is normally available only when you login. I have valid username and password, but the problem is that I need to login through web interface, so using ...
-1
votes
1answer
221 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
2
votes
1answer
112 views

wget on the fly to extract specific info from file without downloading the entire file

I have some files (which are in .bam format(binary)) to download from a location. I can download them using wget and then want to extract some specific information from them only. For that I use ...
0
votes
0answers
271 views

Redirect with axel in linux

I am trying to download a file from googlevideo. This download link may expire in a few minutes but I will post everything I have. When I use axel with the following command, I get $ axel -n 10 ...
8
votes
3answers
3k views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
1
vote
1answer
90 views

Using date -1day with wget

I want to use wget with a cron to pull a file everyday however I want it to pull the previous days file from the remote server and the file is controlled by date variables in the URL as such: ...
0
votes
1answer
342 views

wget: force no default certificates

I'm having trouble with verification of custom server SSL certificates and wget. wget -O- --ca-certificate=myservercert.pem https://www.google.com This should fail but does not, as wget does ...
2
votes
1answer
126 views

Using wget, how to download to a specific location, without creating folders, and always overwrite original files

How do I use wget to download files to a specific location, without creating directories, and overwrite the original every time. I've tried using the -r -P and nc options in combination but this ...
2
votes
1answer
58 views

How to download a lot of documents on the terminal from a server?

I have to write a bash script (.sh) with less than 300 characters which downloads all of the genomes 22 in this server ftp://ftp.ncbi.nih.gov/genomes/Homo_sapiens/ (the ones with fa.gz extention) and ...
0
votes
1answer
5k views

bash cannot find mvn command after install

I tried to install Maven on CentOS 7, but afterwards typing mvn -version in the terminal results in bash: mvn: command not found.... How can I fix this so that I can call maven from the CentOS 7 ...
3
votes
1answer
79 views

Download a file to a server?

After ssh to a server, I want to download two files in https://iain.cx/src/ktmux/#download to the server. So for the first file ktmux_helper, I right click to find its link on my local computer, and ...
2
votes
2answers
421 views

Why wget cannot download page as root?

This is odd, I try to download a html file using wget wget --user=xxx --password=yyy http://192.168.1.1/ When I execute it as myself, it works, when I execute it as root it fails with message: ...
0
votes
3answers
872 views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
5
votes
4answers
2k views

tar a result of wget

I want to wget a file and tar it, in one command, I guess it is simple but I can't get it done. I tried several things. wget <url> | tar -cvz file.gz.tar - tar -cvzf file.tar `wget ...