Questions tagged [wget]
wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)
0
votes
0answers
32 views
How can I make wget rename files based on link name?
Everything I'm finding is based on URL, but I want the opposite.
I have an html index with a list of links.
The links are named like Apples, oranges, grapes, bananas. The files associated with those ...
1
vote
1answer
19 views
wget -i notifying when each file finishes (for processing purposes)
I'd like to be able to process multiple files downloaded by wget -i immediately after they are downloaded (instead of waiting for all files in the list to finish--for the entire wget process to exit). ...
1
vote
2answers
65 views
wget download files from a file list, how to specify names for each of the downloaded file?
When downloading a single file, we can use wget's -O option to specify the file name. When I'm downloading Urls in a file using wget -i filelist.txt (filelist.txt contains list Of Urls I want to ...
0
votes
1answer
90 views
Download ZIP file of private GitHub repository using `wget` like `curl`
There are already some questions with answers how to download the ZIP file of GitHub repositories, but they don't work if the repository is private.
I know it should work with wget somehow since it ...
0
votes
2answers
72 views
Using wget to finish a downloaing
I was downloading a big file but without wget tool, the good news is that the link of the file has the following form:
https://link.com/file.zip
Sadly, the downloaing was broken, so now I want to ...
0
votes
2answers
64 views
wget --spider fails with 404 but works without --spider
I'm trying to do a health check in a docker container. I found this command:
wget --quiet --tries=1 --spider http://localhost:6077 || exit 1
The issue is that while the container is running, if I ...
0
votes
0answers
83 views
wget: error getting response: Address family not supported by protocol
Attempt to retrieve entware script from a test router indicates a wget error:
root@DD-WRT:/opt# wget http://bin.entware.net/mipselsf-k3.4/installer/generic.sh
Connecting to bin.entware.net (104.27....
0
votes
1answer
49 views
How can `wget` get more logs when it gets error
I have several questions about wget tool in linux:
# wget http://www.jdfschool.com
--2019-04-05 02:33:44-- (Trys: 3) http://www.jdfschool.com/
Connecting www.jdfschool.com|115.28.223.13|:80... ...
1
vote
1answer
42 views
How to update a local mirror from a public server?
I already have a mirror of the entire slackware64-current/ directory tree using
wget -r -np -R "index.html*" https://mirror.slackbuilds.org/slackware/slackware64-current/
and that's fine. But it's ...
0
votes
0answers
29 views
How to receive data from a Webpage and send it back to it in the same connection?
I am trying to fill a form of a webpage with curl and give it some arguments. Dumb enough, it also sends a hidden form in which there is an unique token which has to be sent back to the page, a so ...
-1
votes
1answer
88 views
How to wget mirror website faster?
when I use wget -m url it downloads one file at a time.
How to make wget download multiple files at a time?
0
votes
0answers
77 views
Wget wildcard files without temp html files
Hello I've been trying to find the correct solution to download wildcard files using wget - I used to have nexus api calls to download the latest files, but unfortunately I have to use pattern ...
0
votes
0answers
70 views
wget isn't converting some links properly
I'm trying to get a "working copy" of google.com using wget.
The command I use is:
wget --page-requisites --html-extension --convert-links --restrict-file-names=windows --no-host-directories -U "...
0
votes
0answers
54 views
calling wget from a browser using php
My httpd is running with a daemon user. I want to do a wget using php from FTP server. When I run the php from linux server, the same downloads the required file from FTP. But when I call the same php ...
1
vote
0answers
114 views
GET via wget or cURL gives a limited response
I'm bumping my head on a simple HTTP GET request.
Using a tool like Postman, I'm getting the full and expected response (an over 1 thousand lines xml document, displaying the details of my 60 albums)
...
0
votes
1answer
82 views
Troubleshooting: cURL timeouts, Java app doesn't
I'm trying to use curl against a remote server to access a REST API (and right now, just the welcome page...)
Currently, I have a Java application that can send REST request on this server.
But if I ...
0
votes
2answers
163 views
wget not downloading streaming mp3 correctly
I am trying to download a streaming mp3 using wget. this is my basic command:
wget http://sj128.hnux.com/sj128.mp3 -c --timeout=1 --waitretry=0 --tries=0 -O "file.mp3"
i have been doing this in a ...
0
votes
2answers
44 views
How to save a web resource, applying decoding to URL Encoded Characters?
I would like to save this file with this URL "http://pti.regione.sicilia.it/portal/page/portal/PIR_PORTALE/PIR_LaStrutturaRegionale/PIR_AssessoratoEconomia/PIR_DipBilancioTesoro/PIR_Areetematiche/...
-1
votes
1answer
27 views
How to download AWS documentation using linux terminal?
I tried to download AWS documentation using wget and curl commands. I realized and turned off robot and set Mozilla as browser header. But still failed to recursively download the entire documents.
0
votes
0answers
111 views
wget output progress to file per downloaded file
I'm downloading multiple bigger files with wget like this: wget -i 'FILE_WITH_URLS' --content-disposition -b. By default it creates a file named wget-log and appends time download progresses. The file ...
0
votes
0answers
53 views
wget --spider ram memory consumption and use of -c
I am using wget --spider. It is consuming a lot of ram memory. How could I limit the use of ram memory from the wget command?
wget --spider -r -c http://www.example.org 2>&1 | grep '^--' | awk ...
0
votes
1answer
89 views
Set date on embedded linux with limited busybox (no ntp)
I am trying to automatically set a date after boot on an embeddd linux with busybox ( I dont have a RTC).
I also dont have ntp command in busybox, nor curl. My wget is very limited too, and I cant ...
0
votes
0answers
218 views
Download filename-pattern-matching files from a public Google Drive folder with rclone (or alternative)
I'm trying to download all files from a public gdrive folder but with certain filetype restrictions (eg exclude *.avi,*.mkv,*.mp4 etc).
I originally tried with wget since that is what I am familiar ...
8
votes
3answers
1k views
Why can't I see the “wget” job when I execute it in the background?
I use wget command in the background like this
wget -bq
and it prints
Continuing in background, pid 31754.
But when I type the command jobs, I don't see my job(although the downloading is not ...
0
votes
1answer
147 views
wget download specific files
I am learning to use the various wget commands (man wget). But I'm having trouble learning how to download specific files from some sites (on the whole site they are downloaded). On some sites I get ...
2
votes
1answer
49 views
wget image problem for daily wallpaper script
i have a script (that Nuno wrote) to wget an image from one particular site and set my wallpaper. however i can't seem to modify it to be able to wget an image from this site: chromecast
images in ...
2
votes
1answer
259 views
How to download multiple files from website with wget and perform functions on downloaded file before next file is downloaded
I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file.
Is there a ...
6
votes
2answers
404 views
'ping' uses localhost instead of public IP address
Let's say my server has IP address 11.22.33.44 and hostname server1.mydomain.com.
When I ping server1.mydomain.com, it looks as if ping is actually using the public IP address:
# ping server1....
1
vote
2answers
134 views
Can wget download reveal.js image assets?
In using the wget command below,
$ wget \
--recursive \
--no-clobber \
--page-requisites \
--html-extension \
--convert-links \
--restrict-file-names=windows \
--...
0
votes
1answer
66 views
command to download & agree licensed software
I am writing a script that download this software, then install and run it on linux system.
Issue is when I do wget to download, my download doesn't initiate, it halts showing below line, because ...
0
votes
1answer
234 views
wget download regex pattern files from remote URL
I want to download all *httpd* RPM file from remote CentOS mirror and I am trying he following command but it doesn't seem to work
[root@yum foo]# wget -r --no-parent -A "*httpd*" https://mirrors....
2
votes
3answers
119 views
Security issues with “wget”
I was wondering if there are any vulnerabilities that I may face when using "wget" and its options. And specifically, I've used a benchmarking script to test the performance of my VPS. The command is ...
0
votes
1answer
417 views
How to download https file in AIX without wget and curl? [duplicate]
In an UNIX environment running AIX where I'm a not privileged user (without sudo) if I need to download a file from HTTPS server and there is no wget or curl... How I do it?
Is there any replacement ...
2
votes
1answer
344 views
Alternatives to cURL/wget for remote API calls
I'm on IBM AIX 7.1 where cURL and wget are not available. Also, I'm not in a position to install either.
Are there alternatives that would be available that would enable remote API calls using GET/...
-1
votes
1answer
373 views
Bash script to read a web page list from a text file
I want to read webpage list and check if some of them have updated. Is it better to use wget or curl and how should I do that?
Webpage list is in a simple text file. If the contents of a webpage is ...
0
votes
2answers
199 views
wget and scp as a pipeline
Can we use wget and scp as a pipeline. I want to wget a file from a server and copy to another server. I used below command but it is not working.
wget "$Select_Release_Version_ARTIFACT_URL" | sudo ...
0
votes
1answer
55 views
Different page from a browser and wget [closed]
EDITED TO REMOVE OBSOLETE EXAMPLES
I have a very curious case that has dumbfounded me.
I was trying to get a certificate from letsencrypt and I was consistently getting errors that certbot is unable ...
0
votes
1answer
215 views
Sometimes wget just hangs, when it should just restart with --continue
Sometimes wget just hangs in the middle of a download:
50% [===================> ] 500,000 500KB/s eta 5m 5s
I just end up doing a wget --continue to restart it, and it seems ...
0
votes
1answer
234 views
wget not excluding nested directory
I have seen this but it's not working. The remote/ftp directory structure is this:
domain.com/
toplevel/
foo1/
ARCHIVE/
README.stuff
DATA/
README.txt
.....
-3
votes
1answer
2k views
Can not download from an HTTPS site (ssl) with wget?
I know my version of wget is very old, but I can't get a download to work with it,
# wget https://github.com/radare/radare2/archive/master.zip
--2018-11-05 00:07:51-- https://github.com/radare/...
2
votes
0answers
273 views
wget a zip file from GitHub, OpenSSL error [closed]
$ wget -O sindresorhus-is-online-master.zip https://github.com/sindresorhus/is-online/archive/master.zip
--2018-11-03 13:48:20-- https://github.com/sindresorhus/is-online/archive/master.zip
Resolving ...
3
votes
1answer
78 views
If wget follows breadth-first or depth-first
In running wget, I would have expected it to "follow the links", as in depth-first: when it encounters a link, that is the next thing it fetches. But it seems that it first somehow processes the first ...
1
vote
1answer
23 views
How to have wget only follow a small set of paths
Trying to get wget to handle a sort of complicated situation. Basically I want it to only allow files/folders that fit this system:
https://example.com/top/a-file
https://example.com/top/a-file.ext
...
1
vote
1answer
238 views
How to get frame source using wget?
I am trying to get html data from servicenow website. It's using frames and I would like to extract the frame source using wget. I am unable to find the option/flags to do this job. I am only able to ...
-2
votes
1answer
249 views
How to mirror/download a complete website with a browser? [closed]
With wget you can mirror complete websites with: wget --mirror --convert-links --adjust-extension --page-requisites website
Is there some equivalent way for using a browser like Firefox or Chromium ...
3
votes
2answers
228 views
Wget multiple FTP files without re-logging in on every file
Lets say there's a ftp site with a bunch of files I want to download - ftp://ftp.example.com, and they allow for an "anonymous login" over FTP.
Suppose that ftp://ftp.example.com has
file1.zip,
...
0
votes
3answers
915 views
Verify if a URL exists
I would like to verify if a URL exists without downloading. I am using below with curl:
if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null;
then
echo "This page exists."
else
...
1
vote
0answers
56 views
wget behaves differently on cygwin and centos
On cygwin this command works as expected and exits immediately:
wget --post-file oryxblock.2018-10-10_20_40-07_591.204.txt --output-document - --header "Content-Type: text/csv" http://htm-psycho-402....
1
vote
2answers
303 views
Using wget on a directory outside the user's home directory
I'm trying to mirror a directory via FTP with wget. The command I'm using is
wget -m ftp://user:pass@192.168.1.1/foo/bar/
But, when I run it, I get the following:
--2018-10-10 15:01:32-- ftp://...
0
votes
1answer
94 views
Wget for puppet installation
I'm going to install puppet (puppet master) in a CentOs v7 Server which is running on google cloud. I am facing a issue with downloading the puppet tarball into my server.
Once I use below command,
...
