Questions tagged [wget]
wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)
-1
votes
0answers
8 views
How to follow all “HTML links”, and yet only save .zip files
I am trying to do this:
function download() {
wget \
-r \
-c \
-A .zip,.html,.jsp,.php,.cgi \
--reject-regex "(.*)\?(.*)" \
--secure-protocol=TLSv1_2 \
"$1"
}
The problems ...
0
votes
2answers
30 views
wget and scp as a pipeline
Can we use wget and scp as a pipeline. I want to wget a file from a server and copy to another server. I used below command but it is not working.
wget "$Select_Release_Version_ARTIFACT_URL" | sudo ...
0
votes
1answer
25 views
Different page from a browser and wget
EDITED TO REMOVE OBSOLETE EXAMPLES
I have a very curious case that has dumbfounded me.
I was trying to get a certificate from letsencrypt and I was consistently getting errors that certbot is unable ...
0
votes
0answers
15 views
How to handle finding multiple patterns of files/directories in wget
So I have a directory with a million files and 50k directories nested 10 layers deep roughly. It takes a long time simply to go through and fetch a 100 byte file from 1/10 of the directories, because ...
0
votes
0answers
12 views
How to write log of which paths are found using wget
I would like wget to only fetch a few files from a directory tree, but I would at the same time like to list out every single file and directory that is encountered (so I can look through them later ...
0
votes
1answer
23 views
Sometimes wget just hangs, when it should just restart with --continue
Sometimes wget just hangs in the middle of a download:
50% [===================> ] 500,000 500KB/s eta 5m 5s
I just end up doing a wget --continue to restart it, and it seems ...
0
votes
1answer
17 views
wget not excluding nested directory
I have seen this but it's not working. The remote/ftp directory structure is this:
domain.com/
toplevel/
foo1/
ARCHIVE/
README.stuff
DATA/
README.txt
.....
-3
votes
1answer
50 views
Can not download from an HTTPS site (ssl) with wget?
I know my version of wget is very old, but I can't get a download to work with it,
# wget https://github.com/radare/radare2/archive/master.zip
--2018-11-05 00:07:51-- https://github.com/radare/...
2
votes
0answers
76 views
wget a zip file from GitHub, OpenSSL error [closed]
$ wget -O sindresorhus-is-online-master.zip https://github.com/sindresorhus/is-online/archive/master.zip
--2018-11-03 13:48:20-- https://github.com/sindresorhus/is-online/archive/master.zip
Resolving ...
2
votes
1answer
12 views
If wget follows breadth-first or depth-first
In running wget, I would have expected it to "follow the links", as in depth-first: when it encounters a link, that is the next thing it fetches. But it seems that it first somehow processes the first ...
1
vote
1answer
18 views
How to have wget only follow a small set of paths
Trying to get wget to handle a sort of complicated situation. Basically I want it to only allow files/folders that fit this system:
https://example.com/top/a-file
https://example.com/top/a-file.ext
...
1
vote
1answer
31 views
How to get frame source using wget?
I am trying to get html data from servicenow website. It's using frames and I would like to extract the frame source using wget. I am unable to find the option/flags to do this job. I am only able to ...
-2
votes
1answer
44 views
How to mirror/download a complete website with a browser? [closed]
With wget you can mirror complete websites with: wget --mirror --convert-links --adjust-extension --page-requisites website
Is there some equivalent way for using a browser like Firefox or Chromium ...
2
votes
2answers
82 views
Wget multiple FTP files without re-logging in on every file
Lets say there's a ftp site with a bunch of files I want to download - ftp://ftp.example.com, and they allow for an "anonymous login" over FTP.
Suppose that ftp://ftp.example.com has
file1.zip,
...
0
votes
3answers
40 views
Verify if a URL exists
I would like to verify if a URL exists without downloading. I am using below with curl:
if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null;
then
echo "This page exists."
else
...
1
vote
0answers
31 views
wget behaves differently on cygwin and centos
On cygwin this command works as expected and exits immediately:
wget --post-file oryxblock.2018-10-10_20_40-07_591.204.txt --output-document - --header "Content-Type: text/csv" http://htm-psycho-402....
1
vote
2answers
41 views
Using wget on a directory outside the user's home directory
I'm trying to mirror a directory via FTP with wget. The command I'm using is
wget -m ftp://user:pass@192.168.1.1/foo/bar/
But, when I run it, I get the following:
--2018-10-10 15:01:32-- ftp://...
0
votes
1answer
31 views
Wget for puppet installation
I'm going to install puppet (puppet master) in a CentOs v7 Server which is running on google cloud. I am facing a issue with downloading the puppet tarball into my server.
Once I use below command,
...
0
votes
1answer
34 views
Fetching title of browser
[Fetching user title]
Seen a post prior of someone posting how to do it by single account
I wish to do this from the range 1-1000 and have it output to >> .txt
Example:
x@nine:~# wget -qO- 'http:...
1
vote
1answer
44 views
yum install fails with 401 Authorization required [closed]
I am running Red Hat Linux 6.2
Trying to install nfs packages using yum
Following is the
repository list
repo id repo name ...
4
votes
1answer
370 views
Wget: downloading files selectively and recursively?
Question about wget, subfolder, and index.html.
Let's say I am inside "travels/" folder and this is in "website.com": "website.com/travels/".
Folder "travels/" contains a lot of files and other (sub)...
1
vote
1answer
33 views
Get status error from wget if remote file is not newer than local
When doing wget -c http://example.com/foo to periodically update a file, is there any way to get an error status from wget if the file has not been updated since last download?
Something like:
$ ...
0
votes
0answers
33 views
Watch multiple bash commands with dynamic output
I'm doing multiple downloads at the same time with wget, as this:
while IFS= read _addr; do
coproc wget "http://<host>/<subdir>/$_addr"
done < <(cat filelist.txt)
coproc puts ...
1
vote
2answers
154 views
Streaming Data to REST Service Using Unix Pipes
Based on an answer to another question I am using curl to stream the stdout of one process as the entity of a POST request:
myDataGeneratingApp \
| curl -H "Content-Type: application/json" -H "...
1
vote
1answer
51 views
How to download all images with the full resolution (i.e. svg) from a single wikipedia article [duplicate]
I was asking myself how to download
all (.svg) images
from a single wikipedia article
without doing it manually for each single image. Wikipedia is offering images in different resolutions and ...
1
vote
1answer
27 views
Variable is not found by wgrep inside of script
I'm currently re-learning shell scripting. I'm making a script that checks r/EarthPorn and randomly pick a post, go to that post and download the image. Then set it as a background.
For some reason, ...
0
votes
1answer
46 views
Problem with wget and Whitespace [duplicate]
I have a bash script that scrapes a list of urls for links to various kinds of documents. At the end, the script uses wget to download the files. However, I'm having trouble with filenames containing ...
0
votes
1answer
77 views
How could --compression be missing from my wget?
I have experience with the Windows version:
GNU Wget 1.19.4 built on mingw32.
But now on Ubuntu, I have:
GNU Wget 1.19.4, a non-interactive network retriever.
There is a relatively new option ...
1
vote
1answer
37 views
Is it feasible to combine --span-hosts with --no-parent in wget?
I have pages on a site that are a directory of a huge amount of links to another site. I don't want to download anything from the origin site that is the parent of such directory. I prefer to download ...
1
vote
0answers
47 views
Why does wget --mirror not work on this site?
Why is
wget -m "httpS://www.addnature.se"
refused while
wget -m "http://www.addnature.se"
just downloads the start page?
I frequently use wget -m to mirror sites and have never experienced this ...
1
vote
0answers
30 views
How to find the real download URL
I need to download the 64-bit Linux tar file from this webpage:
https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm
The 64-bit Linux URL appears to be:
https://www....
1
vote
0answers
28 views
openssl hangs on server
I have a problem on the server. Commands wget, curl and dependent commands are not working when a request is via 443 port.
Commands hangs on 2-3 minutes and after just say "time out":
# curl -v ...
1
vote
1answer
67 views
Making wget rename existing filenames sanely
When filename already exists using wget, it renames the file to filename.extension.1. Is it possible to make it rename it to filename.1.extension -- which should have been the default?
0
votes
0answers
338 views
How do I download all images from a webpage with wget
I want to download all images from a webpage but when I type:
$ wget -nd -r -P images -A "*.jpg, *.jpeg, *.png" "Path/to/url"
wget not only downloads images from the webpage I want but it also ...
3
votes
2answers
67 views
wget saving files as emply plain text files on boot
I have a bash script in crontab that runs @reboot:
The script itself contains a wget command to pull and download a file from the internet
When I run my script after signing in and opening the ...
1
vote
1answer
55 views
wget not resolving relative links properly?
Back to basics. I have a static web where the links are typically relative.
All works fine in browsers and in Google Crawler, but wget --spider resolve relative links against the site root.
That is, ...
1
vote
4answers
105 views
Make wget create an empty file for everything it downloads?
I am using wget to copy a directory structure from a web server I don't control. I don't care about the content of each resource on this webpage, I just want to create directories and files to mirror ...
2
votes
2answers
98 views
Select a portion of text from `wget` output and log to a file?
I have this small script to test a FTP site:
#!/bin/bash
wget -O /dev/null ftp://someftpsite:password@ftp.test.com/testdump300 2>&1 | \
grep '\([0-9.]\+ [M]B/s\)' >> wget300.log
And ...
0
votes
1answer
44 views
Download website with page requisites but only images and css
I'm using a wget command which downloads an entire webpage with all assets, requisites and save it to custom folder for a later work on it.
My actual code:
$ wget --adjust-extension --span-hosts --...
0
votes
0answers
25 views
What is the fastest tool to sync http listening files
I have a huge number of files (text files) listening on http server.
I am using cron-job to download these files as the number of files will grow dramatically.
I am using wget as follows to download ...
0
votes
1answer
81 views
wget, curl commands accept partial filenames
I am trying to download a file hpraid.tar.gz from a remote server using
wget --no-check-certificate http://xx.xx.xx/hpraid.tar
curl -O http://xx.xx.xx/hpraid.tar
Even though the .gz extension is ...
0
votes
1answer
635 views
How to download multiple files at a time using mget command from FTP Server without pressing y everytime?
I want to download multiple files from a FTP server (Android app). I used FTP client and mget command (Ubuntu Terminal). But it prompts me to enter y or n for every file that I want to download. I ...
2
votes
1answer
164 views
wget doesn't download images of a website?
I'm trying to download threads of a web forum which contains many images with wget. These images are hosted in a separate domain, so I used the -H flag to allow downloading them. The complete wget ...
1
vote
1answer
91 views
wget: how to download a served file keeping its name
A little IoT server returns a file, whose name is given by current date and time, to make it unique. The format is 2018.07.04.18.22.45.dat.
Asking for address XX.XX.XX.XX:5001/read in a browser (with ...
0
votes
1answer
250 views
Download files from different directories with wildcards using wget?
I need to download all files with "h11v10" in the name from a http server, from different directories (DF). I tried different ways but none of them worked.
wget --http-user user --http-password pass ...
0
votes
0answers
48 views
wget with acceptance test, redirect fails the test
I'm running the command:
wget -r --wait=2 --random-wait -P ./pdfs -A pdf -e robots=off https://www.<somesite>/documents
The command has worked on other sites to recursively download all pdfs, ...
0
votes
0answers
36 views
wget: RETR *file* … done. Nothing happens
Until Friday this worked:
wget --no-passive-ftp ftp://xxx/file.dump
Downloaded the file, all good. But now it just does not exit to the CLI, just hangs in the last line and nothing is downloaded:
--...
0
votes
1answer
56 views
How do I output what wget retrieves to stdout and suppress all other wget messaging?
I'm using bash shell on Amazon Linux. I have a command in a shell script
wget -O - "http://localhost:8088/subco/books/$e_id/segments/$segment_id?product=$product_id&audience=teacher" > /dev/...
0
votes
2answers
489 views
How to know the URL when I submit a login form using POST method?
I want to download the html page using linux command wget
Example:
wget "https://stackexchange.com/users/5215687/dipankar-nalui?tab=inbox"
To download this page I need to send the login ...
1
vote
0answers
275 views
wget can't connect to domain on same server
I'm trying to run the following command on the server. The site example.com is hosted on the same server.
wget --spider "https://example.com/script.php?actions=1"
Response:
Spider mode enabled. ...
