Questions tagged [wget]
wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)
6
votes
1answer
243 views
Wget: downloading files selectively and recursively?
Question about wget, subfolder, and index.html.
Let's say I am inside "travels/" folder and this is in "website.com": "website.com/travels/".
Folder "travels/" contains a lot of files and other (sub)...
-1
votes
0answers
27 views
yum -y install wget error
My environment is Windows 10 ;virtual machine ;CentOS 7
When I want to install wget,I get this error:
Could anybody tell me what to do? I Googled and and I didn't find anything to work , for ...
1
vote
1answer
19 views
Get status error from wget if remote file is not newer than local
When doing wget -c http://example.com/foo to periodically update a file, is there any way to get an error status from wget if the file has not been updated since last download?
Something like:
$ ...
1
vote
0answers
23 views
Watch multiple bash commands with dynamic output
I'm doing multiple downloads at the same time with wget, as this:
while IFS= read _addr; do
coproc wget "http://<host>/<subdir>/$_addr"
done < <(cat filelist.txt)
coproc puts ...
1
vote
2answers
55 views
Streaming Data to REST Service Using Unix Pipes
Based on an answer to another question I am using curl to stream the stdout of one process as the entity of a POST request:
myDataGeneratingApp \
| curl -H "Content-Type: application/json" -H "...
1
vote
1answer
33 views
How to download all images with the full resolution (i.e. svg) from a single wikipedia article [duplicate]
I was asking myself how to download
all (.svg) images
from a single wikipedia article
without doing it manually for each single image. Wikipedia is offering images in different resolutions and ...
1
vote
1answer
27 views
Variable is not found by wgrep inside of script
I'm currently re-learning shell scripting. I'm making a script that checks r/EarthPorn and randomly pick a post, go to that post and download the image. Then set it as a background.
For some reason, ...
0
votes
1answer
27 views
Problem with wget and Whitespace [duplicate]
I have a bash script that scrapes a list of urls for links to various kinds of documents. At the end, the script uses wget to download the files. However, I'm having trouble with filenames containing ...
0
votes
1answer
36 views
How could --compression be missing from my wget?
I have experience with the Windows version:
GNU Wget 1.19.4 built on mingw32.
But now on Ubuntu, I have:
GNU Wget 1.19.4, a non-interactive network retriever.
There is a relatively new option ...
1
vote
1answer
20 views
Is it feasible to combine --span-hosts with --no-parent in wget?
I have pages on a site that are a directory of a huge amount of links to another site. I don't want to download anything from the origin site that is the parent of such directory. I prefer to download ...
1
vote
0answers
46 views
Why does wget --mirror not work on this site?
Why is
wget -m "httpS://www.addnature.se"
refused while
wget -m "http://www.addnature.se"
just downloads the start page?
I frequently use wget -m to mirror sites and have never experienced this ...
1
vote
0answers
29 views
How to find the real download URL
I need to download the 64-bit Linux tar file from this webpage:
https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm
The 64-bit Linux URL appears to be:
https://www....
1
vote
0answers
19 views
openssl hangs on server
I have a problem on the server. Commands wget, curl and dependent commands are not working when a request is via 443 port.
Commands hangs on 2-3 minutes and after just say "time out":
# curl -v ...
1
vote
1answer
32 views
Making wget rename existing filenames sanely
When filename already exists using wget, it renames the file to filename.extension.1. Is it possible to make it rename it to filename.1.extension -- which should have been the default?
0
votes
0answers
74 views
How do I download all images from a webpage with wget
I want to download all images from a webpage but when I type:
$ wget -nd -r -P images -A "*.jpg, *.jpeg, *.png" "Path/to/url"
wget not only downloads images from the webpage I want but it also ...
3
votes
2answers
46 views
wget saving files as emply plain text files on boot
I have a bash script in crontab that runs @reboot:
The script itself contains a wget command to pull and download a file from the internet
When I run my script after signing in and opening the ...
1
vote
1answer
22 views
wget not resolving relative links properly?
Back to basics. I have a static web where the links are typically relative.
All works fine in browsers and in Google Crawler, but wget --spider resolve relative links against the site root.
That is, ...
0
votes
2answers
58 views
Make wget create an empty file for everything it downloads?
I am using wget to copy a directory structure from a web server I don't control. I don't care about the content of each resource on this webpage, I just want to create directories and files to mirror ...
2
votes
2answers
46 views
Select a portion of text from `wget` output and log to a file?
I have this small script to test a FTP site:
#!/bin/bash
wget -O /dev/null ftp://someftpsite:password@ftp.test.com/testdump300 2>&1 | \
grep '\([0-9.]\+ [M]B/s\)' >> wget300.log
And ...
0
votes
1answer
31 views
Download website with page requisites but only images and css
I'm using a wget command which downloads an entire webpage with all assets, requisites and save it to custom folder for a later work on it.
My actual code:
$ wget --adjust-extension --span-hosts --...
0
votes
0answers
22 views
What is the fastest tool to sync http listening files
I have a huge number of files (text files) listening on http server.
I am using cron-job to download these files as the number of files will grow dramatically.
I am using wget as follows to download ...
0
votes
1answer
61 views
wget, curl commands accept partial filenames
I am trying to download a file hpraid.tar.gz from a remote server using
wget --no-check-certificate http://xx.xx.xx/hpraid.tar
curl -O http://xx.xx.xx/hpraid.tar
Even though the .gz extension is ...
0
votes
1answer
305 views
How to download multiple files at a time using mget command from FTP Server without pressing y everytime?
I want to download multiple files from a FTP server (Android app). I used FTP client and mget command (Ubuntu Terminal). But it prompts me to enter y or n for every file that I want to download. I ...
2
votes
1answer
70 views
wget doesn't download images of a website?
I'm trying to download threads of a web forum which contains many images with wget. These images are hosted in a separate domain, so I used the -H flag to allow downloading them. The complete wget ...
1
vote
1answer
41 views
wget: how to download a served file keeping its name
A little IoT server returns a file, whose name is given by current date and time, to make it unique. The format is 2018.07.04.18.22.45.dat.
Asking for address XX.XX.XX.XX:5001/read in a browser (with ...
0
votes
1answer
158 views
Download files from different directories with wildcards using wget?
I need to download all files with "h11v10" in the name from a http server, from different directories (DF). I tried different ways but none of them worked.
wget --http-user user --http-password pass ...
0
votes
0answers
33 views
wget with acceptance test, redirect fails the test
I'm running the command:
wget -r --wait=2 --random-wait -P ./pdfs -A pdf -e robots=off https://www.<somesite>/documents
The command has worked on other sites to recursively download all pdfs, ...
0
votes
0answers
27 views
wget: RETR *file* … done. Nothing happens
Until Friday this worked:
wget --no-passive-ftp ftp://xxx/file.dump
Downloaded the file, all good. But now it just does not exit to the CLI, just hangs in the last line and nothing is downloaded:
--...
0
votes
1answer
43 views
How do I output what wget retrieves to stdout and suppress all other wget messaging?
I'm using bash shell on Amazon Linux. I have a command in a shell script
wget -O - "http://localhost:8088/subco/books/$e_id/segments/$segment_id?product=$product_id&audience=teacher" > /dev/...
0
votes
2answers
247 views
How to know the URL when I submit a login form using POST method?
I want to download the html page using linux command wget
Example:
wget "https://stackexchange.com/users/5215687/dipankar-nalui?tab=inbox"
To download this page I need to send the login ...
1
vote
0answers
185 views
wget can't connect to domain on same server
I'm trying to run the following command on the server. The site example.com is hosted on the same server.
wget --spider "https://example.com/script.php?actions=1"
Response:
Spider mode enabled. ...
-1
votes
1answer
34 views
Using Wget to download files in “members only” that require password [closed]
I am almost finished with our new web site and a customer of ours told me today about Wget. He went on to say if someone used Wget they can log in and download the files that we have in a "members ...
0
votes
0answers
105 views
download with curl no direct download files
I want to download installer file from sites like https://evernote.com/download or https://www.dropbox.com/install#downloaded with curl, but I don't know how to do it to download no direct link with ...
1
vote
0answers
54 views
wget regex options ignored over ftp
Wget specifies the --accept-regex and --reject-regex options for limiting recursive downloads. I'm unable to get them to work with an ftp connection.
For example, I would expect --reject-regex='.*' ...
0
votes
0answers
35 views
parse wget html
I would like to parse all the files downloaded from wget in html format, from a url.
I want to pass a complete web page to docx format, but we are talking about 3000 html documents downloaded from ...
0
votes
0answers
114 views
'No such file' when mirroring ftp
When mirroring FTP using wget I keep getting No such file, no matter which folder I choose.
For example:
$ wget -m ftp://ftp.osuosl.org/pub/facebook
--2018-04-22 14:13:42-- ftp://ftp.osuosl.org/pub/...
0
votes
3answers
46 views
wget only parent pages using {..}
I have been trying to download specific pages in website.
The site uses common URL to go to next pages like below.
https://example.com/pages/?p=1
https://example.com/pages/?p=2
https://example.com/...
0
votes
1answer
432 views
gpg: can't open `–': “No such file or directory” during “wget … apt-key add” upgrading Kali Linux
I tried to upgrade my Kali Linux by following their steps.
On this command:
wget -q -O - https://archive.kali.org/archive-key.asc | apt-key add
I got output:
gpg: can't open `–': No such file or ...
0
votes
1answer
30 views
How to FTP directly to a sub-directory?
An FTP site allows anonymous access to a particular directory, and I would like to download a file from it.
PureFTP and wget keep trying to login to the server's root directory, even when I give wget ...
2
votes
1answer
173 views
Wget segfault---how do I know which site is causing this?
I'm trying to mirror a website locally. However, I've been running into a segmentation fault at some consistent point in the download, on a different domain than the site I'm targeting (probably due ...
1
vote
3answers
94 views
How can I wget from a list with multiple lines into one file name?
I would like to wget a list of items that I'm retrieving from an XML file.
I'm using sed to clean up the XML, and I'm ending up with output like this:
CountofMonteCristo.zip
English.
http://www....
0
votes
0answers
108 views
How can I download a PPD (printer driver) from openprinting in a script?
This is for my Raspberry Pi 3 script to add a printer to a new system (to save time the next time I break it).
I want to automate the driver acquisition process using wgetor curl. Any ideas?
Attempt
...
0
votes
1answer
109 views
wget output based on regex
I have a very long list of files in the following format:
https://server.com/Nonsense%20text%20Actual%20Filename01.ext<br>
https://server.com/Non%20essential%20text%20Actual%20Filename02.ext<...
-1
votes
1answer
299 views
Bash wget with variables and wildcard
I've got a file despicting variables:
IP_1="xx.xx.xx.xx" ID_1="yyyyy"
IP_2="xx.xx.xx.xx" ID_2="yyyyy"
...
IP_6="xx.xx.xx.xx" ID_6="yyyyy"
I want to execute a wget command with ...
0
votes
0answers
83 views
How to wget directory files by skipping index 301 redirect
I want to get the files in Domain.com/blah/"[hashed-random-digits].pdf".
Tried
wget -m -e robots=off -np --user-agent=Mozilla/5.0 --max-redirect 0 domain.com/blah/
But it redirects me to root ...
0
votes
1answer
166 views
Limit the number of file downloads with wget?
I am trying to download all pdf files from a website using wget. I only require n files to be downloaded. How can I tell wget to stop after downloading n files?
0
votes
0answers
29 views
File from wget in script not available for next command
I am wget-ing a file and to preserve its original file name i do a find for its known extension.
Everything is working fine on the terminal but in a script the file is never available/there for the ...
0
votes
0answers
39 views
throughput limitation of single TCP session when small RWIN does not seem to be the cause
I have a server with 10GigE NIC connected to ISP network. Bandwidth provided by ISP for this server is also 10Gbps. When I wget a 100GB file from remote server(single TCP session), then I can see the ...
-1
votes
3answers
71 views
Download the part of the web page [duplicate]
I would like to "parse" some list of web pages (html), the "parsing" is only about redirecting what's between <title> </title> tags to some specific files.
Firstly I did for cycle with ...
5
votes
3answers
2k views
Downloading a file from Terminal
I am trying to download a .txt file using terminal. Here is the link
(the download starts immediately you open the link):
https://es.osdn.net/projects/sfnet_kaldi/downloads/wsj0-train-spkrinfo.txt
...
