Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

0
votes
1answer
20 views

Set date on embedded linux with limited busybox (no ntp)

I am trying to automatically set a date after boot on an embeddd linux with busybox ( I dont have a RTC). I also dont have ntp command in busybox, nor curl. My wget is very limited too, and I cant ...
0
votes
0answers
24 views

Download filename-pattern-matching files from a public Google Drive folder with rclone (or alternative)

I'm trying to download all files from a public gdrive folder but with certain filetype restrictions (eg exclude *.avi,*.mkv,*.mp4 etc). I originally tried with wget since that is what I am familiar ...
8
votes
3answers
916 views

Why can't I see the “wget” job when I execute it in the background?

I use wget command in the background like this wget -bq and it prints Continuing in background, pid 31754. But when I type the command jobs, I don't see my job(although the downloading is not ...
0
votes
1answer
33 views

wget download specific files

I am learning to use the various wget commands (man wget). But I'm having trouble learning how to download specific files from some sites (on the whole site they are downloaded). On some sites I get ...
1
vote
1answer
26 views

wget image problem for daily wallpaper script

i have a script (that Nuno wrote) to wget an image from one particular site and set my wallpaper. however i can't seem to modify it to be able to wget an image from this site: chromecast images in ...
2
votes
1answer
43 views

How to download multiple files from website with wget and perform functions on downloaded file before next file is downloaded

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file. Is there a ...
5
votes
2answers
371 views

'ping' uses localhost instead of public IP address

Let's say my server has IP address 11.22.33.44 and hostname server1.mydomain.com. When I ping server1.mydomain.com, it looks as if ping is actually using the public IP address: # ping server1....
1
vote
2answers
37 views

Can wget download reveal.js image assets?

In using the wget command below, $ wget \ --recursive \ --no-clobber \ --page-requisites \ --html-extension \ --convert-links \ --restrict-file-names=windows \ --...
0
votes
1answer
52 views

command to download & agree licensed software

I am writing a script that download this software, then install and run it on linux system. Issue is when I do wget to download, my download doesn't initiate, it halts showing below line, because ...
0
votes
1answer
55 views

how to use Wget for download all pdf from a website? [on hold]

I tried to download all the pdf from this website https://www.restituciondetierras.gov.co/sentencias-por-departamento` but I couldn't I use that: wget -r -e robots=off -A.pdf https://www....
0
votes
1answer
60 views

wget download regex pattern files from remote URL

I want to download all *httpd* RPM file from remote CentOS mirror and I am trying he following command but it doesn't seem to work [root@yum foo]# wget -r --no-parent -A "*httpd*" https://mirrors....
2
votes
3answers
60 views

Security issues with “wget”

I was wondering if there are any vulnerabilities that I may face when using "wget" and its options. And specifically, I've used a benchmarking script to test the performance of my VPS. The command is ...
0
votes
1answer
61 views

How to download https file in AIX without wget and curl? [duplicate]

In an UNIX environment running AIX where I'm a not privileged user (without sudo) if I need to download a file from HTTPS server and there is no wget or curl... How I do it? Is there any replacement ...
2
votes
1answer
85 views

Alternatives to cURL/wget for remote API calls

I'm on IBM AIX 7.1 where cURL and wget are not available. Also, I'm not in a position to install either. Are there alternatives that would be available that would enable remote API calls using GET/...
-1
votes
1answer
52 views

Bash script to read a web page list from a text file

I want to read webpage list and check if some of them have updated. Is it better to use wget or curl and how should I do that? Webpage list is in a simple text file. If the contents of a webpage is ...
0
votes
2answers
46 views

wget and scp as a pipeline

Can we use wget and scp as a pipeline. I want to wget a file from a server and copy to another server. I used below command but it is not working. wget "$Select_Release_Version_ARTIFACT_URL" | sudo ...
0
votes
1answer
40 views

Different page from a browser and wget [closed]

EDITED TO REMOVE OBSOLETE EXAMPLES I have a very curious case that has dumbfounded me. I was trying to get a certificate from letsencrypt and I was consistently getting errors that certbot is unable ...
0
votes
0answers
20 views

How to handle finding multiple patterns of files/directories in wget

So I have a directory with a million files and 50k directories nested 10 layers deep roughly. It takes a long time simply to go through and fetch a 100 byte file from 1/10 of the directories, because ...
0
votes
0answers
15 views

How to write log of which paths are found using wget

I would like wget to only fetch a few files from a directory tree, but I would at the same time like to list out every single file and directory that is encountered (so I can look through them later ...
0
votes
1answer
52 views

Sometimes wget just hangs, when it should just restart with --continue

Sometimes wget just hangs in the middle of a download: 50% [===================> ] 500,000 500KB/s eta 5m 5s I just end up doing a wget --continue to restart it, and it seems ...
0
votes
1answer
34 views

wget not excluding nested directory

I have seen this but it's not working. The remote/ftp directory structure is this: domain.com/ toplevel/ foo1/ ARCHIVE/ README.stuff DATA/ README.txt .....
-3
votes
1answer
321 views

Can not download from an HTTPS site (ssl) with wget?

I know my version of wget is very old, but I can't get a download to work with it, # wget https://github.com/radare/radare2/archive/master.zip --2018-11-05 00:07:51-- https://github.com/radare/...
2
votes
0answers
131 views

wget a zip file from GitHub, OpenSSL error [closed]

$ wget -O sindresorhus-is-online-master.zip https://github.com/sindresorhus/is-online/archive/master.zip --2018-11-03 13:48:20-- https://github.com/sindresorhus/is-online/archive/master.zip Resolving ...
2
votes
1answer
19 views

If wget follows breadth-first or depth-first

In running wget, I would have expected it to "follow the links", as in depth-first: when it encounters a link, that is the next thing it fetches. But it seems that it first somehow processes the first ...
1
vote
1answer
19 views

How to have wget only follow a small set of paths

Trying to get wget to handle a sort of complicated situation. Basically I want it to only allow files/folders that fit this system: https://example.com/top/a-file https://example.com/top/a-file.ext ...
1
vote
1answer
62 views

How to get frame source using wget?

I am trying to get html data from servicenow website. It's using frames and I would like to extract the frame source using wget. I am unable to find the option/flags to do this job. I am only able to ...
-2
votes
1answer
88 views

How to mirror/download a complete website with a browser? [closed]

With wget you can mirror complete websites with: wget --mirror --convert-links --adjust-extension --page-requisites website Is there some equivalent way for using a browser like Firefox or Chromium ...
2
votes
2answers
117 views

Wget multiple FTP files without re-logging in on every file

Lets say there's a ftp site with a bunch of files I want to download - ftp://ftp.example.com, and they allow for an "anonymous login" over FTP. Suppose that ftp://ftp.example.com has file1.zip, ...
0
votes
3answers
111 views

Verify if a URL exists

I would like to verify if a URL exists without downloading. I am using below with curl: if [[ $(curl ftp://ftp.somewhere.com/bigfile.gz) ]] 2>/dev/null; then echo "This page exists." else ...
1
vote
0answers
34 views

wget behaves differently on cygwin and centos

On cygwin this command works as expected and exits immediately: wget --post-file oryxblock.2018-10-10_20_40-07_591.204.txt --output-document - --header "Content-Type: text/csv" http://htm-psycho-402....
1
vote
2answers
96 views

Using wget on a directory outside the user's home directory

I'm trying to mirror a directory via FTP with wget. The command I'm using is wget -m ftp://user:pass@192.168.1.1/foo/bar/ But, when I run it, I get the following: --2018-10-10 15:01:32-- ftp://...
0
votes
1answer
46 views

Wget for puppet installation

I'm going to install puppet (puppet master) in a CentOs v7 Server which is running on google cloud. I am facing a issue with downloading the puppet tarball into my server. Once I use below command, ...
0
votes
1answer
36 views

Fetching title of browser

[Fetching user title] Seen a post prior of someone posting how to do it by single account I wish to do this from the range 1-1000 and have it output to >> .txt Example: x@nine:~# wget -qO- 'http:...
1
vote
1answer
76 views

yum install fails with 401 Authorization required [closed]

I am running Red Hat Linux 6.2 Trying to install nfs packages using yum Following is the repository list repo id repo name ...
4
votes
1answer
640 views

Wget: downloading files selectively and recursively?

Question about wget, subfolder, and index.html. Let's say I am inside "travels/" folder and this is in "website.com": "website.com/travels/". Folder "travels/" contains a lot of files and other (sub)...
1
vote
1answer
40 views

Get status error from wget if remote file is not newer than local

When doing wget -c http://example.com/foo to periodically update a file, is there any way to get an error status from wget if the file has not been updated since last download? Something like: $ ...
0
votes
0answers
49 views

Watch multiple bash commands with dynamic output

I'm doing multiple downloads at the same time with wget, as this: while IFS= read _addr; do coproc wget "http://<host>/<subdir>/$_addr" done < <(cat filelist.txt) coproc puts ...
1
vote
2answers
405 views

Streaming Data to REST Service Using Unix Pipes

Based on an answer to another question I am using curl to stream the stdout of one process as the entity of a POST request: myDataGeneratingApp \ | curl -H "Content-Type: application/json" -H "...
1
vote
1answer
87 views

How to download all images with the full resolution (i.e. svg) from a single wikipedia article [duplicate]

I was asking myself how to download all (.svg) images from a single wikipedia article without doing it manually for each single image. Wikipedia is offering images in different resolutions and ...
1
vote
1answer
27 views

Variable is not found by wgrep inside of script

I'm currently re-learning shell scripting. I'm making a script that checks r/EarthPorn and randomly pick a post, go to that post and download the image. Then set it as a background. For some reason, ...
0
votes
1answer
80 views

Problem with wget and Whitespace [duplicate]

I have a bash script that scrapes a list of urls for links to various kinds of documents. At the end, the script uses wget to download the files. However, I'm having trouble with filenames containing ...
0
votes
1answer
125 views

How could --compression be missing from my wget?

I have experience with the Windows version: GNU Wget 1.19.4 built on mingw32. But now on Ubuntu, I have: GNU Wget 1.19.4, a non-interactive network retriever. There is a relatively new option ...
1
vote
1answer
64 views

Is it feasible to combine --span-hosts with --no-parent in wget?

I have pages on a site that are a directory of a huge amount of links to another site. I don't want to download anything from the origin site that is the parent of such directory. I prefer to download ...
1
vote
0answers
53 views

Why does wget --mirror not work on this site?

Why is wget -m "httpS://www.addnature.se" refused while wget -m "http://www.addnature.se" just downloads the start page? I frequently use wget -m to mirror sites and have never experienced this ...
1
vote
0answers
34 views

How to find the real download URL

I need to download the 64-bit Linux tar file from this webpage: https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm The 64-bit Linux URL appears to be: https://www....
1
vote
0answers
39 views

openssl hangs on server

I have a problem on the server. Commands wget, curl and dependent commands are not working when a request is via 443 port. Commands hangs on 2-3 minutes and after just say "time out": # curl -v ...
1
vote
1answer
96 views

Making wget rename existing filenames sanely

When filename already exists using wget, it renames the file to filename.extension.1. Is it possible to make it rename it to filename.1.extension -- which should have been the default?
0
votes
0answers
518 views

How do I download all images from a webpage with wget

I want to download all images from a webpage but when I type: $ wget -nd -r -P images -A "*.jpg, *.jpeg, *.png" "Path/to/url" wget not only downloads images from the webpage I want but it also ...
3
votes
2answers
81 views

wget saving files as emply plain text files on boot

I have a bash script in crontab that runs @reboot: The script itself contains a wget command to pull and download a file from the internet When I run my script after signing in and opening the ...
1
vote
1answer
79 views

wget not resolving relative links properly?

Back to basics. I have a static web where the links are typically relative. All works fine in browsers and in Google Crawler, but wget --spider resolve relative links against the site root. That is, ...