Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

51
votes
8answers
131k views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget "https://drive.google.com/folderview?id=0B-Zc9K0k9q-WWUlqMXAyTG40MjA&usp=sharing"...
2
votes
2answers
2k views

wget recursive with files without extension

I'm trying to get a site with wget. The problem is that it: Have user friendly name for the pages http://domain/wiki/Section/Home, http://domain/wiki/Section/Not+Home http://domain/wiki/Section/...
0
votes
3answers
1k views

How to also download duplicate images?

I have a list of 900 URLs. Each page contains one image. Some images are duplicates (with same URL). I want to download 900 images, including duplicates. I was able to download all pages and the ...
1
vote
1answer
5k views

How can I update certificates on my system to fix wget's error: “Unable to establish SSL connection.”?

Sometimes I notice that I am unable to connect via SSL with curl and wget to websites that have certificates that should be valid. One example might be https://squareup.com/ which has a Verisign ...
1
vote
1answer
681 views

wget not downloading files recursively

I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line: wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/...
8
votes
3answers
44k views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
2
votes
1answer
782 views

Using wget to download images in facebook conversations

This looks a bit harder than usual. I'm a new wget user and I was thinking that maybe it's possible to login into Facebook and download images from a specific conversation. I mean, I'd like to tell ...
1
vote
1answer
124 views

remove the bit per bit log of wget

I have a php script that generates bash commands to wget files. I am trying to log wget activity using this: --directory-prefix='xxx' --append-output='xxx.txt' -c -t0 -S" But the log is too big due ...
1
vote
2answers
393 views

Downloading nested pdf files with wget

I am trying to download dozens of PDF files located on pages linked from here: http://machineknittingetc.com/passap.html?limit=all Each PDF is referred to by a URL ending with /downloadable/download/...
0
votes
1answer
30 views

How can `wget` get more logs when it gets error

I have several questions about wget tool in linux: # wget http://www.jdfschool.com --2019-04-05 02:33:44-- (Trys: 3) http://www.jdfschool.com/ Connecting www.jdfschool.com|115.28.223.13|:80... ...
1
vote
1answer
25 views

How to update a local mirror from a public server?

I already have a mirror of the entire slackware64-current/ directory tree using wget -r -np -R "index.html*" https://mirror.slackbuilds.org/slackware/slackware64-current/ and that's fine. But it's ...
1
vote
1answer
959 views

How do I use wget or CURL in php to grab a snapshot of a web page with an asterick * in its URL? (Wayback Machine archive.org question)

The following web page on the Wayback Machine (archive.org) has an asterick * in its URL: https://web.archive.org/web/*/https://www.nasa.gov/ Neither CURL nor wget will allow me to grab the page. I ...
1
vote
2answers
3k views

How to get http server response in case of 401 using wget?

I need to test my http server responses in various cases, even for failed authentication. In case of failed authentication, my server returns 401 Unauthorized and also a response body which simple ...
1
vote
2answers
17k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
0
votes
0answers
29 views

How to receive data from a Webpage and send it back to it in the same connection?

I am trying to fill a form of a webpage with curl and give it some arguments. Dumb enough, it also sends a hidden form in which there is an unique token which has to be sent back to the page, a so ...
-1
votes
1answer
42 views

How to wget mirror website faster?

when I use wget -m url it downloads one file at a time. How to make wget download multiple files at a time?
0
votes
0answers
34 views

Wget wildcard files without temp html files

Hello I've been trying to find the correct solution to download wildcard files using wget - I used to have nexus api calls to download the latest files, but unfortunately I have to use pattern ...
0
votes
2answers
948 views

403 http code appears when I go to a certain website

I wrote a PHP script to a particular site using a home computer for debugging. Script worked properly, when I tested it on a home computer. The script has stopped working when I deploy the script on ...
1
vote
1answer
380 views

Installing Apache Tomcat on CentOS giving read error on Wget

I have a remote VM to which I connect using SSH. Now i need to install Tomcat 8 for which I run wget command wget http://mirror.fibergrid.in/apache/tomcat/tomcat-8/v8.5.14/bin/apache-tomcat-8.5.14....
2
votes
2answers
5k views

bash: wget: command not found

I'm using ASUS-Chromebook-Flip-C302CA / Google Chrome OS - Version 65.0.3325.35 (Official Build) dev (64-bit) and I'm trying to follow Visual Studio Code for Chromebooks and Raspberry Pi, yet failing ...
3
votes
2answers
965 views

Wget with different filename, but same extension

I need wget to save to a different folder, different filename, and same extension, but I do not have the filename before downloading. For example, let's say I have: http://ge.tt/api/1/files/someFile/...
0
votes
0answers
51 views

wget isn't converting some links properly

I'm trying to get a "working copy" of google.com using wget. The command I use is: wget --page-requisites --html-extension --convert-links --restrict-file-names=windows --no-host-directories -U "...
0
votes
1answer
630 views

How do I download all images from a webpage with wget

I want to download all images from a webpage but when I type: $ wget -nd -r -P images -A "*.jpg, *.jpeg, *.png" "Path/to/url" wget not only downloads images from the webpage I want but it also ...
0
votes
1answer
51 views

Troubleshooting: cURL timeouts, Java app doesn't

I'm trying to use curl against a remote server to access a REST API (and right now, just the welcome page...) Currently, I have a Java application that can send REST request on this server. But if I ...
402
votes
3answers
553k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
2
votes
3answers
884 views

Question about wget -qO-

In the string wget -qO- What exactly am I doing? I've never seen a switch that has a dash after the expression. But, to install Docker, I am using wget -qO- https://get.docker.com | sh
0
votes
2answers
390 views

Owncloud does not show files, that are uploaded via SSH

After spending 10 Hours configuring owncloud on Fedora 24 (because of NO documentation and proper tutorial), I finally got OwnCloud up and running perfectly. My owncloud configuration files are in /...
0
votes
0answers
48 views

calling wget from a browser using php

My httpd is running with a daemon user. I want to do a wget using php from FTP server. When I run the php from linux server, the same downloads the required file from FTP. But when I call the same php ...
230
votes
3answers
117k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
1
vote
0answers
77 views

GET via wget or cURL gives a limited response

I'm bumping my head on a simple HTTP GET request. Using a tool like Postman, I'm getting the full and expected response (an over 1 thousand lines xml document, displaying the details of my 60 albums) ...
52
votes
3answers
51k views

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz ...
0
votes
2answers
102 views

wget not downloading streaming mp3 correctly

I am trying to download a streaming mp3 using wget. this is my basic command: wget http://sj128.hnux.com/sj128.mp3 -c --timeout=1 --waitretry=0 --tries=0 -O "file.mp3" i have been doing this in a ...
47
votes
4answers
162k views

Set a network range in the no_proxy environment variable

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP. The network is 10.0.0.0/8. My proxy is 10.1.1.1:81, so I set it up accordingly: ...
0
votes
2answers
42 views

How to save a web resource, applying decoding to URL Encoded Characters?

I would like to save this file with this URL "http://pti.regione.sicilia.it/portal/page/portal/PIR_PORTALE/PIR_LaStrutturaRegionale/PIR_AssessoratoEconomia/PIR_DipBilancioTesoro/PIR_Areetematiche/...
-1
votes
1answer
26 views

How to download AWS documentation using linux terminal?

I tried to download AWS documentation using wget and curl commands. I realized and turned off robot and set Mozilla as browser header. But still failed to recursively download the entire documents.
4
votes
1answer
2k views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
2
votes
1answer
356 views

Using /dev/tcp with proxy

Is it possible to use /dev/tcp with a proxy? I only found wget is possible to use with proxy, but wget doesn't work.
0
votes
0answers
67 views

wget output progress to file per downloaded file

I'm downloading multiple bigger files with wget like this: wget -i 'FILE_WITH_URLS' --content-disposition -b. By default it creates a file named wget-log and appends time download progresses. The file ...
0
votes
0answers
46 views

wget --spider ram memory consumption and use of -c

I am using wget --spider. It is consuming a lot of ram memory. How could I limit the use of ram memory from the wget command? wget --spider -r -c http://www.example.org 2>&1 | grep '^--' | awk ...
0
votes
1answer
76 views

Set date on embedded linux with limited busybox (no ntp)

I am trying to automatically set a date after boot on an embeddd linux with busybox ( I dont have a RTC). I also dont have ntp command in busybox, nor curl. My wget is very limited too, and I cant ...
0
votes
0answers
205 views

Download filename-pattern-matching files from a public Google Drive folder with rclone (or alternative)

I'm trying to download all files from a public gdrive folder but with certain filetype restrictions (eg exclude *.avi,*.mkv,*.mp4 etc). I originally tried with wget since that is what I am familiar ...
8
votes
3answers
1k views

Why can't I see the “wget” job when I execute it in the background?

I use wget command in the background like this wget -bq and it prints Continuing in background, pid 31754. But when I type the command jobs, I don't see my job(although the downloading is not ...
0
votes
1answer
2k views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
0
votes
1answer
85 views

wget download specific files

I am learning to use the various wget commands (man wget). But I'm having trouble learning how to download specific files from some sites (on the whole site they are downloaded). On some sites I get ...
2
votes
1answer
45 views

wget image problem for daily wallpaper script

i have a script (that Nuno wrote) to wget an image from one particular site and set my wallpaper. however i can't seem to modify it to be able to wget an image from this site: chromecast images in ...
13
votes
4answers
23k views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
6
votes
4answers
10k views

How to monitor WGET download progress after closing SSH session

I have started downloading my ISO's etc directly to my fileserver using wget. After I close the ssh session, how can I check back on the download process? Scenario: I start the download, then shut ...
2
votes
1answer
2k views

How to install the wget package on AIX server 7.1 without having a local repository?

I have remote access to my AIX 7.1 server. It has no local repository. It has internet access but I cannot use wget command in this machine as the wget package is not installed. I downloaded a tar ...
2
votes
1answer
232 views

How to download multiple files from website with wget and perform functions on downloaded file before next file is downloaded

I am using wget to download a number of json files from a website. However, I want to perform some sort of function on these files to reduce their size before downloading the next file. Is there a ...
6
votes
2answers
394 views

'ping' uses localhost instead of public IP address

Let's say my server has IP address 11.22.33.44 and hostname server1.mydomain.com. When I ping server1.mydomain.com, it looks as if ping is actually using the public IP address: # ping server1....