Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

0
votes
0answers
44 views

xargs: wget: No such file or directory

Am not good in unix commands. am downloading files listed in the raw_data_urls.txt to my desktop. My sh command is as below: cat setup_files/raw_data_urls.txt | xargs -n 1 -P 6 wget -c -P data/ ...
-2
votes
1answer
28 views

How to grep for multiple text lines in between unknown and changing html code

Here is my function. ph_relwords () { search="" echo "related words: $@" for term in $@; do search="$search+$term" done wget -q -x "https://www.pornhub.com/video/search?...
0
votes
1answer
23 views

How to get triggered download from site with delay

Normally I use wget to get files that trigger a download. However, I'm currently using a site that takes a few seconds to generate the download, so wget downloads the html instead of the zip file I ...
1
vote
1answer
24 views

wget content-disposition ubuntu vs alpine

In Ubuntu, I can use the following wget flag to get a debian, i.e., wget --content-disposition https://packagecloud.io/xxxxx/download.deb dpkg -i ... In Alpine, wget complains that --content-...
-1
votes
2answers
25 views

How to use wget to download directory recursively, into directory with the same name as on the server?

Let's say I have a URL like so: https://mywebsite.com/files/myfolder How do I download all the files in the myfolder part (which is a directory), excluding files like index files, into a directory ...
0
votes
2answers
49 views

Get web page complete data

I'm using busybox tools and I want to take all http links in a web page. I save an example link page by using curl or wget. However, it saves the page as html. How to do it with curl or wget commands? ...
425
votes
3answers
579k views

Does curl have a --no-check-certificate option like wget?

I am trying to make a curl request to one of our local development servers running a dev site with a self-signed SSL cert. I am using curl from the command line. I saw some blog posts mentioning that ...
0
votes
1answer
911 views

How to handle Last-modified header missing — time-stamps turned off

I was mirroring a site but something happened and wget died. So I had to run it again but I realized that Last-modified header missing -- time-stamps turned off. was repeated so often in the output so ...
-1
votes
1answer
12 views

how to download ebi fastq link via command line using wget/curl

I have loads of data (fastq) to download from ENA database. Example: ERS076383 ERS012262 ERS160574 ERS349280 etc Link to download via browser: https://www.ebi.ac.uk/ena/data/warehouse/filereport?...
241
votes
3answers
125k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
1
vote
0answers
23 views

Recursive wget, append something to each found URL

My main goal is to easily backup the page source for a bunch of MediaWiki pages of mine. I have every page I want downloaded, listed on my username page (https://wiki.example.com/index.php?title=User:...
4
votes
1answer
2k views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
0
votes
1answer
65 views

Unable to wget for ipv6

I tried wget using the following command wget -6 https://[fe80::] But it displayed the following error Connecting to [fe80::]:80... failed: Invalid argument.
0
votes
0answers
34 views

How can I make wget rename files based on link name?

Everything I'm finding is based on URL, but I want the opposite. I have an html index with a list of links. The links are named like Apples, oranges, grapes, bananas. The files associated with those ...
1
vote
1answer
27 views

wget -i notifying when each file finishes (for processing purposes)

I'd like to be able to process multiple files downloaded by wget -i immediately after they are downloaded (instead of waiting for all files in the list to finish--for the entire wget process to exit). ...
0
votes
1answer
2k views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
0
votes
1answer
242 views

wget apache kafka not working

I have Debian 32bit installed and I am using the Terminal (no GUI). I would like to install Apache Kafka, so I am doing: wget https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.12-1.0.0....
1
vote
2answers
193 views

wget download files from a file list, how to specify names for each of the downloaded file?

When downloading a single file, we can use wget's -O option to specify the file name. When I'm downloading Urls in a file using wget -i filelist.txt (filelist.txt contains list Of Urls I want to ...
2
votes
1answer
2k views

How to install the wget package on AIX server 7.1 without having a local repository?

I have remote access to my AIX 7.1 server. It has no local repository. It has internet access but I cannot use wget command in this machine as the wget package is not installed. I downloaded a tar ...
5
votes
2answers
1k views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called mirror.de/�%...
0
votes
1answer
164 views

Download ZIP file of private GitHub repository using `wget` like `curl`

There are already some questions with answers how to download the ZIP file of GitHub repositories, but they don't work if the repository is private. I know it should work with wget somehow since it ...
0
votes
2answers
82 views

Using wget to finish a downloaing

I was downloading a big file but without wget tool, the good news is that the link of the file has the following form: https://link.com/file.zip Sadly, the downloaing was broken, so now I want to ...
55
votes
8answers
136k views

How to download a folder from google drive using terminal?

I want to download a folder from my google drive using terminal? Is there any way to do that? I tried this: $ wget "https://drive.google.com/folderview?id=0B-Zc9K0k9q-WWUlqMXAyTG40MjA&usp=sharing"...
0
votes
2answers
97 views

wget --spider fails with 404 but works without --spider

I'm trying to do a health check in a docker container. I found this command: wget --quiet --tries=1 --spider http://localhost:6077 || exit 1 The issue is that while the container is running, if I ...
15
votes
7answers
16k views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
1
vote
1answer
2k views

curl, wget do not return anything

I am trying this curl -I zomato.com | head -n 1 and I am not getting any response. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 ...
0
votes
0answers
139 views

wget: error getting response: Address family not supported by protocol

Attempt to retrieve entware script from a test router indicates a wget error: root@DD-WRT:/opt# wget http://bin.entware.net/mipselsf-k3.4/installer/generic.sh Connecting to bin.entware.net (104.27....
2
votes
2answers
2k views

wget recursive with files without extension

I'm trying to get a site with wget. The problem is that it: Have user friendly name for the pages http://domain/wiki/Section/Home, http://domain/wiki/Section/Not+Home http://domain/wiki/Section/...
1
vote
3answers
1k views

How to also download duplicate images?

I have a list of 900 URLs. Each page contains one image. Some images are duplicates (with same URL). I want to download 900 images, including duplicates. I was able to download all pages and the ...
1
vote
1answer
5k views

How can I update certificates on my system to fix wget's error: “Unable to establish SSL connection.”?

Sometimes I notice that I am unable to connect via SSL with curl and wget to websites that have certificates that should be valid. One example might be https://squareup.com/ which has a Verisign ...
1
vote
1answer
717 views

wget not downloading files recursively

I saw some of the posts on this website about how to download files from a directory recursively. So, I executed the following line: wget -r -nH --cut-dirs=3 -A '*.bz2' -np http://www.xfce.org/...
8
votes
3answers
46k views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
2
votes
1answer
836 views

Using wget to download images in facebook conversations

This looks a bit harder than usual. I'm a new wget user and I was thinking that maybe it's possible to login into Facebook and download images from a specific conversation. I mean, I'd like to tell ...
1
vote
1answer
127 views

remove the bit per bit log of wget

I have a php script that generates bash commands to wget files. I am trying to log wget activity using this: --directory-prefix='xxx' --append-output='xxx.txt' -c -t0 -S" But the log is too big due ...
1
vote
2answers
437 views

Downloading nested pdf files with wget

I am trying to download dozens of PDF files located on pages linked from here: http://machineknittingetc.com/passap.html?limit=all Each PDF is referred to by a URL ending with /downloadable/download/...
0
votes
1answer
62 views

How can `wget` get more logs when it gets error

I have several questions about wget tool in linux: # wget http://www.jdfschool.com --2019-04-05 02:33:44-- (Trys: 3) http://www.jdfschool.com/ Connecting www.jdfschool.com|115.28.223.13|:80... ...
1
vote
1answer
53 views

How to update a local mirror from a public server?

I already have a mirror of the entire slackware64-current/ directory tree using wget -r -np -R "index.html*" https://mirror.slackbuilds.org/slackware/slackware64-current/ and that's fine. But it's ...
1
vote
1answer
1k views

How do I use wget or CURL in php to grab a snapshot of a web page with an asterick * in its URL? (Wayback Machine archive.org question)

The following web page on the Wayback Machine (archive.org) has an asterick * in its URL: https://web.archive.org/web/*/https://www.nasa.gov/ Neither CURL nor wget will allow me to grab the page. I ...
1
vote
2answers
3k views

How to get http server response in case of 401 using wget?

I need to test my http server responses in various cases, even for failed authentication. In case of failed authentication, my server returns 401 Unauthorized and also a response body which simple ...
1
vote
2answers
17k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
0
votes
0answers
30 views

How to receive data from a Webpage and send it back to it in the same connection?

I am trying to fill a form of a webpage with curl and give it some arguments. Dumb enough, it also sends a hidden form in which there is an unique token which has to be sent back to the page, a so ...
-1
votes
1answer
141 views

How to wget mirror website faster?

when I use wget -m url it downloads one file at a time. How to make wget download multiple files at a time?
0
votes
0answers
119 views

Wget wildcard files without temp html files

Hello I've been trying to find the correct solution to download wildcard files using wget - I used to have nexus api calls to download the latest files, but unfortunately I have to use pattern ...
0
votes
2answers
1k views

403 http code appears when I go to a certain website

I wrote a PHP script to a particular site using a home computer for debugging. Script worked properly, when I tested it on a home computer. The script has stopped working when I deploy the script on ...
0
votes
1answer
428 views

Installing Apache Tomcat on CentOS giving read error on Wget

I have a remote VM to which I connect using SSH. Now i need to install Tomcat 8 for which I run wget command wget http://mirror.fibergrid.in/apache/tomcat/tomcat-8/v8.5.14/bin/apache-tomcat-8.5.14....
2
votes
2answers
6k views

bash: wget: command not found

I'm using ASUS-Chromebook-Flip-C302CA / Google Chrome OS - Version 65.0.3325.35 (Official Build) dev (64-bit) and I'm trying to follow Visual Studio Code for Chromebooks and Raspberry Pi, yet failing ...
3
votes
2answers
1k views

Wget with different filename, but same extension

I need wget to save to a different folder, different filename, and same extension, but I do not have the filename before downloading. For example, let's say I have: http://ge.tt/api/1/files/someFile/...
0
votes
0answers
83 views

wget isn't converting some links properly

I'm trying to get a "working copy" of google.com using wget. The command I use is: wget --page-requisites --html-extension --convert-links --restrict-file-names=windows --no-host-directories -U "...
0
votes
1answer
725 views

How do I download all images from a webpage with wget

I want to download all images from a webpage but when I type: $ wget -nd -r -P images -A "*.jpg, *.jpeg, *.png" "Path/to/url" wget not only downloads images from the webpage I want but it also ...
0
votes
1answer
111 views

Troubleshooting: cURL timeouts, Java app doesn't

I'm trying to use curl against a remote server to access a REST API (and right now, just the welcome page...) Currently, I have a Java application that can send REST request on this server. But if I ...