Questions tagged [wget]
wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)
645
questions
0
votes
1answer
33 views
How to log successfully finished downloads with wget
I am downloading a huge list of files in parallel using xargs. I have the problem that some downloads fail after some time, so once I have run through my list, not all downloads have completed ...
-1
votes
2answers
68 views
Downloading and writing a text file onto tmpfs is slower than writing to disk, why?
I have a tmpfs named /rtmp/ that has 1GB of RAM allocated to it on Ubuntu OS.
Using a bash script, I am testing to see if it is faster using to write a small text file to the hard disk or if it is ...
0
votes
0answers
18 views
Wget: save links to files named by the order in which the links appear
I am using wget to assemble a book. The book is published as a webpage with links to individual PDFs per chapter. The links are in the correct order, but the PDF filenames are not alphabetical. For ...
0
votes
0answers
16 views
wget not following all recursions?
I am trying to recursively download a website using
wget -m -erobots=off --adjust-extension https://www.example
(Initially I just did -r, but in order to counter suspicions about search depth or ...
-2
votes
1answer
33 views
Download files (game mods) from a website
My goal:
To run a script or command every day in order to get the newest mods which were released in the Transport Fever 2 mod section.
https://www.transportfever.net/filebase/index.php?filebase/80-...
0
votes
1answer
52 views
wget / curl is not following the redirect of my URL
If you are on the website there is automatically started a download of pdf: https://magazines-static.raspberrypi.org/issues/full_pdfs/000/000/308/original/MagPi89.pdf?1576682030
I was writing to ...
0
votes
1answer
52 views
How does wget resume downloads?
I'm going to assume that wget starts off with a file filled with zeros and begins filling it with data.
If the download is interrupted, how does wget know where it left off? Looking at the zeros ...
0
votes
1answer
121 views
wget ERROR 400: Bad Request
I have a list of URLs to be downloaded through wget. For a single file when I enter following command in shell it downloads and saves file correctly.
wget -O IPSL-CM5A-LR-tasmin-Historical-19500101-...
0
votes
2answers
54 views
wget doesn't get all of the page content
I am trying to extract the artists of this page. I have tried many variations of
wget https://northside.dk/artister/
and
wget --random-wait -r -p -e robots=off -U mozilla https://northside.dk/...
2
votes
0answers
50 views
How Can I Use shc to Convert Scripts that contain `wget` Into Binaries? [closed]
I have a bash script that I wish to convert into a binary using shc. It includes the line wget www.google.com.
If I run shc -f myscript.sh to convert it into ./myscript.sh.x (the resulting program), ...
0
votes
0answers
41 views
Extreme command shortening
I'm converting some commands into bar codes for others to easily scan to implement. It cuts down on repetitive typing and typos. The biggest issue are some of the commands that are long, as the longer ...
0
votes
2answers
36 views
No wget domain resolution in crontab
I need to download a file when my computer boots up, so I created this line in my cronjob:
@reboot wget -a /log/here -O /save/here https://domain.tld/file.ext
Unfortunately, it creates a blank file ...
2
votes
3answers
122 views
Why do wget/curl not download all source code of a web page?
I try to download the 1198 pages of signatures on the petition page at http://www.musikskatt.se/ (at the bottom) but both curl and wget leaves the div with the signatures empty in the resulting file.
...
0
votes
0answers
87 views
wget not working on Arch and having depdendencies to both libnettle.so.6 and libnettle.so.7
I think upgrading the system with pacman -Syu after a year of inactivity is the issue.
Now pacman can't download anything as wget won't run anymore with the following error:
wget: error while ...
0
votes
1answer
114 views
how to resume download in wget for google takeout data
I have 13 GB gmail data based on a discussion here
https://superuser.com/questions/1502126/13-66-gb-out-of-15-gb-used-in-gmail-delete-emails-that-consumed-space
I decided to download all emails in ...
2
votes
3answers
56 views
running same command for multiple links within a terminal
I have a file which has many links. As an example here is my file:
link1
link2
link3
link4
link5
link6
link7
link8
link9
link10
link11
link12
link13
link14
link15
link16
I want to run a particular ...
0
votes
1answer
54 views
How can I download the entire contents of a directory using wget, but excluding files with a particular suffix with the use of a wildcard?
I have a locally-hosted server, and am attempting to download all files to my remote Ubuntu-based machine via wget. I need to download all files from my HTTP server in a single directory, ensuring ...
0
votes
2answers
22 views
WordPress not found when using wget
When I run wget https://wordpress.org/latest.tar.gz in my Ubuntu 16.04 server, I get this result:
I have tried other sources but they do not have the specific error that I do.
It seems to not find ...
-5
votes
1answer
77 views
Why does this wget command fail to download this file? [duplicate]
I was wondering why the following command fails to download a pdf file, while pasting the URL in browser's address bar can? (I am trying to learn a little about HTTP, web services, or web applications ...
0
votes
1answer
163 views
How to bind wget to download through a specific port?
I'm trying to download a file through wget. But I want to download it through a port which I configured as a proxy.
How can I configure wget to download through a specific port in my computer?
In ...
1
vote
0answers
31 views
How to tell wget to mirror website where links don't have html extension?
I have website (book store) https://helion.pl that have links that look like this:
/kategoria/programowanie
There is probably one php application, and not actual files on the disk.
How can I mirror ...
0
votes
1answer
24 views
How to save cookies when the app write to document.cookie?
I'd like to understand how to save cookies that the application writes to document.cookie (https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie), as I've noticed that the common methods ...
-1
votes
1answer
106 views
Understanding the command “wget -q -O- | apt-key add -”
I am trying to understand this command:
#wget -q https://packages.sury.org/php/apt.gpg -O- | apt-key add -
wget -q
Download the open keyrings but should not output anything
-O-
Here https://unix....
1
vote
0answers
255 views
How to download a folder from Onedrive business using wget?
I've tried the "wget --no-check-certificate 'https://my.sharepoint.com/cid=6EBB03E38A53ED3E?donwload=1" -O name' " to download a folder from OneDrive business. But it turns out that this approach only ...
0
votes
0answers
92 views
Gnutls error when running wget
When I try and do the following:
[john@seneca gnutls-3.6.9]$ wget https://www.atlassian.com/software/stash/downloads/binary/atlassian-bitbucket-6.6.0-x64.bin
--2019-09-24 11:13:08-- https://www....
1
vote
1answer
1k views
wget forbidden but can download with firefox [closed]
I'm used to download addons from https://www.curseforge.com with my own script, but now, it's not working anymore.
Each try I do, I get :
$ LANG=C wget https://www.curseforge.com/wow/addons/...
0
votes
1answer
55 views
Using wget to download only the first depth of external links
I would like to get a copy of this page: http://databyte.ch/services/tony/index.html obviously this page is already a copy of the original source: http://www.tonyvanroon.com/oldwebsite/circ/circuits....
0
votes
1answer
261 views
wget not retrieving correct-size file (corrupt or incomplete file?)
I don't understand...
The actual download link appears to not be a http link but rather some Javascript action?
javascript:SendFileDownloadCall('PRODIMAGES.CIF.zip','PRODIMAGES.CIF.zip');
So after ...
-2
votes
2answers
2k views
Linux wget https://kali.sh/xfce4.sh problem [closed]
I am trying to install https://kali.sh/xfce4.sh in my kali linux to use GUI. However, if i type "sudo wget https://kali.sh/xfce4.sh", it says this:
--2019-09-08 16:43:20-- https://kali.sh/xfce4.sh
...
0
votes
0answers
323 views
Error “Cannot write to” using wget
I want to make a backup script that downloads the content of a SharePoint folder. I got a working command using wget (when previously login and export the cookie.txt) using the following command:
...
-7
votes
1answer
53 views
Why am I getting these error messages? [closed]
Debian 10
sudo wget -O protonvpn-cli.sh gives wget: missing url.
sudo protonvpn-cli -connect gives command not found
I have not changed anything in any of the computer files. I do use a firewall ...
1
vote
1answer
167 views
rsync-like --delete functionality with wget (not wget's --delete-after)
Background: my particular problem, which gave rise to this question, is as follows. I'm a slackware linux user, and on 23-March-2019 mirrored their distribution with the following command
wget -r -np ...
0
votes
1answer
673 views
Parsing output from wget and grep using bash
I can get a url link and search for text that starts with file: but I'm having issues parsing it from there.
Example:
wget -qO- http://website.com/site/ | tr \" \\n | grep -w file:\* > output....
0
votes
1answer
103 views
download entire website with wget
I want to download the entire wikipedia to have an offline copy, I try using that command:
wget --recursive --domains --convert-links --no-clobber https://it.wikipedia.org/wiki/
But it download only ...
-2
votes
3answers
357 views
Piping of stream from wget to tar, and extracting it to a specific location failed
I try to pipe a stream from wget to tar and extracting it to a specific location.
The file is downloaded by wget but not extracted as desired with tar:
war="/var/www/html"
domain="example.com"
...
0
votes
0answers
140 views
Mediafire Files with using WGET - Centos7
I just looked like 100 websites and didnt find anything. I cant download anything from MediaFire with using wget command on centos7. Actual file size is 700mb but wget is downloading like 20kb.
i ...
0
votes
0answers
32 views
Wget check if local file matches remote file but do not download if the local file does not match or does not exist
I need wget -nc -c $remote_file for filelist, a fast way to check if possible local file matches the remote file but in case the local file doesn't match or doesn't even exist, download should not ...
0
votes
0answers
35 views
wget average data rate is not consistent with download file size divided by time
For a few experiments I ran wget to download a file to measure data rates. The file (size) is always the same, about 292 MiB.
The average download speed shown by wget does not match the file size ...
0
votes
1answer
44 views
File transfer gone wrong
Newbie here
I am trying to transfer backup files of several servers to a central server, using wget, using a script and a cronjob. But when the last file is being transferred, something goes wrong ...
0
votes
0answers
37 views
How to look for directory in http address
I'm trying to create a regex formula in shell script which will allow me to find desired folder in HTTP directory.
Let's assume that this is the address: ftp://google.com/directories/
In this ...
0
votes
1answer
79 views
How to get triggered download from site with delay
Normally I use wget to get files that trigger a download.
However, I'm currently using a site that takes a few seconds to generate the download, so wget downloads the html instead of the zip file I ...
1
vote
1answer
341 views
wget content-disposition ubuntu vs alpine
In Ubuntu, I can use the following wget flag to get a debian, i.e.,
wget --content-disposition https://packagecloud.io/xxxxx/download.deb
dpkg -i ...
In Alpine, wget complains that --content-...
-1
votes
2answers
281 views
How to use wget to download directory recursively, into directory with the same name as on the server?
Let's say I have a URL like so:
https://mywebsite.com/files/myfolder
How do I download all the files in the myfolder part (which is a directory), excluding files like index files, into a directory ...
0
votes
2answers
73 views
Get web page complete data
I'm using busybox tools and I want to take all http links in a web page. I save an example link page by using curl or wget. However, it saves the page as html. How to do it with curl or wget commands?
...
-1
votes
1answer
143 views
how to download ebi fastq link via command line using wget/curl
I have loads of data (fastq) to download from ENA database.
Example: ERS076383 ERS012262 ERS160574 ERS349280 etc
Link to download via browser:
https://www.ebi.ac.uk/ena/data/warehouse/filereport?...
1
vote
0answers
53 views
Recursive wget, append something to each found URL
My main goal is to easily backup the page source for a bunch of MediaWiki pages of mine. I have every page I want downloaded, listed on my username page (https://wiki.example.com/index.php?title=User:...
0
votes
1answer
344 views
Unable to wget for ipv6
I tried wget using the following command
wget -6 https://[fe80::]
But it displayed the following error
Connecting to [fe80::]:80... failed: Invalid argument.
0
votes
0answers
54 views
How can I make wget rename files based on link name?
Everything I'm finding is based on URL, but I want the opposite.
I have an html index with a list of links.
The links are named like Apples, oranges, grapes, bananas. The files associated with those ...
1
vote
1answer
124 views
wget -i notifying when each file finishes (for processing purposes)
I'd like to be able to process multiple files downloaded by wget -i immediately after they are downloaded (instead of waiting for all files in the list to finish--for the entire wget process to exit). ...
3
votes
2answers
1k views
wget download files from a file list, how to specify names for each of the downloaded file?
When downloading a single file, we can use wget's -O option to specify the file name. When I'm downloading Urls in a file using wget -i filelist.txt (filelist.txt contains list Of Urls I want to ...
