Questions tagged [wget]
wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)
845
questions
0
votes
1
answer
33
views
Add a prefix to filenames retrieved with wget using the --content-disposition option
I have to download some files through a URL pattern: https://www.server.org/data/2021/{i} where i can vary from 1 to 1000.
Files are pdfs and they all have they own name, e.g.:
https://www.server.org/...
- 429
0
votes
1
answer
38
views
Is it possible to download in parallel with wget?
I am downloading a bunch of files from a list, using wget, with this
wget -nc --input-file=list
The problem with this method is that it downloads files in sequence and takes a long time.
I know I can ...
- 4,586
1
vote
2
answers
57
views
wget — download multiple files over multiple nodes on a cluster
Hi there I'm trying to download a large number of files at once; 279 to be precise. These are large BAM (~90GB) each. The cluster where I'm working has several nodes and fortunately I can allocate ...
- 11
0
votes
0
answers
40
views
unable to update wget version
I want to update my wget version to 1.22 which is currently 1.19 using the command :
curl -O https://ftp.gnu.org/gnu/wget/wget-1.21.tar.gz
but getting following error:
% Total % Received % Xferd ...
- 127
24
votes
1
answer
2k
views
What is the undocumented -Y option to wget?
I noticed a script using wget -Y off ...(bunch of stuff)... which I hadn't encountered before.
Looked it up in the man page and in wget --help and found nothing.
But it definitely exists as an option; ...
- 36.1k
-1
votes
1
answer
24
views
Error installing Geckodriver
I need to download this file : "https://github.com/curl -s https://github.com/mozilla/geckodriver/releases | grep -E -o '/mozilla/geckodriver/releases/download/[a-z0-9/.-]{5,}linux64.tar.gz' | ...
- 1
0
votes
1
answer
589
views
wget error: certificate owner does not match hostname
wget command fails to download a file with error "The certificate's owner does not match hostname"
Full command output is show below:
wget --continue --directory-prefix=/download https://...
- 109
-3
votes
3
answers
59
views
How wget in linux can show contents as window
I am using wget to show html content.
In window, it can show the content
In linux, it can not show:
Are there any ways to show all content in Linux
- 1
0
votes
1
answer
51
views
Is it possible to use wget or curl to download documents from the FCC ECFS site?
I'm trying to use the FCC's Electronic Comment Filing System (ECFS) to bulk download filings in individual proceedings. They have an API that will return every filing in a proceeding. It returns a URL ...
- 1
1
vote
0
answers
100
views
FTP & WGET remote file upload fail [closed]
I'm using virtualbox with kali linux installed.
While on SSH shell, i used wget to upload resources from my host machine but it's failing, FTP put command fails too.
The following are the command ...
- 19
0
votes
0
answers
38
views
Downloaded data with wget -c option gets corrupted on disconnection and retrying
Downloaded data with wget -c option gets corrupted on disconnection and retrying. Is there any way to solve my problem?
- 1
0
votes
1
answer
40
views
Using bash functionality in rc files
I am trying to do something similar to Avoiding plain-text password in http_proxy but inside the .wgetrc/.curlrc configuration files. For whatever reason, I have spotty luck by passing proxy settings ...
- 107
0
votes
0
answers
28
views
wget : several downloaded *.xz files aren't of the expected size
When I run sudo apt update on Repo DMZ after syncing all Commands-amd64.xz files from https://archive.ubuntu.com, the command completes without any errors but the same command on Repo LAN, after ...
- 1
0
votes
0
answers
359
views
403 Forbidden Error on recursive downloads using wget but not single files
I'm trying to download a directory using a recursive wget command
wget -m -nH --cut-dirs=5 https://data.darts.isas.jaxa.jp/pub/pds3/sln-l-spice-6-v1.0/slnsp_1000/
This works for some of the files, ...
- 113
0
votes
1
answer
152
views
wget - do not overwrite file if the download fails
Imagine you download a file to ~/img.txt
wget https://picsum.photos/200 -O ~/img.jpg
The image file is stored. Now imagine you try again but with a wrong URL
wget https://picsum.photooooooos/200 -O ~/...
0
votes
0
answers
58
views
How do I force wget to save directory files as index.html?
When recursive wget encounters a URL like http://example.com/directory that redirects to http://example.com/directory/ it saves the file as example.com/directory based on the pre-redirect URL. If it ...
- 2,230
2
votes
0
answers
78
views
Download tool to keep multiple ISOs up to date
I currently use this script (https://gist.github.com/Phill93/3c2b3503810a71bb98d6edb950d7aab2) to keep my ISO collection up to date. I use it to download ISOs via Rsync or HTTP/S. Now I would like to ...
- 21
4
votes
2
answers
671
views
Piping of stream from wget to tar
I want to download the Linux kernel and unarchive it in one step. There are a lot manuals about it. So I tried:
$ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.3.1.tar.xz | tar -xav -C /...
- 223
0
votes
1
answer
200
views
how do you get the total size of wget recursive without downloading all the files first?
i have something large to download recursively
the size is estimated to be > 150GB (someone told me this is roughly the size)
right now i do not have the storage space
so i need to figure out how ...
- 3,842
0
votes
2
answers
269
views
Problem using dirname in a subshell
I am writing a (one-line) script which should recurse thru subdirectories. Find .txt files containing hyperlinks. Use wget to get the contents and download it in the same directory where the text file ...
- 129
1
vote
0
answers
31
views
How can I wget a series of directories, taking only the highest of a set of numbered directories (and its subdirs/files)?
I'm looping over a series of top-level directories that all have a similar structure beneath, like so:
some_dir/some_other_dir/a_different_dir/{1,2,3,4,5,6...}
I would like to pull only the highest ...
- 43
3
votes
1
answer
190
views
Horrible wget download speed to CIFS mount; local download and cp to share is fine
I'm having a horrible wget download speed to a CIFS share.
This is the speed of wget download directly to the share:
d64-18ef-43d5-b295- 2%[ ] 100.18M 11.7MB/s eta 6m 2s
This ...
1
vote
1
answer
81
views
Run one-line programs in parallel on separate lines
So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick.
parallel -u wget -qc --...
- 139
0
votes
0
answers
123
views
Download entire website directory structure, without downloading any file contents
Need to download the entire website tree structure with directory/files names, but without download files - just structure.
like this:
www.example.com
|
dir1
| file1.txt
file2.txt
dir2
...
- 95
0
votes
0
answers
59
views
Downloading website via wget only yields index page
a friend has asked me to make some small modifications to a website for him. I'm familiar with HTML & CSS, but turned to wget as an initial way to accelerate the process.
Right now, only the index ...
0
votes
1
answer
82
views
Is it possible to include wget connection info in output file?
Is it possible to include wget connection info in the output file?
I need connection info such as "Connection timed out, " "Connection refused," "Network is unreachable," ...
- 15
0
votes
1
answer
828
views
Getting the issuer or subject hash from a server's SSL certificate
Is there a way to check the issuer of subject hash of some server's SSL certificate?
I know that curl or wget search for the matching root certificate in /var/lib/ca-certificates/openssl or /etc/ssl/...
- 4,025
0
votes
0
answers
247
views
wget does not always retrive data
I've got a problem with wget, but I do not know where real issue is:
my wget (normally POST data, seems to be correct, tried with GET also the output is the same) does not allways retrive the result (...
- 3
0
votes
0
answers
250
views
bash Implementing parallel chunk download using curl and mimic aria2 functionality [duplicate]
For large file download, aria2 supports -x 16 -s 32, options that are for multithreaded multichunk download.
Using python asyncio we can do this, by getting file size, and creating chunk parts from ...
- 1
0
votes
0
answers
732
views
How to download all files shared in a google drive folder?
None of the other solutions answers are working.
I have hundreds of images and some subfolders in a publicly shared google drive folder.
I want to download all of them.
How can I do that?
Thank you.
- 101
0
votes
1
answer
240
views
evaluate a string that is a bash if-statement
My script has a function do_command() that prints a command and then executes it. I want to put my commands in a bash array of strings, and then print _ execute them with this function. But when I try ...
- 559
0
votes
1
answer
395
views
Download the file by following too many redirections
I need to download an apk from this webpage,
https://www.apkmirror.com/apk/google-inc/youtube-music/youtube-music-5-46-52-release/youtube-music-5-46-52-2-android-apk-download/
This redirects to ...
- 182
0
votes
2
answers
244
views
get latest file on github via script
I am trying to do the same for https://github.com/eneshecan/whatsapp-for-linux/releases as is described here: Download and install latest deb package from github via terminal
but cant get it to work, ...
- 45
0
votes
1
answer
126
views
Only download first file matched by pattern using Wget
I am trying to retrieve Anaconda's install script for automated installation of it in my dotfiles. I want to retrieve the latest version of their installer script(Anaconda3-2022.10-Linux-x86_64.sh) ...
- 130
12
votes
4
answers
3k
views
How can I download only certain columns of a compressed (bgz) file using wget?
We need to download a 182 GB compressed file (uncompressed as a TSV). However, we only need the first five columns of the file, which equates to approximately 1 GB.
Is there some fancy shell magic ...
2
votes
0
answers
260
views
How do you fetch a large file over http in parallel?
Question:
Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
- 303
1
vote
1
answer
80
views
Using wget for adminer but is something wrong?
I'm new to Linux and recently I've been thinking to use adminer with my sql so I used the command
wget https://www.adminer.org/latest-mysql-en.php
And I got this output:
https://www.adminer.org/...
- 15
0
votes
0
answers
74
views
Dialog --mixedgauge help
I currently use
wget "url1" 2>&1 | stdbuf -o0 cut -b 63-65 | stdbuf -o0 grep '[0-9]'| dialog --gauge "Download Manager" 0 0
This shows the exact and perfect dialog while ...
- 182
0
votes
1
answer
132
views
How can I use wget to download files through html links?
I am trying to use wget to download files from a database through a website. The provider is now only providing web access to what was previously an ftp server that I could easily use wget to download ...
- 1
0
votes
0
answers
217
views
Using wget to get pdfs from a specific page of a website
I need to download all PDF files that are linked from this webpage https://www.physicsandmathstutor.com/maths-revision/gcse-number/.
Everything I have tried so far either does not get the files or ...
0
votes
1
answer
702
views
How to curl/wget a script, run it with arguments, and make its functions available locally?
This function will either use cURL or wGet to download a script and execute it with additional arguments:
wexec() {
url=$1
shift
if command -v "curl" >/dev/null 2>&1; ...
- 248
0
votes
1
answer
209
views
How to set my own Content-Length when downloading files with fixed size from the internet?
I want to download 100 files that are 1GB each from a website but the file size is unknown since there is no Content-Length set in headers. My download is slow and unstable. When the download crash ...
- 185
2
votes
4
answers
866
views
wget - how to download list of files from CSV with custom folder structure
I have a CSV with 2 columns, ID and a list of image urls separated by |, the number of files is variable, example:
ID_0;https://via.placeholder.com/350x150.jpg|https://via.placeholder.com/250x150.jpg
...
- 133
0
votes
2
answers
223
views
pattern matching in url and to use wget command to download multiple files [closed]
Say my links are:
https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&laDate=20220101
https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&...
- 1
2
votes
1
answer
799
views
Downloaded image from cURL is failed to open, but can open from url directly
Here is my curl command to download the file.
The downloaded file is corrupted and cannot be opened.
The file url is https://www.carscoops.com/wp-content/uploads/2022/12/Buick-Encore-GX-Scoop-4.jpg
...
- 133
1
vote
1
answer
1k
views
curl -L equivalent command in wget?
I'm trying to switch this curl command to wget because curl is posing some vulnerability to the system.
curl -fsSL https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini-static-amd64
...
- 11
0
votes
1
answer
462
views
Download a list of files from "wget" with an script gives an error
I'm trying to download a list of files with specified filenames with wget. Since it doesn't have this option with -i I've written a simple script for it:
#!/bin/bash
FILENAME="list.txt"
...
- 3
0
votes
0
answers
249
views
WGet fails to download with an IPv6 address
When I try to download something with WGet, usually it fails if the domain name is translated to an IPv6 address.
That happens more often when there's a link redirection.
For example:
$ wget --inet6-...
- 325
1
vote
0
answers
316
views
getting WGET to save clean file extensions, without query parameter
I am using wget on mac to download hundreds of thousands of images...
however the saved files all have a '?' saved at the end of the file extension (like the start of a query parameter)
How can i get ...
0
votes
0
answers
3k
views
curl(28) : ssl connection timeout
I try to tell only the details
My browsers have no problem with https sites. But when they get to tools like links, wget and curl, they run into trouble, sites like gitlab, github, google, etc. are ...
- 109