Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

Filter by
Sorted by
Tagged with
0 votes
1 answer
33 views

Add a prefix to filenames retrieved with wget using the --content-disposition option

I have to download some files through a URL pattern: https://www.server.org/data/2021/{i} where i can vary from 1 to 1000. Files are pdfs and they all have they own name, e.g.: https://www.server.org/...
s.k's user avatar
  • 429
0 votes
1 answer
38 views

Is it possible to download in parallel with wget?

I am downloading a bunch of files from a list, using wget, with this wget -nc --input-file=list The problem with this method is that it downloads files in sequence and takes a long time. I know I can ...
Duck's user avatar
  • 4,586
1 vote
2 answers
57 views

wget — download multiple files over multiple nodes on a cluster

Hi there I'm trying to download a large number of files at once; 279 to be precise. These are large BAM (~90GB) each. The cluster where I'm working has several nodes and fortunately I can allocate ...
Matteo's user avatar
  • 11
0 votes
0 answers
40 views

unable to update wget version

I want to update my wget version to 1.22 which is currently 1.19 using the command : curl -O https://ftp.gnu.org/gnu/wget/wget-1.21.tar.gz but getting following error: % Total % Received % Xferd ...
Aviator's user avatar
  • 127
24 votes
1 answer
2k views

What is the undocumented -Y option to wget?

I noticed a script using wget -Y off ...(bunch of stuff)... which I hadn't encountered before. Looked it up in the man page and in wget --help and found nothing. But it definitely exists as an option; ...
Wildcard's user avatar
  • 36.1k
-1 votes
1 answer
24 views

Error installing Geckodriver

I need to download this file : "https://github.com/curl -s https://github.com/mozilla/geckodriver/releases | grep -E -o '/mozilla/geckodriver/releases/download/[a-z0-9/.-]{5,}linux64.tar.gz' | ...
pwwoxzhw's user avatar
0 votes
1 answer
589 views

wget error: certificate owner does not match hostname

wget command fails to download a file with error "The certificate's owner does not match hostname" Full command output is show below: wget --continue --directory-prefix=/download https://...
Amol's user avatar
  • 109
-3 votes
3 answers
59 views

How wget in linux can show contents as window

I am using wget to show html content. In window, it can show the content In linux, it can not show: Are there any ways to show all content in Linux
trung's user avatar
  • 1
0 votes
1 answer
51 views

Is it possible to use wget or curl to download documents from the FCC ECFS site?

I'm trying to use the FCC's Electronic Comment Filing System (ECFS) to bulk download filings in individual proceedings. They have an API that will return every filing in a proceeding. It returns a URL ...
Sam's user avatar
  • 1
1 vote
0 answers
100 views

FTP & WGET remote file upload fail [closed]

I'm using virtualbox with kali linux installed. While on SSH shell, i used wget to upload resources from my host machine but it's failing, FTP put command fails too. The following are the command ...
Muath's user avatar
  • 19
0 votes
0 answers
38 views

Downloaded data with wget -c option gets corrupted on disconnection and retrying

Downloaded data with wget -c option gets corrupted on disconnection and retrying. Is there any way to solve my problem?
emirzaei's user avatar
0 votes
1 answer
40 views

Using bash functionality in rc files

I am trying to do something similar to Avoiding plain-text password in http_proxy but inside the .wgetrc/.curlrc configuration files. For whatever reason, I have spotty luck by passing proxy settings ...
Sterling Butters's user avatar
0 votes
0 answers
28 views

wget : several downloaded *.xz files aren't of the expected size

When I run sudo apt update on Repo DMZ after syncing all Commands-amd64.xz files from https://archive.ubuntu.com, the command completes without any errors but the same command on Repo LAN, after ...
Juntao's user avatar
  • 1
0 votes
0 answers
359 views

403 Forbidden Error on recursive downloads using wget but not single files

I'm trying to download a directory using a recursive wget command wget -m -nH --cut-dirs=5 https://data.darts.isas.jaxa.jp/pub/pds3/sln-l-spice-6-v1.0/slnsp_1000/ This works for some of the files, ...
2cents's user avatar
  • 113
0 votes
1 answer
152 views

wget - do not overwrite file if the download fails

Imagine you download a file to ~/img.txt wget https://picsum.photos/200 -O ~/img.jpg The image file is stored. Now imagine you try again but with a wrong URL wget https://picsum.photooooooos/200 -O ~/...
João Pimentel Ferreira's user avatar
0 votes
0 answers
58 views

How do I force wget to save directory files as index.html?

When recursive wget encounters a URL like http://example.com/directory that redirects to http://example.com/directory/ it saves the file as example.com/directory based on the pre-redirect URL. If it ...
user253751's user avatar
  • 2,230
2 votes
0 answers
78 views

Download tool to keep multiple ISOs up to date

I currently use this script (https://gist.github.com/Phill93/3c2b3503810a71bb98d6edb950d7aab2) to keep my ISO collection up to date. I use it to download ISOs via Rsync or HTTP/S. Now I would like to ...
Phill93's user avatar
  • 21
4 votes
2 answers
671 views

Piping of stream from wget to tar

I want to download the Linux kernel and unarchive it in one step. There are a lot manuals about it. So I tried: $ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.3.1.tar.xz | tar -xav -C /...
Андрей Тернити's user avatar
0 votes
1 answer
200 views

how do you get the total size of wget recursive without downloading all the files first?

i have something large to download recursively the size is estimated to be > 150GB (someone told me this is roughly the size) right now i do not have the storage space so i need to figure out how ...
Trevor Boyd Smith's user avatar
0 votes
2 answers
269 views

Problem using dirname in a subshell

I am writing a (one-line) script which should recurse thru subdirectories. Find .txt files containing hyperlinks. Use wget to get the contents and download it in the same directory where the text file ...
Johannes Linkels's user avatar
1 vote
0 answers
31 views

How can I wget a series of directories, taking only the highest of a set of numbered directories (and its subdirs/files)?

I'm looping over a series of top-level directories that all have a similar structure beneath, like so: some_dir/some_other_dir/a_different_dir/{1,2,3,4,5,6...} I would like to pull only the highest ...
pants_towel's user avatar
3 votes
1 answer
190 views

Horrible wget download speed to CIFS mount; local download and cp to share is fine

I'm having a horrible wget download speed to a CIFS share. This is the speed of wget download directly to the share: d64-18ef-43d5-b295- 2%[ ] 100.18M 11.7MB/s eta 6m 2s This ...
Janne Paalijarvi's user avatar
1 vote
1 answer
81 views

Run one-line programs in parallel on separate lines

So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick. parallel -u wget -qc --...
Eduardo Perez's user avatar
0 votes
0 answers
123 views

Download entire website directory structure, without downloading any file contents

Need to download the entire website tree structure with directory/files names, but without download files - just structure. like this: www.example.com | dir1 | file1.txt file2.txt dir2 ...
tuytuy20's user avatar
0 votes
0 answers
59 views

Downloading website via wget only yields index page

a friend has asked me to make some small modifications to a website for him. I'm familiar with HTML & CSS, but turned to wget as an initial way to accelerate the process. Right now, only the index ...
John Clack's user avatar
0 votes
1 answer
82 views

Is it possible to include wget connection info in output file?

Is it possible to include wget connection info in the output file? I need connection info such as "Connection timed out, " "Connection refused," "Network is unreachable," ...
Gary C. New's user avatar
0 votes
1 answer
828 views

Getting the issuer or subject hash from a server's SSL certificate

Is there a way to check the issuer of subject hash of some server's SSL certificate? I know that curl or wget search for the matching root certificate in /var/lib/ca-certificates/openssl or /etc/ssl/...
aviro's user avatar
  • 4,025
0 votes
0 answers
247 views

wget does not always retrive data

I've got a problem with wget, but I do not know where real issue is: my wget (normally POST data, seems to be correct, tried with GET also the output is the same) does not allways retrive the result (...
Erkman's user avatar
  • 3
0 votes
0 answers
250 views

bash Implementing parallel chunk download using curl and mimic aria2 functionality [duplicate]

For large file download, aria2 supports -x 16 -s 32, options that are for multithreaded multichunk download. Using python asyncio we can do this, by getting file size, and creating chunk parts from ...
Naira's user avatar
  • 1
0 votes
0 answers
732 views

How to download all files shared in a google drive folder?

None of the other solutions answers are working. I have hundreds of images and some subfolders in a publicly shared google drive folder. I want to download all of them. How can I do that? Thank you.
Furkan Gözükara's user avatar
0 votes
1 answer
240 views

evaluate a string that is a bash if-statement

My script has a function do_command() that prints a command and then executes it. I want to put my commands in a bash array of strings, and then print _ execute them with this function. But when I try ...
alle_meije's user avatar
0 votes
1 answer
395 views

Download the file by following too many redirections

I need to download an apk from this webpage, https://www.apkmirror.com/apk/google-inc/youtube-music/youtube-music-5-46-52-release/youtube-music-5-46-52-2-android-apk-download/ This redirects to ...
decipher's user avatar
  • 182
0 votes
2 answers
244 views

get latest file on github via script

I am trying to do the same for https://github.com/eneshecan/whatsapp-for-linux/releases as is described here: Download and install latest deb package from github via terminal but cant get it to work, ...
Questi's user avatar
  • 45
0 votes
1 answer
126 views

Only download first file matched by pattern using Wget

I am trying to retrieve Anaconda's install script for automated installation of it in my dotfiles. I want to retrieve the latest version of their installer script(Anaconda3-2022.10-Linux-x86_64.sh) ...
unrealapex's user avatar
12 votes
4 answers
3k views

How can I download only certain columns of a compressed (bgz) file using wget?

We need to download a 182 GB compressed file (uncompressed as a TSV). However, we only need the first five columns of the file, which equates to approximately 1 GB. Is there some fancy shell magic ...
Matthaeus Gaius Caesar's user avatar
2 votes
0 answers
260 views

How do you fetch a large file over http in parallel?

Question: Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
KJ7LNW's user avatar
  • 303
1 vote
1 answer
80 views

Using wget for adminer but is something wrong?

I'm new to Linux and recently I've been thinking to use adminer with my sql so I used the command wget https://www.adminer.org/latest-mysql-en.php And I got this output: https://www.adminer.org/...
linuxverse's user avatar
0 votes
0 answers
74 views

Dialog --mixedgauge help

I currently use wget "url1" 2>&1 | stdbuf -o0 cut -b 63-65 | stdbuf -o0 grep '[0-9]'| dialog --gauge "Download Manager" 0 0 This shows the exact and perfect dialog while ...
decipher's user avatar
  • 182
0 votes
1 answer
132 views

How can I use wget to download files through html links?

I am trying to use wget to download files from a database through a website. The provider is now only providing web access to what was previously an ftp server that I could easily use wget to download ...
BashN3wb's user avatar
0 votes
0 answers
217 views

Using wget to get pdfs from a specific page of a website

I need to download all PDF files that are linked from this webpage https://www.physicsandmathstutor.com/maths-revision/gcse-number/. Everything I have tried so far either does not get the files or ...
robotsheepboy's user avatar
0 votes
1 answer
702 views

How to curl/wget a script, run it with arguments, and make its functions available locally?

This function will either use cURL or wGet to download a script and execute it with additional arguments: wexec() { url=$1 shift if command -v "curl" >/dev/null 2>&1; ...
Martin Braun's user avatar
0 votes
1 answer
209 views

How to set my own Content-Length when downloading files with fixed size from the internet?

I want to download 100 files that are 1GB each from a website but the file size is unknown since there is no Content-Length set in headers. My download is slow and unstable. When the download crash ...
pwrsheller's user avatar
2 votes
4 answers
866 views

wget - how to download list of files from CSV with custom folder structure

I have a CSV with 2 columns, ID and a list of image urls separated by |, the number of files is variable, example: ID_0;https://via.placeholder.com/350x150.jpg|https://via.placeholder.com/250x150.jpg ...
LS_'s user avatar
  • 133
0 votes
2 answers
223 views

pattern matching in url and to use wget command to download multiple files [closed]

Say my links are: https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&laDate=20220101 https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&...
Pranjal's user avatar
2 votes
1 answer
799 views

Downloaded image from cURL is failed to open, but can open from url directly

Here is my curl command to download the file. The downloaded file is corrupted and cannot be opened. The file url is https://www.carscoops.com/wp-content/uploads/2022/12/Buick-Encore-GX-Scoop-4.jpg ...
TomSawyer's user avatar
  • 133
1 vote
1 answer
1k views

curl -L equivalent command in wget?

I'm trying to switch this curl command to wget because curl is posing some vulnerability to the system. curl -fsSL https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini-static-amd64 ...
ROdna's user avatar
  • 11
0 votes
1 answer
462 views

Download a list of files from "wget" with an script gives an error

I'm trying to download a list of files with specified filenames with wget. Since it doesn't have this option with -i I've written a simple script for it: #!/bin/bash FILENAME="list.txt" ...
mokazemi's user avatar
0 votes
0 answers
249 views

WGet fails to download with an IPv6 address

When I try to download something with WGet, usually it fails if the domain name is translated to an IPv6 address. That happens more often when there's a link redirection. For example: $ wget --inet6-...
markfree's user avatar
  • 325
1 vote
0 answers
316 views

getting WGET to save clean file extensions, without query parameter

I am using wget on mac to download hundreds of thousands of images... however the saved files all have a '?' saved at the end of the file extension (like the start of a query parameter) How can i get ...
Matthew Swart's user avatar
0 votes
0 answers
3k views

curl(28) : ssl connection timeout

I try to tell only the details My browsers have no problem with https sites. But when they get to tools like links, wget and curl, they run into trouble, sites like gitlab, github, google, etc. are ...
ahmadreza1383's user avatar

1
2 3 4 5
17