Skip to main content

Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

Filter by
Sorted by
Tagged with
0 votes
0 answers
41 views

I want to know the redirected URL from current URL

URL = "http://downloads.sourceforge.net/sourceforge/gptfdisk" I can access this URL in my browser, but it is getting redirected to a newer URL. Problem: When I give this command to get ...
Viraj Patel's user avatar
0 votes
1 answer
89 views

How can I download the latest .deb file from a Github release page using curl and wget?

I'm trying to find a way to download the latest released .deb file from a Github releases page, for example: https://github.com/grafviktor/goto/releases. the solution would first try to curl the page ...
Ilgar's user avatar
  • 3
2 votes
1 answer
30 views

How to check consistency of a generated web site using recursive HTML parsing

I have a FOSS project whose web site is generated by asciidoc and some custom scripts as an horde (thousands) of static files locally in the source files' repo, copied into another workspace and ...
Jim Klimov's user avatar
0 votes
1 answer
24 views

Downloading files from url that return save dialog box

I have trying download files form a url that return save dialog box. I am using wget. It is not working! I am using the follow command wget -ci LinkDownloadHw.csv Bellow are a sample of links used in ...
THIAGO CARLOS LOPES RIBEIRO's user avatar
1 vote
1 answer
49 views

Does an echo to network command exist in debian?

Does some sort of command like: echotonet "this" or cat file.txt > echonet that could allow one to easily paste text document data to a service like paste.debian.net ? I often use ...
user3450548's user avatar
  • 2,994
0 votes
0 answers
19 views

wget -I='/my/subpath,/my/subpath/*' (/my/subpath/) is excluded/not-included. Decided NOT to load it

I am trying to index some pages, corresponding to a Documentation. For that I wanted to wget --no-parent -r -l 5 https://developer.mozilla.org/en-US/docs/Web/JavaScript/ But as the request will be ...
freddie_ventura's user avatar
4 votes
1 answer
298 views

Download files in parallel with wget over a single connection

This question is a follow-up to How to download multiple files simultaniously with wget? Similar to that question, I need to download many files. With the accepted answer above recommends the use of ...
sneumann's user avatar
0 votes
1 answer
58 views

Wget and chrome downloads

While I can download a file putting an HTTP link in Chrome this format http://someplace.some/get.php?username=myusername&password=mypass&type=m3u_plus&output=ts if I try the same with wget ...
Jon Rotaetxe's user avatar
0 votes
2 answers
154 views

How can I download a very large list of URLs so that the downloaded files are split into subfolders containing the first letter of the filenames?

I want to download many files (> tens of millions). I have the URL for each file. I have the list of URLs in a file URLs.txt: http://mydomain.com/0wd.pdf http://mydomain.com/asz.pdf http://mydomain....
Franck Dernoncourt's user avatar
10 votes
1 answer
2k views

Why does wget work and curl fail with a specific REST URL?

In answering a question on another site, I came across a strange discrepancy between curl and wget when trying to fetch from this URL: https://www.uniprot.org/uniprot/A2Z669.fasta For some reason, ...
terdon's user avatar
  • 245k
0 votes
1 answer
121 views

Add a prefix to filenames retrieved with wget using the --content-disposition option

I have to download some files through a URL pattern: https://www.server.org/data/2021/{i} where i can vary from 1 to 1000. Files are pdfs and they all have they own name, e.g.: https://www.server.org/...
s.k's user avatar
  • 471
0 votes
1 answer
98 views

Is it possible to download in parallel with wget?

I am downloading a bunch of files from a list, using wget, with this wget -nc --input-file=list The problem with this method is that it downloads files in sequence and takes a long time. I know I can ...
Duck's user avatar
  • 4,714
1 vote
3 answers
458 views

wget — download multiple files over multiple nodes on a cluster

Hi there I'm trying to download a large number of files at once; 279 to be precise. These are large BAM (~90GB) each. The cluster where I'm working has several nodes and fortunately I can allocate ...
Matteo's user avatar
  • 113
0 votes
0 answers
59 views

unable to update wget version

I want to update my wget version to 1.22 which is currently 1.19 using the command : curl -O https://ftp.gnu.org/gnu/wget/wget-1.21.tar.gz but getting following error: % Total % Received % Xferd ...
Aviator's user avatar
  • 127
24 votes
1 answer
3k views

What is the undocumented -Y option to wget?

I noticed a script using wget -Y off ...(bunch of stuff)... which I hadn't encountered before. Looked it up in the man page and in wget --help and found nothing. But it definitely exists as an option; ...
Wildcard's user avatar
  • 36.8k
-1 votes
1 answer
44 views

Error installing Geckodriver

I need to download this file : "https://github.com/curl -s https://github.com/mozilla/geckodriver/releases | grep -E -o '/mozilla/geckodriver/releases/download/[a-z0-9/.-]{5,}linux64.tar.gz' | ...
pwwoxzhw's user avatar
0 votes
1 answer
3k views

wget error: certificate owner does not match hostname

wget command fails to download a file with error "The certificate's owner does not match hostname" Full command output is show below: wget https://server.example.com/dist/software-1.0.rpm ...
Amol's user avatar
  • 109
-3 votes
3 answers
113 views

How wget in linux can show contents as window

I am using wget to show html content. In window, it can show the content In linux, it can not show: Are there any ways to show all content in Linux
trung's user avatar
  • 1
0 votes
1 answer
98 views

Is it possible to use wget or curl to download documents from the FCC ECFS site?

I'm trying to use the FCC's Electronic Comment Filing System (ECFS) to bulk download filings in individual proceedings. They have an API that will return every filing in a proceeding. It returns a URL ...
Sam's user avatar
  • 1
1 vote
0 answers
214 views

FTP & WGET remote file upload fail [closed]

I'm using virtualbox with kali linux installed. While on SSH shell, i used wget to upload resources from my host machine but it's failing, FTP put command fails too. The following are the command ...
Muath's user avatar
  • 19
0 votes
0 answers
74 views

Downloaded data with wget -c option gets corrupted on disconnection and retrying

Downloaded data with wget -c option gets corrupted on disconnection and retrying. Is there any way to solve my problem?
emirzaei's user avatar
0 votes
1 answer
72 views

Using bash functionality in rc files

I am trying to do something similar to Avoiding plain-text password in http_proxy but inside the .wgetrc/.curlrc configuration files. For whatever reason, I have spotty luck by passing proxy settings ...
Sterling Butters's user avatar
0 votes
0 answers
939 views

403 Forbidden Error on recursive downloads using wget but not single files

I'm trying to download a directory using a recursive wget command wget -m -nH --cut-dirs=5 https://data.darts.isas.jaxa.jp/pub/pds3/sln-l-spice-6-v1.0/slnsp_1000/ This works for some of the files, ...
2cents's user avatar
  • 123
0 votes
1 answer
270 views

wget - do not overwrite file if the download fails

Imagine you download a file to ~/img.txt wget https://picsum.photos/200 -O ~/img.jpg The image file is stored. Now imagine you try again but with a wrong URL wget https://picsum.photooooooos/200 -O ~/...
João Pimentel Ferreira's user avatar
2 votes
0 answers
221 views

Download tool to keep multiple ISOs up to date

I currently use this script (https://gist.github.com/Phill93/3c2b3503810a71bb98d6edb950d7aab2) to keep my ISO collection up to date. I use it to download ISOs via Rsync or HTTP/S. Now I would like to ...
Phill93's user avatar
  • 21
4 votes
2 answers
785 views

Piping of stream from wget to tar

I want to download the Linux kernel and unarchive it in one step. There are a lot manuals about it. So I tried: $ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.3.1.tar.xz | tar -xav -C /...
Андрей Тернити's user avatar
0 votes
1 answer
477 views

how do you get the total size of wget recursive without downloading all the files first?

i have something large to download recursively the size is estimated to be > 150GB (someone told me this is roughly the size) right now i do not have the storage space so i need to figure out how ...
Trevor Boyd Smith's user avatar
0 votes
2 answers
571 views

Problem using dirname in a subshell

I am writing a (one-line) script which should recurse thru subdirectories. Find .txt files containing hyperlinks. Use wget to get the contents and download it in the same directory where the text file ...
Johannes Linkels's user avatar
1 vote
0 answers
66 views

How can I wget a series of directories, taking only the highest of a set of numbered directories (and its subdirs/files)?

I'm looping over a series of top-level directories that all have a similar structure beneath, like so: some_dir/some_other_dir/a_different_dir/{1,2,3,4,5,6...} I would like to pull only the highest ...
pants_towel's user avatar
3 votes
1 answer
280 views

Horrible wget download speed to CIFS mount; local download and cp to share is fine

I'm having a horrible wget download speed to a CIFS share. This is the speed of wget download directly to the share: d64-18ef-43d5-b295- 2%[ ] 100.18M 11.7MB/s eta 6m 2s This ...
Janne Paalijarvi's user avatar
1 vote
1 answer
202 views

Run one-line programs in parallel on separate lines

So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick. parallel -u wget -qc --...
Eduardo Perez's user avatar
0 votes
0 answers
190 views

Download entire website directory structure, without downloading any file contents

Need to download the entire website tree structure with directory/files names, but without download files - just structure. like this: www.example.com | dir1 | file1.txt file2.txt dir2 ...
tuytuy20's user avatar
0 votes
0 answers
103 views

Downloading website via wget only yields index page

a friend has asked me to make some small modifications to a website for him. I'm familiar with HTML & CSS, but turned to wget as an initial way to accelerate the process. Right now, only the index ...
John Clack's user avatar
0 votes
1 answer
208 views

Is it possible to include wget connection info in output file?

Is it possible to include wget connection info in the output file? I need connection info such as "Connection timed out, " "Connection refused," "Network is unreachable," ...
Gary C. New's user avatar
0 votes
1 answer
1k views

Getting the issuer or subject hash from a server's SSL certificate

Is there a way to check the issuer of subject hash of some server's SSL certificate? I know that curl or wget search for the matching root certificate in /var/lib/ca-certificates/openssl or /etc/ssl/...
aviro's user avatar
  • 5,912
0 votes
0 answers
594 views

wget does not always retrive data

I've got a problem with wget, but I do not know where real issue is: my wget (normally POST data, seems to be correct, tried with GET also the output is the same) does not allways retrive the result (...
Erkman's user avatar
  • 3
0 votes
0 answers
401 views

bash Implementing parallel chunk download using curl and mimic aria2 functionality [duplicate]

For large file download, aria2 supports -x 16 -s 32, options that are for multithreaded multichunk download. Using python asyncio we can do this, by getting file size, and creating chunk parts from ...
Naira's user avatar
  • 1
0 votes
0 answers
1k views

How to download all files shared in a google drive folder?

None of the other solutions answers are working. I have hundreds of images and some subfolders in a publicly shared google drive folder. I want to download all of them. How can I do that? Thank you.
Furkan Gözükara's user avatar
0 votes
1 answer
380 views

evaluate a string that is a bash if-statement

My script has a function do_command() that prints a command and then executes it. I want to put my commands in a bash array of strings, and then print _ execute them with this function. But when I try ...
alle_meije's user avatar
0 votes
1 answer
764 views

Download the file by following too many redirections

I need to download an apk from this webpage, https://www.apkmirror.com/apk/google-inc/youtube-music/youtube-music-5-46-52-release/youtube-music-5-46-52-2-android-apk-download/ This redirects to ...
decipher's user avatar
  • 312
0 votes
2 answers
443 views

get latest file on github via script

I am trying to do the same for https://github.com/eneshecan/whatsapp-for-linux/releases as is described here: Download and install latest deb package from github via terminal but cant get it to work, ...
Questi's user avatar
  • 45
0 votes
1 answer
184 views

Only download first file matched by pattern using Wget

I am trying to retrieve Anaconda's install script for automated installation of it in my dotfiles. I want to retrieve the latest version of their installer script(Anaconda3-2022.10-Linux-x86_64.sh) ...
unrealapex's user avatar
12 votes
4 answers
3k views

How can I download only certain columns of a compressed (bgz) file using wget?

We need to download a 182 GB compressed file (uncompressed as a TSV). However, we only need the first five columns of the file, which equates to approximately 1 GB. Is there some fancy shell magic ...
Matthaeus Gaius Caesar's user avatar
2 votes
0 answers
479 views

How do you fetch a large file over http in parallel?

Question: Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
KJ7LNW's user avatar
  • 495
1 vote
1 answer
91 views

Using wget for adminer but is something wrong?

I'm new to Linux and recently I've been thinking to use adminer with my sql so I used the command wget https://www.adminer.org/latest-mysql-en.php And I got this output: https://www.adminer.org/...
linuxverse's user avatar
0 votes
1 answer
249 views

How can I use wget to download files through html links?

I am trying to use wget to download files from a database through a website. The provider is now only providing web access to what was previously an ftp server that I could easily use wget to download ...
BashN3wb's user avatar
0 votes
1 answer
923 views

How to curl/wget a script, run it with arguments, and make its functions available locally?

This function will either use cURL or wGet to download a script and execute it with additional arguments: wexec() { url=$1 shift if command -v "curl" >/dev/null 2>&1; ...
Martin Braun's user avatar
0 votes
1 answer
363 views

How to set my own Content-Length when downloading files with fixed size from the internet?

I want to download 100 files that are 1GB each from a website but the file size is unknown since there is no Content-Length set in headers. My download is slow and unstable. When the download crash ...
pwrsheller's user avatar
2 votes
4 answers
1k views

wget - how to download list of files from CSV with custom folder structure

I have a CSV with 2 columns, ID and a list of image urls separated by |, the number of files is variable, example: ID_0;https://via.placeholder.com/350x150.jpg|https://via.placeholder.com/250x150.jpg ...
LS_'s user avatar
  • 133
0 votes
2 answers
299 views

pattern matching in url and to use wget command to download multiple files [closed]

Say my links are: https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&laDate=20220101 https://www.horoscope.com/us/horoscopes/general/horoscope-archive.aspx?sign=1&...
Pranjal's user avatar

1
2 3 4 5
18