Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

Filter by
Sorted by
Tagged with
0
votes
1answer
33 views

How to log successfully finished downloads with wget

I am downloading a huge list of files in parallel using xargs. I have the problem that some downloads fail after some time, so once I have run through my list, not all downloads have completed ...
-1
votes
2answers
68 views

Downloading and writing a text file onto tmpfs is slower than writing to disk, why?

I have a tmpfs named /rtmp/ that has 1GB of RAM allocated to it on Ubuntu OS. Using a bash script, I am testing to see if it is faster using to write a small text file to the hard disk or if it is ...
0
votes
0answers
18 views

Wget: save links to files named by the order in which the links appear

I am using wget to assemble a book. The book is published as a webpage with links to individual PDFs per chapter. The links are in the correct order, but the PDF filenames are not alphabetical. For ...
0
votes
0answers
16 views

wget not following all recursions?

I am trying to recursively download a website using wget -m -erobots=off --adjust-extension https://www.example (Initially I just did -r, but in order to counter suspicions about search depth or ...
-2
votes
1answer
33 views

Download files (game mods) from a website

My goal: To run a script or command every day in order to get the newest mods which were released in the Transport Fever 2 mod section. https://www.transportfever.net/filebase/index.php?filebase/80-...
0
votes
1answer
52 views

wget / curl is not following the redirect of my URL

If you are on the website there is automatically started a download of pdf: https://magazines-static.raspberrypi.org/issues/full_pdfs/000/000/308/original/MagPi89.pdf?1576682030 I was writing to ...
0
votes
1answer
52 views

How does wget resume downloads?

I'm going to assume that wget starts off with a file filled with zeros and begins filling it with data. If the download is interrupted, how does wget know where it left off? Looking at the zeros ...
0
votes
1answer
121 views

wget ERROR 400: Bad Request

I have a list of URLs to be downloaded through wget. For a single file when I enter following command in shell it downloads and saves file correctly. wget -O IPSL-CM5A-LR-tasmin-Historical-19500101-...
0
votes
2answers
54 views

wget doesn't get all of the page content

I am trying to extract the artists of this page. I have tried many variations of wget https://northside.dk/artister/ and wget --random-wait -r -p -e robots=off -U mozilla https://northside.dk/...
2
votes
0answers
50 views

How Can I Use shc to Convert Scripts that contain `wget` Into Binaries? [closed]

I have a bash script that I wish to convert into a binary using shc. It includes the line wget www.google.com. If I run shc -f myscript.sh to convert it into ./myscript.sh.x (the resulting program), ...
0
votes
0answers
41 views

Extreme command shortening

I'm converting some commands into bar codes for others to easily scan to implement. It cuts down on repetitive typing and typos. The biggest issue are some of the commands that are long, as the longer ...
0
votes
2answers
36 views

No wget domain resolution in crontab

I need to download a file when my computer boots up, so I created this line in my cronjob: @reboot wget -a /log/here -O /save/here https://domain.tld/file.ext Unfortunately, it creates a blank file ...
2
votes
3answers
122 views

Why do wget/curl not download all source code of a web page?

I try to download the 1198 pages of signatures on the petition page at http://www.musikskatt.se/ (at the bottom) but both curl and wget leaves the div with the signatures empty in the resulting file. ...
0
votes
0answers
87 views

wget not working on Arch and having depdendencies to both libnettle.so.6 and libnettle.so.7

I think upgrading the system with pacman -Syu after a year of inactivity is the issue. Now pacman can't download anything as wget won't run anymore with the following error: wget: error while ...
0
votes
1answer
114 views

how to resume download in wget for google takeout data

I have 13 GB gmail data based on a discussion here https://superuser.com/questions/1502126/13-66-gb-out-of-15-gb-used-in-gmail-delete-emails-that-consumed-space I decided to download all emails in ...
2
votes
3answers
56 views

running same command for multiple links within a terminal

I have a file which has many links. As an example here is my file: link1 link2 link3 link4 link5 link6 link7 link8 link9 link10 link11 link12 link13 link14 link15 link16 I want to run a particular ...
0
votes
1answer
54 views

How can I download the entire contents of a directory using wget, but excluding files with a particular suffix with the use of a wildcard?

I have a locally-hosted server, and am attempting to download all files to my remote Ubuntu-based machine via wget. I need to download all files from my HTTP server in a single directory, ensuring ...
0
votes
2answers
22 views

WordPress not found when using wget

When I run wget https://wordpress.org/latest.tar.gz in my Ubuntu 16.04 server, I get this result: I have tried other sources but they do not have the specific error that I do. It seems to not find ...
-5
votes
1answer
77 views

Why does this wget command fail to download this file? [duplicate]

I was wondering why the following command fails to download a pdf file, while pasting the URL in browser's address bar can? (I am trying to learn a little about HTTP, web services, or web applications ...
0
votes
1answer
163 views

How to bind wget to download through a specific port?

I'm trying to download a file through wget. But I want to download it through a port which I configured as a proxy. How can I configure wget to download through a specific port in my computer? In ...
1
vote
0answers
31 views

How to tell wget to mirror website where links don't have html extension?

I have website (book store) https://helion.pl that have links that look like this: /kategoria/programowanie There is probably one php application, and not actual files on the disk. How can I mirror ...
0
votes
1answer
24 views

How to save cookies when the app write to document.cookie?

I'd like to understand how to save cookies that the application writes to document.cookie (https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie), as I've noticed that the common methods ...
-1
votes
1answer
106 views

Understanding the command “wget -q -O- | apt-key add -”

I am trying to understand this command: #wget -q https://packages.sury.org/php/apt.gpg -O- | apt-key add - wget -q Download the open keyrings but should not output anything -O- Here https://unix....
1
vote
0answers
255 views

How to download a folder from Onedrive business using wget?

I've tried the "wget --no-check-certificate 'https://my.sharepoint.com/cid=6EBB03E38A53ED3E?donwload=1" -O name' " to download a folder from OneDrive business. But it turns out that this approach only ...
0
votes
0answers
92 views

Gnutls error when running wget

When I try and do the following: [john@seneca gnutls-3.6.9]$ wget https://www.atlassian.com/software/stash/downloads/binary/atlassian-bitbucket-6.6.0-x64.bin --2019-09-24 11:13:08-- https://www....
1
vote
1answer
1k views

wget forbidden but can download with firefox [closed]

I'm used to download addons from https://www.curseforge.com with my own script, but now, it's not working anymore. Each try I do, I get : $ LANG=C wget https://www.curseforge.com/wow/addons/...
0
votes
1answer
55 views

Using wget to download only the first depth of external links

I would like to get a copy of this page: http://databyte.ch/services/tony/index.html obviously this page is already a copy of the original source: http://www.tonyvanroon.com/oldwebsite/circ/circuits....
0
votes
1answer
261 views

wget not retrieving correct-size file (corrupt or incomplete file?)

I don't understand... The actual download link appears to not be a http link but rather some Javascript action? javascript:SendFileDownloadCall('PRODIMAGES.CIF.zip','PRODIMAGES.CIF.zip'); So after ...
-2
votes
2answers
2k views

Linux wget https://kali.sh/xfce4.sh problem [closed]

I am trying to install https://kali.sh/xfce4.sh in my kali linux to use GUI. However, if i type "sudo wget https://kali.sh/xfce4.sh", it says this: --2019-09-08 16:43:20-- https://kali.sh/xfce4.sh ...
0
votes
0answers
323 views

Error “Cannot write to” using wget

I want to make a backup script that downloads the content of a SharePoint folder. I got a working command using wget (when previously login and export the cookie.txt) using the following command: ...
-7
votes
1answer
53 views

Why am I getting these error messages? [closed]

Debian 10 sudo wget -O protonvpn-cli.sh gives wget: missing url. sudo protonvpn-cli -connect gives command not found I have not changed anything in any of the computer files. I do use a firewall ...
1
vote
1answer
167 views

rsync-like --delete functionality with wget (not wget's --delete-after)

Background: my particular problem, which gave rise to this question, is as follows. I'm a slackware linux user, and on 23-March-2019 mirrored their distribution with the following command wget -r -np ...
0
votes
1answer
673 views

Parsing output from wget and grep using bash

I can get a url link and search for text that starts with file: but I'm having issues parsing it from there. Example: wget -qO- http://website.com/site/ | tr \" \\n | grep -w file:\* > output....
0
votes
1answer
103 views

download entire website with wget

I want to download the entire wikipedia to have an offline copy, I try using that command: wget --recursive --domains --convert-links --no-clobber https://it.wikipedia.org/wiki/ But it download only ...
-2
votes
3answers
357 views

Piping of stream from wget to tar, and extracting it to a specific location failed

I try to pipe a stream from wget to tar and extracting it to a specific location. The file is downloaded by wget but not extracted as desired with tar: war="/var/www/html" domain="example.com" ...
0
votes
0answers
140 views

Mediafire Files with using WGET - Centos7

I just looked like 100 websites and didnt find anything. I cant download anything from MediaFire with using wget command on centos7. Actual file size is 700mb but wget is downloading like 20kb. i ...
0
votes
0answers
32 views

Wget check if local file matches remote file but do not download if the local file does not match or does not exist

I need wget -nc -c $remote_file for filelist, a fast way to check if possible local file matches the remote file but in case the local file doesn't match or doesn't even exist, download should not ...
0
votes
0answers
35 views

wget average data rate is not consistent with download file size divided by time

For a few experiments I ran wget to download a file to measure data rates. The file (size) is always the same, about 292 MiB. The average download speed shown by wget does not match the file size ...
0
votes
1answer
44 views

File transfer gone wrong

Newbie here I am trying to transfer backup files of several servers to a central server, using wget, using a script and a cronjob. But when the last file is being transferred, something goes wrong ...
0
votes
0answers
37 views

How to look for directory in http address

I'm trying to create a regex formula in shell script which will allow me to find desired folder in HTTP directory. Let's assume that this is the address: ftp://google.com/directories/ In this ...
0
votes
1answer
79 views

How to get triggered download from site with delay

Normally I use wget to get files that trigger a download. However, I'm currently using a site that takes a few seconds to generate the download, so wget downloads the html instead of the zip file I ...
1
vote
1answer
341 views

wget content-disposition ubuntu vs alpine

In Ubuntu, I can use the following wget flag to get a debian, i.e., wget --content-disposition https://packagecloud.io/xxxxx/download.deb dpkg -i ... In Alpine, wget complains that --content-...
-1
votes
2answers
281 views

How to use wget to download directory recursively, into directory with the same name as on the server?

Let's say I have a URL like so: https://mywebsite.com/files/myfolder How do I download all the files in the myfolder part (which is a directory), excluding files like index files, into a directory ...
0
votes
2answers
73 views

Get web page complete data

I'm using busybox tools and I want to take all http links in a web page. I save an example link page by using curl or wget. However, it saves the page as html. How to do it with curl or wget commands? ...
-1
votes
1answer
143 views

how to download ebi fastq link via command line using wget/curl

I have loads of data (fastq) to download from ENA database. Example: ERS076383 ERS012262 ERS160574 ERS349280 etc Link to download via browser: https://www.ebi.ac.uk/ena/data/warehouse/filereport?...
1
vote
0answers
53 views

Recursive wget, append something to each found URL

My main goal is to easily backup the page source for a bunch of MediaWiki pages of mine. I have every page I want downloaded, listed on my username page (https://wiki.example.com/index.php?title=User:...
0
votes
1answer
344 views

Unable to wget for ipv6

I tried wget using the following command wget -6 https://[fe80::] But it displayed the following error Connecting to [fe80::]:80... failed: Invalid argument.
0
votes
0answers
54 views

How can I make wget rename files based on link name?

Everything I'm finding is based on URL, but I want the opposite. I have an html index with a list of links. The links are named like Apples, oranges, grapes, bananas. The files associated with those ...
1
vote
1answer
124 views

wget -i notifying when each file finishes (for processing purposes)

I'd like to be able to process multiple files downloaded by wget -i immediately after they are downloaded (instead of waiting for all files in the list to finish--for the entire wget process to exit). ...
3
votes
2answers
1k views

wget download files from a file list, how to specify names for each of the downloaded file?

When downloading a single file, we can use wget's -O option to specify the file name. When I'm downloading Urls in a file using wget -i filelist.txt (filelist.txt contains list Of Urls I want to ...

1 2 3 4 5 13