Questions tagged [wget]

wget - command-line utility to download content non-interactively(can be called from scripts, cron jobs , terminals without the X-Windows support, etc.)

5
votes
0answers
97 views

Continue a download if timestamp has not changed

I'd like to mirror a site with large, updating database dumps. I don't entirely understand the -N and -c options and how they combine after reading the wget man page. Is there a way to download a ...
3
votes
0answers
1k views

Stop wget reusing existing connection?

so I am trying to wget a specific webpage using this command in bash scripting: wget --no-cookies --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) ...
3
votes
0answers
437 views

WGET - Authenticating root AND all children folders

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below. There's a website that I'm trying to mirror. It is something ...
2
votes
0answers
178 views

wget regex options ignored over ftp

Wget specifies the --accept-regex and --reject-regex options for limiting recursive downloads. I'm unable to get them to work with an ftp connection. For example, I would expect --reject-regex='.*' ...
2
votes
0answers
170 views

Turning on accept/reject options in wget changes file download behavior

I am trying to connect to a Sharepoint site and fetch Excel workbooks from a specific folder. The issues I am having are mainly that wget is fetching files from every single level. Here is the code I ...
2
votes
0answers
459 views

cURL and wget cannot resolve hosts, but apt-get works

I've installed Ubuntu server 16.04 in a VirtualBox. If I try to install software with apt-get it works fine, but curl test.de or wget get.typo3.org/8.4.0 shows me that the host could not be resolved. ...
2
votes
0answers
468 views

Problem using wget to create static archive of website

I have been using the wget command to create static archives of a number of websites that I no longer want to keep updating. This command has worked well for a number of sites: wget --recursive --no-...
2
votes
0answers
1k views

wget resume download from dropbox gives can't write error

I want to resume download from dropbox using wget. I was downloading a file wget -O SQLyog.exe "https://www.dropbox.com/s/dfv2z0le0c2bo7z/SQLyog-9.3.3-0Trial.exe?dl=1" .......... Length: 15775640 (...
2
votes
0answers
857 views

Why can't I transfer a file via wget (FTP) using exec() function in PHP?

Put simply, I can't work out why the code below won't carry out a file transfer? I have been assured by the sysadmin staff on the remote server that there are no permissions/firewall issues that could ...
1
vote
0answers
23 views

Recursive wget, append something to each found URL

My main goal is to easily backup the page source for a bunch of MediaWiki pages of mine. I have every page I want downloaded, listed on my username page (https://wiki.example.com/index.php?title=User:...
1
vote
0answers
169 views

GET via wget or cURL gives a limited response

I'm bumping my head on a simple HTTP GET request. Using a tool like Postman, I'm getting the full and expected response (an over 1 thousand lines xml document, displaying the details of my 60 albums) ...
1
vote
0answers
63 views

wget behaves differently on cygwin and centos

On cygwin this command works as expected and exits immediately: wget --post-file oryxblock.2018-10-10_20_40-07_591.204.txt --output-document - --header "Content-Type: text/csv" http://htm-psycho-402....
1
vote
0answers
65 views

Why does wget --mirror not work on this site?

Why is wget -m "httpS://www.addnature.se" refused while wget -m "http://www.addnature.se" just downloads the start page? I frequently use wget -m to mirror sites and have never experienced this ...
1
vote
0answers
62 views

How to find the real download URL

I need to download the 64-bit Linux tar file from this webpage: https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm The 64-bit Linux URL appears to be: https://www....
1
vote
0answers
88 views

openssl hangs on server

I have a problem on the server. Commands wget, curl and dependent commands are not working when a request is via 443 port. Commands hangs on 2-3 minutes and after just say "time out": # curl -v ...
1
vote
0answers
611 views

wget can't connect to domain on same server

I'm trying to run the following command on the server. The site example.com is hosted on the same server. wget --spider "https://example.com/script.php?actions=1" Response: Spider mode enabled. ...
1
vote
0answers
747 views

Automatic Login with cURL without php

I'm working in a little project using a script to do scraping. I'm trying login in https://www.plusdede.com/login, it's a simple login with a numeric captcha. The code which is important is: <div ...
1
vote
0answers
328 views

Can I wget or download files that are not linked to from a server?

I have a url such as myurl.com/contents and I am trying to fetch all contents in the folder. Some files, such as myurl.com/contents/unlisted-file.1 are not linked to anywhere. These files are ...
1
vote
0answers
50 views

Ignore a GET variable from links, when downloading a website for offline

Quite simply the site I'm downloading has every single link ending with &sid=335345346fdsfdsfs. That keeps changing and creates bunch of the same index.html and the same for many other pages. I ...
1
vote
0answers
51 views

Why does wget keep fetching recursively despite -l5?

I'm trying to mirror a website with wget. It works out just fine until a strange thing occurs. I'm using this line: wget -e robots=off --wait 1 -l5 --mirror --convert-links --adjust-extension --page-...
1
vote
0answers
620 views

wget - How do I download webpages based on URL Pattern?

Consider a website www.music.com with the following directory structure: /piano /covers /Chopin apple.html bannan.js balloon.html index.html /Franz Liszt ...
1
vote
0answers
93 views

How to download files from a webpage that points to a directory?

I'm trying to download all the .mp3 files from here; but the download links are not .mp3 . (ex:https://freemusicarchive.org/music/download/98281de10b6cc09df9e99a5a3bd2c7ee0eddaf82) links but a ...
1
vote
0answers
359 views

Problem with downloading large file from internet

Since a while ago, whenever I am downloading a large file with maven it takes a long time. I also tried with just wget and still had the same problem. It will freeze for 10 or more minutes after ...
1
vote
0answers
1k views

how to download a private google spreadsheet from linux command-line?

I have a Google Sheets spreadsheet associated to a Google account, which is private and cannot be made public. I would like to know how to download a private google spreadsheet from the command-line ...
1
vote
0answers
129 views

How to get all files of a certain type from a virtual web path?

I try to use wget to download certain files from a website. But probably the virtual locations of the files do not represent their actual location. For example, if the link to the file is http://file....
1
vote
0answers
399 views

How to download thousands of files quickly?

I need to download thousands of files (approximately 2-12 MB each) - Currently I am using the following command wget --reject "index.html*" -A *.gz -N -r -l1 -nd --no-check-certificate --user ...
1
vote
0answers
191 views

Download a webpage by putting everything it needs in a dedicated directory?

To download a webpage ( its html file and other files needed by it), wget seems to mirror the directory structure of the files of the webpage on the website server When in a web browser (e.g. Firefox ...
1
vote
0answers
223 views

How to abort wget after getting N times file not found?

I try to grab pictures from few galleries of some site. I do not know how many files are there, but I know the files are continuous named. So I use a regex like that: wget http://example.org/...
1
vote
0answers
361 views

Using wget, download a website translated by google translate

I want to download(recursively) http://e-maxx.ru/algo , after it has been translated to english using Google translate.
1
vote
0answers
209 views

Wget, downloading outdated files

I am over writing a particular file (called "demo") on my server and wget-ting it. However, sometimes wget retrieves an outdated file, despite my checking on the net that the file has been updated. I'...
0
votes
0answers
43 views

xargs: wget: No such file or directory

Am not good in unix commands. am downloading files listed in the raw_data_urls.txt to my desktop. My sh command is as below: cat setup_files/raw_data_urls.txt | xargs -n 1 -P 6 wget -c -P data/ ...
0
votes
0answers
34 views

How can I make wget rename files based on link name?

Everything I'm finding is based on URL, but I want the opposite. I have an html index with a list of links. The links are named like Apples, oranges, grapes, bananas. The files associated with those ...
0
votes
0answers
136 views

wget: error getting response: Address family not supported by protocol

Attempt to retrieve entware script from a test router indicates a wget error: root@DD-WRT:/opt# wget http://bin.entware.net/mipselsf-k3.4/installer/generic.sh Connecting to bin.entware.net (104.27....
0
votes
0answers
30 views

How to receive data from a Webpage and send it back to it in the same connection?

I am trying to fill a form of a webpage with curl and give it some arguments. Dumb enough, it also sends a hidden form in which there is an unique token which has to be sent back to the page, a so ...
0
votes
0answers
119 views

Wget wildcard files without temp html files

Hello I've been trying to find the correct solution to download wildcard files using wget - I used to have nexus api calls to download the latest files, but unfortunately I have to use pattern ...
0
votes
0answers
83 views

wget isn't converting some links properly

I'm trying to get a "working copy" of google.com using wget. The command I use is: wget --page-requisites --html-extension --convert-links --restrict-file-names=windows --no-host-directories -U "...
0
votes
0answers
64 views

calling wget from a browser using php

My httpd is running with a daemon user. I want to do a wget using php from FTP server. When I run the php from linux server, the same downloads the required file from FTP. But when I call the same php ...
0
votes
0answers
126 views

wget output progress to file per downloaded file

I'm downloading multiple bigger files with wget like this: wget -i 'FILE_WITH_URLS' --content-disposition -b. By default it creates a file named wget-log and appends time download progresses. The file ...
0
votes
0answers
63 views

wget --spider ram memory consumption and use of -c

I am using wget --spider. It is consuming a lot of ram memory. How could I limit the use of ram memory from the wget command? wget --spider -r -c http://www.example.org 2>&1 | grep '^--' | awk ...
0
votes
0answers
241 views

Download filename-pattern-matching files from a public Google Drive folder with rclone (or alternative)

I'm trying to download all files from a public gdrive folder but with certain filetype restrictions (eg exclude *.avi,*.mkv,*.mp4 etc). I originally tried with wget since that is what I am familiar ...
0
votes
0answers
28 views

What is the fastest tool to sync http listening files

I have a huge number of files (text files) listening on http server. I am using cron-job to download these files as the number of files will grow dramatically. I am using wget as follows to download ...
0
votes
0answers
105 views

wget with acceptance test, redirect fails the test

I'm running the command: wget -r --wait=2 --random-wait -P ./pdfs -A pdf -e robots=off https://www.<somesite>/documents The command has worked on other sites to recursively download all pdfs, ...
0
votes
0answers
53 views

File from wget in script not available for next command

I am wget-ing a file and to preserve its original file name i do a find for its known extension. Everything is working fine on the terminal but in a script the file is never available/there for the ...
0
votes
0answers
1k views

How do I figure out why I cant connect using “wget” to an external web site?

I'm using CentOS 7. I'm trying to figure out why I can't connect via wget to an external web site, although most other web sites work and I can connect to this other web site from other computers. ...
0
votes
0answers
950 views

Download a URL containing a token query string

To explain what I want I will give an example (My foreign friends sorry it is in Arabic not English :D ) . Let's take this URL for example. When you open it a blue square will appear after a 5 ...
0
votes
0answers
123 views

How do I figure out what is blocking port 80?

We are running Apache on a Debian 8 box, which is behind a router. I can tell Apache is running and serving pages because when I log in to the box, I'm able to wget the web page, using both the "...
0
votes
0answers
655 views

Crawl Website With Wget With Filter by URL Patterns

I am trying to crawl a website and want wget to recursively follow all the links (that are within the same domain) but only download the page if the URL of that page matches a pattern. I tried ...
0
votes
0answers
281 views

Recursive download of only one format with wget?

I want to download only jpg files present a HTTP server. I should be able to do that with wget -r -A "*.jpg" 192.168.1.2 but it does not download ".jpg" files. It only downloads index.html and then ...
0
votes
0answers
891 views

How to compile wget or curl with https support statically on armv5tejl?

I have an IP camera with BusyBox. It has armv5tejl architecture When I want to use wget to download something from https://.... When I try to use it with https I get an error: wget: not an http or ...
0
votes
0answers
95 views

wgetting links that are written in script form - not plain html

I use wget to download all the html pages in an online gallery - in order to grab all the links to the full size images as follows: wget --wait=3 --user-agent=Webtography --no-cookies --timestamping -...