wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

0
votes
1answer
29 views

Continuing an interrupted `wget` session?

Is it possible to continue an interrupted wget session - eg. by parsing the log-file (created with -o or -a ), or after somehow have wget store additional information to disk (like it's list over ...
1
vote
1answer
40 views

wget fails to exclude directories

I want to download files from a site, but I want to exclude specific directories. I am using the following command: wget --user="xxxx" --password="xxx" -r -np -nH --cut-dirs=7 \ --reject ...
0
votes
0answers
56 views

How to download files and folders from Onedrive using wget?

How to use wget to download files from Onedrive? (and batch files and entire folders, if possible)
0
votes
1answer
23 views

Get total size of a list (containing HTTP links)

I have a file containing a list of files on the web, and I would like to know the total files' size (in human-readable format) and check their availability. Is there a command/solution to do so? P.S. ...
0
votes
2answers
36 views

wget, logging the output and the response

I calling an url using wget. This url gives me a response, its a Message id. I want to write the logs to a log file, with the message id as well. Also the log should be appended each time. I trying to ...
0
votes
2answers
39 views

Issue when running wget inside a bash script with an url stored in a variable

I'm having the following bash script: URL=`grep -E -m 1 -o "<ExportCatalogResult>(.*)</ExportCatalogResult>" costa_export.xml| sed -e ...
0
votes
1answer
51 views

WGET Continue without --no-check-certificate

I'm using wget to check if several boxes have access to other servers. The problem is sometimes the servers attempt to self-sign and the script hangs when this happens. It simply stops on the line ...
2
votes
1answer
151 views

wget doesn't support https

I want to download https://example.com using the wget command, but it says “unsupported/unrecognized protocol”. I tried using the --secure protocol='' but that does not work. I downloaded the tar file ...
1
vote
1answer
37 views

How do I display all URLs in a redirect chain?

I'm looking for a way to show all of the URLs in a redirect chain, preferably simply and from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like ...
1
vote
1answer
20 views

wget a url from my public outdoor wx cam

I am trying to grab an image from my weathercam using wget and cron. For now I just need to see how to get an image downloaded and timestamp the filename. ...
1
vote
2answers
41 views

How do I get size of (deb) file from download link/URL?

I use --print-uris of apt-get to get url (download-link) of debs I want to download and install on my Debian (Trisquel) GNU/Linux. --print-uris Instead of fetching the files to install ...
1
vote
0answers
44 views

How to download thousands of files quickly?

I need to download thousands of files (approximately 2-12 MB each) - Currently I am using the following command wget --reject "index.html*" -A *.gz -N -r -l1 -nd --no-check-certificate --user ...
1
vote
2answers
42 views

Put two Cron jobs into crontab using wget

I am trying to put these two Cron Jobs: 0 3 * * * ! sudo -u asterisk /var/lib/asterisk/bin/module_admin --repos extended,standard,unsupported upgradeall 30 3 * * * ! sudo -u asterisk ...
1
vote
0answers
31 views

Download a webpage by putting everything it needs in a dedicated directory?

To download a webpage ( its html file and other files needed by it), wget seems to mirror the directory structure of the files of the webpage on the website server When in a web browser (e.g. Firefox ...
5
votes
2answers
268 views

Download a list of webpages in a file and rename them to avoid overwriting?

$ wget -r -np -nH --cut-dirs=2 -p -k "http://www.example.com/1" -O test.html Cannot specify both -k and -O if multiple URLs are given, or in combination with -p or -r. See the manual for details. ...
1
vote
2answers
52 views

Equal command between wget and cURL (with options)?

I usually use wget to download remote file, the command I use: wget --timeout=5 --tries=1 --referer=http://www.example.com 'http://www.example.com/abcdefg123.jpg' -O image.jpg As you can see, there ...
0
votes
1answer
49 views

Whats the best way to copy a website onto your harddrive?

I like the website www.globalresearch.ca and I have made copies of it. This command works for me wget -r -l14 -t2 -T60 -E -k --no-check-certificate --restrict-file-names="windows,nocontrol" ...
3
votes
1answer
19 views

wget from ftp: supress default page creation

I'm trying to download subfolder from ftp. There are only data files. I'm using: wget -nH -np --cut-dirs=5 -r http://sss.com/1/2/3/4/5 Unfortunately wget creates one additional file: index.html ...
4
votes
5answers
328 views

Download HTTPS website available only through username and password with wget?

I'm trying to recursively download website which is normally available only when you login. I have valid username and password, but the problem is that I need to login through web interface, so using ...
-1
votes
1answer
68 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
2
votes
1answer
70 views

wget on the fly to extract specific info from file without downloading the entire file

I have some files (which are in .bam format(binary)) to download from a location. I can download them using wget and then want to extract some specific information from them only. For that I use ...
0
votes
0answers
140 views

Redirect with axel in linux

I am trying to download a file from googlevideo. This download link may expire in a few minutes but I will post everything I have. When I use axel with the following command, I get $ axel -n 10 ...
5
votes
3answers
933 views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
1
vote
1answer
47 views

Using date -1day with wget

I want to use wget with a cron to pull a file everyday however I want it to pull the previous days file from the remote server and the file is controlled by date variables in the URL as such: ...
0
votes
1answer
118 views

wget: force no default certificates

I'm having trouble with verification of custom server SSL certificates and wget. wget -O- --ca-certificate=myservercert.pem https://www.google.com This should fail but does not, as wget does ...
2
votes
1answer
62 views

Using wget, how to download to a specific location, without creating folders, and always overwrite original files

How do I use wget to download files to a specific location, without creating directories, and overwrite the original every time. I've tried using the -r -P and nc options in combination but this ...
2
votes
1answer
37 views

How to download a lot of documents on the terminal from a server?

I have to write a bash script (.sh) with less than 300 characters which downloads all of the genomes 22 in this server ftp://ftp.ncbi.nih.gov/genomes/Homo_sapiens/ (the ones with fa.gz extention) and ...
0
votes
1answer
1k views

bash cannot find mvn command after install

I tried to install Maven on CentOS 7, but afterwards typing mvn -version in the terminal results in bash: mvn: command not found.... How can I fix this so that I can call maven from the CentOS 7 ...
2
votes
1answer
68 views

Download a file to a server?

After ssh to a server, I want to download two files in https://iain.cx/src/ktmux/#download to the server. So for the first file ktmux_helper, I right click to find its link on my local computer, and ...
2
votes
2answers
166 views

Why wget cannot download page as root?

This is odd, I try to download a html file using wget wget --user=xxx --password=yyy http://192.168.1.1/ When I execute it as myself, it works, when I execute it as root it fails with message: ...
0
votes
2answers
180 views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
4
votes
4answers
1k views

tar a result of wget

I want to wget a file and tar it, in one command, I guess it is simple but I can't get it done. I tried several things. wget <url> | tar -cvz file.gz.tar - tar -cvzf file.tar `wget ...
5
votes
2answers
758 views

How can I download a file from a FTP server, then automatically delete it from the server once the download completes?

From the command line, I want to download a file from a FTP server. Once the download completes, I want the file to be deleted on the server. Is there any way to do this? Originally I considered ...
1
vote
1answer
60 views

Combine xargs argument with some text

How can I use xargs to pass an argument through to wget, but also add some text to that argument?
1
vote
1answer
44 views

remove the bit per bit log of wget

I have a php script that generates bash commands to wget files. I am trying to log wget activity using this: --directory-prefix='xxx' --append-output='xxx.txt' -c -t0 -S" But the log is too big due ...
0
votes
2answers
96 views

Filter hyperlinks from web page and download all that match a certain pattern

I wanted to download all graphic files from our organisation's graphics repository web page. They are Illustrator (.ai) format and Corel Draw (.cdr) format. They are directly hyperlinked (i.e. <a ...
0
votes
2answers
297 views

linux curl POST request not working

I am trying to make a POST request using curl to retrieve exam results from the page : result odd .... I am using the following syntax but various errors are showing up (like content has moved ...
0
votes
1answer
85 views

Re-download Corrupted Part of ISO

I have download an ISO file from one of the public FTP server. It took a lot of time to download the file because of my Internet Connection Issues. I used the wget command to download the ISO in ...
2
votes
1answer
530 views

CURL won't connect to self

I'm having a weird issue where CURL cannot connect to its own server. For example if I'm on server example.com wget https://example.com/ <--- WORKS FIND curl https://example.com gets error: 35 ...
3
votes
3answers
71 views

how to remove accidentally created '~' and '..'

I'm using wget to download some webpages and I used the -P option without knowing that it wouldn't expend .. or ~. So I found I've created them after running the program. Now the problem is, how do I ...
1
vote
2answers
127 views

Why doesn't wget -r get all FTP subdirectories?

When I did the command : wget -r ftp://user:paswword@ftp.xxxx.ovh.net/ It's missing any sub-sub-directories. Does recursive FTP have a limit?
0
votes
1answer
208 views

Download Forum Attachments

Is there a way to download forum jpeg image attachments, possibly using Wget or Curl or some other tool? I would like to download jpeg attachments from specific pages on forums. I'm not interested in ...
0
votes
2answers
202 views

Downloading images from csv file to a folder based on its id

I have a csv-file that looks like this (short example, there are actually about 20 images per id). ...
0
votes
0answers
181 views

wget not working but apt-get does

Although this question relates to Raspbian running on my Raspberry Pi, it feels like it is a more general Linux query.... The Pi is sat on a network which includes a Microsoft TMG server. I have ...
1
vote
1answer
222 views

wget -k doesn't convert all links

I am trying to mirror a complete website to another domain. I am trying to use wget -mk http://example.com However the -k option leaves some URLs original while other are converted such as: ...
2
votes
1answer
335 views

How do I securely download files in shell scripting?

I am writing some shell scripts to automatically download software packages off my server and set them up on customer servers. At some point within the script, the download command is wget ...
0
votes
0answers
35 views

How to keep directory structure with aria2?

I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure. Or maybe I should try another program?
2
votes
2answers
354 views

How to use wget to download certain files from a certain directory when the directory itself does not have an index.html?

There are a number of questions similar to this on StackExchange but none address this issue. More specifically I want to download all the pdf files in the 2007 directory at ...
1
vote
1answer
275 views

wget and preventing files from downloading on a recursive wget

I can use a command like wget -m -A.jpg http://www.mysite.tld/ but that will download every single file, then if it is not a jpg, it will delete the file, like this: `HTTP request sent, awaiting ...
-2
votes
2answers
298 views

Wget - Download pics with particular url string

Is there a Wget command that will allow me to download all jpegs that include the following url: http://www.sample.com/images/imag/ So all jpegs under that particular url will be downloaded, such as ...