wget - command-line utility to download content non-interactively.

learn more… | top users | synonyms

1
vote
1answer
18 views

Download HTTPS website available only through username and password with wget?

I'm trying to recursively download website which is normally available only when you login. I have valid username and password, but the problem is that I need to login through web interface, so using ...
-1
votes
1answer
29 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
32
votes
3answers
8k views

Why doesn't cp have a progress bar like wget?

Please note that I don't ask how. I already know options like pv and rsync -P. I want to ask why doesn't cp implement a progress bar, at least as a flag ?
2
votes
1answer
358 views

How to implement retry mechanism while executing the server url in the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
2
votes
1answer
31 views

wget on the fly to extract specific info from file without downloading the entire file

I have some files (which are in .bam format(binary)) to download from a location. I can download them using wget and then want to extract some specific information from them only. For that I use ...
0
votes
0answers
30 views

Redirect with axel in linux

I am trying to download a file from googlevideo. This download link may expire in a few minutes but I will post everything I have. When I use axel with the following command, I get $ axel -n 10 ...
0
votes
2answers
2k views

How to disable gzip compression with wget?

There are other questions asking how to enable gzip compression with wget, and lots of web pages out there telling how to do this, but I need the opposite. I'm trying to locally mirror a site, and ...
5
votes
3answers
345 views

command-line tool for a single download of a torrent (like wget or curl)

I'm interested in a single command that would download the contents of a torrent (and perhaps participate as a seed following the download, until I stop it). Usually, there is a torrent-client daemon ...
0
votes
1answer
39 views

wget: force no default certificates

I'm having trouble with verification of custom server SSL certificates and wget. wget -O- --ca-certificate=myservercert.pem https://www.google.com This should fail but does not, as wget does ...
1
vote
1answer
24 views

Using date -1day with wget

I want to use wget with a cron to pull a file everyday however I want it to pull the previous days file from the remote server and the file is controlled by date variables in the URL as such: ...
2
votes
1answer
44 views

Using wget, how to download to a specific location, without creating folders, and always overwrite original files

How do I use wget to download files to a specific location, without creating directories, and overwrite the original every time. I've tried using the -r -P and nc options in combination but this ...
2
votes
1answer
30 views

How to download a lot of documents on the terminal from a server?

I have to write a bash script (.sh) with less than 300 characters which downloads all of the genomes 22 in this server ftp://ftp.ncbi.nih.gov/genomes/Homo_sapiens/ (the ones with fa.gz extention) and ...
0
votes
2answers
870 views

download multiple files from sourceforge

Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool? I've seen some examples using rsync or scp but I think that works only for SF ...
2
votes
1answer
56 views

Download a file to a server?

After ssh to a server, I want to download two files in https://iain.cx/src/ktmux/#download to the server. So for the first file ktmux_helper, I right click to find its link on my local computer, and ...
0
votes
1answer
109 views

bash cannot find mvn command after install

I tried to install Maven on CentOS 7, but afterwards typing mvn -version in the terminal results in bash: mvn: command not found.... How can I fix this so that I can call maven from the CentOS 7 ...
2
votes
2answers
5k views

wget to get all the files in a directory only returns index.html

I'm new to using bash, and I have been trying to wget all the files from a website to the server I have been working on. However all I'm getting back is an index.html file. I let it run for 15 minutes ...
2
votes
2answers
86 views

Why wget cannot download page as root?

This is odd, I try to download a html file using wget wget --user=xxx --password=yyy http://192.168.1.1/ When I execute it as myself, it works, when I execute it as root it fails with message: ...
0
votes
2answers
56 views

How do you rename files specifically in a list that wget will use?

I have a file with a list of links to some youtube videos. When I run the command: wget -i list Everything works just fine, however wget is automatically renaming the files to the gibberish at ...
3
votes
2answers
2k views

Problem with recursive download using wget

I am trying to learn how to use recursive download with wget from the wget info page. For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd ...
2
votes
1answer
27 views

Multiple user credentials in wget

Using wget, is it possible to supply more than one user/password pair when mirroring a website that has multiple password-protected areas with different credentials? I would like to mirror an entire ...
4
votes
2answers
321 views

Bash Script - Wget Graphical Download Progress

I'm trying to port this code from wget + zenity to wget + kdialog. The code with zenity: #!/bin/bash # Start wget | zenity # Note the & at the end of the pipe, this allows the script to continue ...
4
votes
3answers
11k views

Can't wget from github, sslv3 handshake error

We have a script that downloads mod security from github that has recently started failing. The servers run CentOS 6 but RHEL 6 probably has the same issue. The output is: # wget ...
7
votes
2answers
6k views

Using wget, What is the right command to get gzipped version instead of the actual html

I stumbled upon this website which talk about this. So when downloading whole website by getting the gzipped version, what is the right command? I've tested this command, but I dont know if wget ...
-2
votes
3answers
971 views

What is the meaning of wget -O -

I am trying to install dropbox onto my Debian machine, and I saw the instruction cd ~ && wget -O - "some website leading to the download of a tar file" | tar xzf - But what I did was ...
3
votes
4answers
601 views

tar a result of wget

I want to wget a file and tar it, in one command, I guess it is simple but I can't get it done. I tried several things. wget <url> | tar -cvz file.gz.tar - tar -cvzf file.tar `wget ...
21
votes
3answers
12k views

why would curl and wget result in a 403 forbidden?

I try to download a file with wget and curl and it is rejected with a 403 error (forbidden). I can view the file using the web browser on the same machine. I try again with my browser's user agent, ...
5
votes
2answers
455 views

How can I download a file from a FTP server, then automatically delete it from the server once the download completes?

From the command line, I want to download a file from a FTP server. Once the download completes, I want the file to be deleted on the server. Is there any way to do this? Originally I considered ...
1
vote
0answers
137 views

Wget, downloading outdated files

I am over writing a particular file (called "demo") on my server and wget-ting it. However, sometimes wget retrieves an outdated file, despite my checking on the net that the file has been updated. ...
1
vote
1answer
38 views

Combine xargs argument with some text

How can I use xargs to pass an argument through to wget, but also add some text to that argument?
1
vote
1answer
21 views

remove the bit per bit log of wget

I have a php script that generates bash commands to wget files. I am trying to log wget activity using this: --directory-prefix='xxx' --append-output='xxx.txt' -c -t0 -S" But the log is too big due ...
0
votes
2answers
38 views

Filter hyperlinks from web page and download all that match a certain pattern

I wanted to download all graphic files from our organisation's graphics repository web page. They are Illustrator (.ai) format and Corel Draw (.cdr) format. They are directly hyperlinked (i.e. <a ...
0
votes
2answers
151 views

linux curl POST request not working

I am trying to make a POST request using curl to retrieve exam results from the page : result odd .... I am using the following syntax but various errors are showing up (like content has moved ...
3
votes
4answers
5k views

Is there a way to disable wget from getting files from parent directories to given depth?

wget has such option as -np which disables getting files from any parent directory. I need something similar but a bit more flexible. Consider: www.foo.com/bar1/bar2/bar3/index.html I would like to ...
0
votes
0answers
38 views

Re-download Corrupted Part of ISO

I have download an ISO file from one of the public FTP server. It took a lot of time to download the file because of my Internet Connection Issues. I used the wget command to download the ISO in ...
0
votes
2answers
65 views

Why doesn't wget -r get all FTP subdirectories?

When I did the command : wget -r ftp://user:paswword@ftp.xxxx.ovh.net/ It's missing any sub-sub-directories. Does recursive FTP have a limit?
2
votes
1answer
174 views

CURL won't connect to self

I'm having a weird issue where CURL cannot connect to its own server. For example if I'm on server example.com wget https://example.com/ <--- WORKS FIND curl https://example.com gets error: 35 ...
3
votes
3answers
66 views

how to remove accidentally created '~' and '..'

I'm using wget to download some webpages and I used the -P option without knowing that it wouldn't expend .. or ~. So I found I've created them after running the program. Now the problem is, how do I ...
6
votes
3answers
6k views

How to download files with wget where the page makes you wait for download?

I am trying to download a file from sourceforge using wget, but as we all know we have to click on the download button and then wait for it to auto download. how do you download this type of file ...
0
votes
1answer
134 views

Download Forum Attachments

Is there a way to download forum jpeg image attachments, possibly using Wget or Curl or some other tool? I would like to download jpeg attachments from specific pages on forums. I'm not interested in ...
0
votes
2answers
113 views

Downloading images from csv file to a folder based on its id

I have a csv-file that looks like this (short example, there are actually about 20 images per id). ...
0
votes
0answers
77 views

wget not working but apt-get does

Although this question relates to Raspbian running on my Raspberry Pi, it feels like it is a more general Linux query.... The Pi is sat on a network which includes a Microsoft TMG server. I have ...
1
vote
1answer
97 views

wget -k doesn't convert all links

I am trying to mirror a complete website to another domain. I am trying to use wget -mk http://example.com However the -k option leaves some URLs original while other are converted such as: ...
2
votes
1answer
96 views

How do I securely download files in shell scripting?

I am writing some shell scripts to automatically download software packages off my server and set them up on customer servers. At some point within the script, the download command is wget ...
0
votes
0answers
22 views

How to keep directory structure with aria2?

I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure. Or maybe I should try another program?
2
votes
2answers
154 views

How to use wget to download certain files from a certain directory when the directory itself does not have an index.html?

There are a number of questions similar to this on StackExchange but none address this issue. More specifically I want to download all the pdf files in the 2007 directory at ...
47
votes
3answers
14k views

What is the difference between curl and wget?

I am keen to know the difference between curl and wget. Both are used to get files and documents but what the key difference between them. Why are there two different programs?
0
votes
1answer
92 views

wget and preventing files from downloading on a recursive wget

I can use a command like wget -m -A.jpg http://www.mysite.tld/ but that will download every single file, then if it is not a jpg, it will delete the file, like this: `HTTP request sent, awaiting ...
-2
votes
2answers
104 views

Wget - Download pics with particular url string

Is there a Wget command that will allow me to download all jpegs that include the following url: http://www.sample.com/images/imag/ So all jpegs under that particular url will be downloaded, such as ...
2
votes
0answers
63 views

Wget: convert-links and avoid redownloading already fetched files?

I am downloading data spread among multiple files that don't change once published. Therefore --timestamping is not good enough, because it is constantly checking whatever resource is changed, which ...
0
votes
0answers
32 views

wget --exclude-directories not working for all

I am making a script for downloading a single file from a website. So I am trying to exclude the remaining directories. Here is it: #!/bin/bash grep -O '<a href=".*>'*html wget \ ...