Hot answers tagged wget
25
The main differences are:
wget's major strong side compared to curl is its ability to download recursively.
wget is command line only. There's no lib or anything but curl Features and is powered by libcurl.
curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...
9
wget will only retrieve the document. If the document is in HTML, what you want is the result of parsing the document.
You could, for example, use lynx -dump -nolist, if you have lynx around.
lynx is a lightweight, simple web browser, which has the -dump feature, used to output the result of the parsing process. -nolist avoids the list of links at the end, ...
6
On many Linux/Unix systems, your pseudocode will just work in any shell, although your paths should really be full URLs.
For instance, on Debian-based systems, the package libwww-perl installs three symlinks to lwp-request which are called /usr/bin/GET, /usr/bin/HEAD, and /usr/bin/POST. These do what you would expect. Recent versions of OpenSuse's ...
5
Yes. From the manpage:
-k, --insecure
(SSL) This option explicitly allows curl to perform "insecure"
SSL connections and transfers. All SSL connections are attempted
to be made secure by using the CA certificate bundle installed
by default. This makes all connections considered ...
5
lftp:
$ lftp http://repo.xplico.org/pool/
cd ok, cwd=/pool
lftp repo.xplico.org:/pool> ls
drwxr-xr-x -- /
drwxr-xr-x - 2012-02-13 09:48 main
lftp repo.xplico.org:/pool> cd main
lftp repo.xplico.org:/pool/main> ls
drwxr-xr-x -- ..
drwxr-xr-x - 2012-02-13 09:48 x
Directory listings only work for websites that do send ...
4
Since the django directory itself is a subdirectory of the resulting gzip content, you'll need to use the --strip-components=1 option of GNU tar. Assuming you're using GNU tar. Then you need to tell tar that you're looking for a subdirectory called django.
wget -qO- https://github.com/django-nonrel/django-nonrel/tarball/develop | tar --strip-components=1 ...
4
Try looking at the file you've received:
$ wget http://downloads.raspberrypi.org/images/raspbian/2012-07-15-wheezy-raspbian/2012-07-15-wheezy-raspbian.zip
-- snip --
2012-08-13 16:31:23 (9.78 MB/s) - `2012-07-15-wheezy-raspbian.zip' saved [6501]
$ file 2012-07-15-wheezy-raspbian.zip
2012-07-15-wheezy-raspbian.zip: HTML document text
We can see that ...
4
As you've discovered, this method is suboptimal at best. In addition to the HTML you'll need at a minimum all images and CSS. You may also need all of the Javascript. And then there's the whole deal of rendering this mess.
But for you there is good news in the form of a Command Line Printing extension you can install in Firefox.
Then:
firefox -print ...
3
No, but you could build Git without the Curl dependency on libcurl. It will disable features. Remember that wget is just a binary, whereas Curl provides a shared library as well and that is used by Git. Three options here:
./configure Git with the option --without-curl. Docs say:
--with-curl support http(s):// transports (default is YES)
...
3
You could specify a list of allowed resp. disallowed filename patterns:
Allowed:
-A LIST
--accept LIST
Disallowed:
-R LIST
--reject LIST
LIST is comma-separated list of filename patterns/extensions.
You can use the following reserved characters to specify patterns:
*
?
[
]
Examples:
only download PNG files: -A png
don't download CSS files: -R ...
3
This kind of site tries to make it difficult not to use a graphical web browser, because if you use wget you'll be missing all these ads that pay for the bandwidth.
Some sites don't make advanced checks and can be tricked easily: tell wget to pretend that it's really Mozilla and that it's coming from the download site.
wget --user-agent='Mozilla/5.0 ...
3
from the wget man page:
--no-use-server-timestamps
Don't set the local file's timestamp by the one on the server.
By default, when a file is downloaded, it's timestamps are set to
match those from the remote file. This allows the use of
--timestamping on subsequent invocations of wget. However, it is
...
3
No.
The -k option is not applied until the download has completed.
If wget is cancelled (ie, ctrl+c) then the -k option will not be invoked.
You can see this when wget completes, the last steps is the conversion of the links.
It tells you at the end, how many .html files it converted. Their timestamps also get updated during this process.
UPDATE:
Try ...
3
I think you're looking for
-np, --no-parent don't ascend to the parent directory.
Thus:
wget -r -l 0 -np --user=josh --ask-password http://morris.cs.example.com/files/Software/MySoftware/V2012_Linux/
Wget doesn't give much flexibility with output file names. If you want recursive downloading, you have to let the structure of the ...
3
As pointed out in an answer to a related question, wget's documention says:
Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version ...
3
I would suggest using curl to do this instead of wget. It can follow the redirection using the switches -L, -J, and -O.
curl -O -J -L http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.1/bitcoin-0.8.1-linux.tar.gz/download
switch definitions
-O/--remote-name
Write output to a local file named like the remote file we get.
(Only the ...
2
You should be able to watch the web server logs, in conjunction with the wget run. Look for the 404's in the log file and pull the referrer field. That will tell you the page that contains the broken link.
It should then just be a matter of examining that page for the offending link.
2
This is most likely because the device is out of space or out of inodes. Use df to find how much space is left. Also use df -i to find out if there are any inodes left.
If you are simply out of space, you may be able to fix it by changing your partition sizes. gParted is a program that can do this without destroying data. If you are out of inodes, or if ...
2
It seems redirection is disabled at your wget, try specifying --max-redirect=2 or a higher number. You probably also want to specify --trust-server-names to allow wget to update the file name after redirection, otherwise the downloaded files will still be called download.
2
You have to parse the download-page. Disable scripting in your browser and open the web-page-source code.
When I do this, I see the following "interesting" part:
<p>
<img alt="Download Video" src="http://media.videolectures.net/rel.1350055487/icons/download.png" />
Download <a ...
2
The problem is difficult because the full pictures are not under the parent's tree, so it's hard to distinguish those paths from any others on the site. Also, the links to the full pictures are actually links to pages, in which the full resolution picture is embedded. There may be a more elegant solution, but here's one way to do it that works.
#!/bin/bash
...
2
By default, wget writes to a file whose name is the last component of the URL that you pass to it. Many servers redirect URLs like http://www.url1.com/app?q=123&gibb=erish&gar=ble to a different URL with a nice-looking file name like http://download.url1.com/files/something.pdf. You can tell wget to use the name from the redirected URL (i.e. ...
2
You could loop over the entries in your list_of_urls. Something like this:
while read -r url; do
wget -O foo $url
done < list_of_urls
Note that you'll have to add your own way of determining foo for each entry of the list_of_urls (also, I'm assuming this is a file on your disk).
2
You could use any local DNS caching daemon like Dnsmasq or Bind in a caching-only configuration. Then the most recent responses will be cached locally and multiple instances of wget will not trigger extra queries over the network for the same names. Wget may be set to --no-dns-cache to save some memory at cost of performance.
2
Unfortunately the true fix will most likely be using the latest version of wget as the issues have probably been fixed there. 1.11 is about 5 years old. There is a 1.11.4 too if you happen to be using the original 1.11.
Alternatively (and this is horrible.. and might not work depending on what is triggering the file descriptor usage):
Increase the file ...
2
Wget and Curl only parse links within the anchor tags on a HTML Document.
The page you are referring to, uses a POST Method with the link to the document to download it.
You will have to download the file and parse it manually for all links. This is something that wget will not do for you.
EDIT: However I do not know why you are receiving a protocol ...
2
I'd use httrack if you're trying to mirror a directory of *.pdf files. A command like this worked for me:
% httrack http://my.server.com/dir/with/pdfs/ +mime:text/html +*.pdf --update
You can check out more about httrack at the following URLS:
Basic Users Guide
Download Links
example commands
Blog article was additional examples
I should mention one ...
1
i think this will work
grep -oP "http:\//[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(:[a-zA-Z0-9]*)?/?([a-zA-Z0-9\-\._\?\,\'/\\\+&%\$#\=~])*.jpg" inputfile | xargs -n1 wget
or if you have list then you can use
cat list.txt | xargs -n1 wget
1
You have 2^H4 ways:
wget one page, gunzip it and process it again from the html... iterate until finish:
wget -m http://example.org/page.html
find . -name \*gz -exec gzip -d {} \;
find . -name \*html -exec wget -M -F {} \;
this will be slow, but should work
install privoxy and configure it to request uncompress the pages
+prevent-compression
...
1
The reason it's not saving duplicates for you is because you are using the --page-requisites (-p) option. Try adding the --no-directories (-nd) option to your command.
From the manpage (strong emphasis added by me):
When running Wget without -N, -nc, -r, or -p, downloading the same file in the same directory will result in the original copy of file ...
Only top voted, non community-wiki answers of a minimum length are eligible
