When should I use Wget rather than a browser? I heard that Stallman uses it instead of a browser for anonymity reasons. What does the server think/see when you get its files using Wget?
closed as off-topic by Julie Pelletier, jasonwryan, Ulrich Schwarz, G-Man, Michael Homer Sep 5 '16 at 7:15
This question appears to be off-topic. The users who voted to close gave this specific reason:
In what case I should prefer use Wget rather than some browser ?
From the man page of wget:
GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
Wget is non-interactive, meaning that it can work in the background, while the user is not logged on. This allows you to start a retrieval and disconnect from the system, letting Wget finish the work. By contrast, most of the Web browsers require constant user's presence, which can be a great hindrance when transferring a lot of data.
The good thing about wget is you can use it in command line, or in a script and download file in background.
What server thinks/sees when you get its files using Wget ?
They see it as a browser requesting a file just like other browsers.
Here is the header of wget (this is what servers see):
GET / HTTP/1.1 User-Agent: Wget/1.18 (linux-gnu) Accept: */* Accept-Encoding: identity Host: localhost:80 Connection: Keep-Alive
And here is the header of firefox:
GET / HTTP/1.1 Host: localhost:80 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: zh-TW,en-US;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Connection: keep-alive Upgrade-Insecure-Requests: 1
As you can see, they are similar.
Maybe one of the best arguments is that
wget is capable of retrying if a connection is unreliable, and can even instruct the server to 'reget', to continue download from the point where the previous download failed - which is very welcome when downloading large files. Some browser plugins are capable of this too.
Also, very useful, is that
wget can download entire directories (including subdirectories), and does so avoiding repetitions in the download.
About anonymity - that is very relative. Normally,
wget identifies as such, and more than a few site take that as a bad thing - some block wget.
But, it is possible to make
wget identify as any other string, using the
--user-agent=agent-string command line option. (Again, similar tricks are possible with some browser plugins)
So, basically, this is a question of taste... A good argument is that
wget can be used easily in scripts, which is more complicated with browsers.
There are a lot of features in a graphical browser(
Chrome, etc...), they will be used to identify the user.
Not just HTTP headers.
They are called "browser fingerprint".
- Installed plugins
- Current screen resolution
- Installed fonts
- Supported features
curl, etc...) does not have a lot of these "fingerprint", because these are very simple.
This increases the anonymity, make it difficult to the tracking of the user.