1

When should I use Wget rather than a browser? I heard that Stallman uses it instead of a browser for anonymity reasons. What does the server think/see when you get its files using Wget?

closed as off-topic by Julie Pelletier, jasonwryan, Ulrich Schwarz, G-Man, Michael Homer Sep 5 '16 at 7:15

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question has been posted on multiple sites. Cross-posting is strongly discouraged; see the help center and community FAQ for more information." – Julie Pelletier, jasonwryan, G-Man, Michael Homer

2

In what case I should prefer use Wget rather than some browser ?

From the man page of wget:

GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

Wget is non-interactive, meaning that it can work in the background, while the user is not logged on. This allows you to start a retrieval and disconnect from the system, letting Wget finish the work. By contrast, most of the Web browsers require constant user's presence, which can be a great hindrance when transferring a lot of data.

The good thing about wget is you can use it in command line, or in a script and download file in background.

What server thinks/sees when you get its files using Wget ?

They see it as a browser requesting a file just like other browsers.

Here is the header of wget (this is what servers see):

GET / HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: localhost:80
Connection: Keep-Alive

And here is the header of firefox:

GET / HTTP/1.1
Host: localhost:80
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-TW,en-US;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

As you can see, they are similar.

  • "They see it as a browser requesting a file just like other browsers." - I understand that you didn't mean that, but sounds like you wanted to say that server don't recognize Wget, jcoppens said it more clearly: "wget identifies as such, and more than a few site take that as a bad thing". – R S Sep 5 '16 at 14:19
  • The server actually doesn't care what program your using to get it's content, it only cares the requests you send and what should it response. It's even possible to use net cat to get web contents while the server think you are an IE. – Rio6 Sep 5 '16 at 22:22
0

Maybe one of the best arguments is that wget is capable of retrying if a connection is unreliable, and can even instruct the server to 'reget', to continue download from the point where the previous download failed - which is very welcome when downloading large files. Some browser plugins are capable of this too.

Also, very useful, is that wget can download entire directories (including subdirectories), and does so avoiding repetitions in the download.

About anonymity - that is very relative. Normally, wget identifies as such, and more than a few site take that as a bad thing - some block wget.

But, it is possible to make wget identify as any other string, using the --user-agent=agent-string command line option. (Again, similar tricks are possible with some browser plugins)

So, basically, this is a question of taste... A good argument is that wget can be used easily in scripts, which is more complicated with browsers.

0

There are a lot of features in a graphical browser(Firefox, Chrome, etc...), they will be used to identify the user. Not just HTTP headers. They are called "browser fingerprint".

  • Installed plugins
  • Current screen resolution
  • Installed fonts
  • Supported features

and many others... (reference: "Am I unique?", "Panopticlick", "BrowserLeaks.com" )

CLI browser(wget, curl, etc...) does not have a lot of these "fingerprint", because these are very simple. This increases the anonymity, make it difficult to the tracking of the user.

Not the answer you're looking for? Browse other questions tagged or ask your own question.