How to log all domains accessed?

Question

Maybe I could use tcpdump, filter out all non-HTTP packets to reduce disk consumption, and then parse the output looking for domain names. Maybe there is a better solution. Do you know one?

Answer 1

Since you only want to get the domains accessed via http you could setup up a transparent http-proxy with Polipo or Privoxy and evaluate the log file.

1. Set up the proxy
   Install and configure the proxy, that he listen for example on the address 127.0.0.1:8080 and enable logging.
2. Set up the firewall rule
   Write a firewall rule, which redirect all outgoing traffic to the port 80 to the address of proxy 127.0.0.1:8080, which then forwards the traffic to requested domain.
3. Parse the log file
   Set up a parser to extract all the accessed domains from the log file.

This solution has it's up and downs:

• UP's:
  • your users don't have to change something on their system, browser, ...
  • you gain some capabilities, like caching, filtering, ... depending on the used proxy
  • you can expand it to log other protocols like https, ftp ... when the proxy supports it
• DOWN's:
  • it's another system you to have manage and that's maybe failing
  • it's raises some legal/privacy issues, when you keep extensive logs on your users browsing habits