115

For debugging purposes I want to monitor the http requests on a network interface.

Using a naive tcpdump command line I get too much low-level information and the information I need is not very clearly represented.

Dumping the traffic via tcpdump to a file and then using wireshark has the disadvantage that it is not on-the-fly.

I imagine a tool usage like this:

$ monitorhttp -ieth0 --only-get --just-urls
2011-01-23 20:00:01 GET http://foo.example.org/blah.js
2011-01-23 20:03:01 GET http://foo.example.org/bar.html
...

I am using Linux.

1

6 Answers 6

Reset to default
151

Try tcpflow:

tcpflow -p -c -i eth0 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'

Output is like this:

GET /search?q=stack+exchange&btnI=I%27m+Feeling+Lucky HTTP/1.1
Host: www.google.com

You can obviously add additional HTTP methods to the grep statement, and use sed to combine the two lines into a full URL.

5
  • An advantage of tcpflow is that it is already available in the default repositories in Ubuntu 10.04 (justsniffer, httpry are not). The package info states that IP fragments are not recorded properly - don't know, if this matters for this use case - perhaps justsniffer can handle them better.
    – maxschlepzig
    Jan 22, 2011 at 23:11
  • Since you're just grabbing the URL it doesn't seem like it'll matter. Tcpflow will display packets in the order they were received on the interface. Thus, if you were trying to capture file contents you can get packets that arrive out of order and will produce a corrupt file. But your use case listed in the question I think this will work for you. You can also widen your grep (or remove the -o) to see more of the packet data for sorting or whatnot later.
    – bahamat
    Jan 23, 2011 at 0:01
  • 1
    @bahamat Can "tcpflow" work with https URL?
    – Maulik patel
    Oct 23, 2018 at 9:39
  • Increasingly, the answer is no. In the past SSL was weak enough that if you had access to the key used for the flow you could decrypt any traffic used with that key. Today, most sites will use perfect-forward-secrecy and negotiate ephemeral keys. The best option today is a so-called "bump in the wire" transparent proxy.
    – bahamat
    Oct 28, 2018 at 17:02
  • 4
    got nothing by, while browsing,using wifi: sudo tcpflow -p -c -i wlo2 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'
    – ses
    Jan 17, 2019 at 0:48
28

You can use httpry or Justniffer to do that.

httpry is available e.g. via the Fedora package repository.

Example call:

# httpry -i em1

(where em1 denotes an network interface name)

Example output:

2013-09-30 21:35:20    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/6281/editor-heartbeat/edit    HTTP/1.1
2013-09-30 21:35:20    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:35:49    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/validate-body                 HTTP/1.1
2013-09-30 21:35:49    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:33:33    192.168.0.1      92.197.129.26    >    GET     cdn4.spiegel.de    /images/image-551203-breitwandaufmacher-fgoe.jpg    HTTP/1.1

(output is a little bit shortened)

2
  • How can I show the header or the body of the request or response?
    – Mohammed Noureldin
    Jan 7, 2018 at 10:42
  • 1
    got nothing sudo httpry -i wlo2 (where wlo2 is by wifi device name)
    – ses
    Jan 17, 2019 at 0:50
15

I was looking for something similar, with the added requirement that it should work for https too.

pcap based tools like tcpflow httpry urlsnarf and other tcpdump kung fu work well for http, but for secure requests you're out of luck.

I came up with urldump, which is a small wrapper around mitmproxy.
iptables is used to redirect traffic to the proxy, so it works transparently. 

$ sudo urldump   
http://docs.mitmproxy.org/en/stable/certinstall.html
http://docs.mitmproxy.org/en/stable/_static/js/modernizr.min.js
https://media.readthedocs.org/css/sphinx_rtd_theme.css
https://media.readthedocs.org/css/readthedocs-doc-embed.css
https://media.readthedocs.org/javascript/readthedocs-doc-embed.js
...

See README for more info.

3

I think Wireshark is capable of doing what you want

On the plus side, it's very powerful, you can install it via apt-get, and it comes with a GUI.

However, the filter system is complicated - but there are good tutorials built in, and it will give you a live or start/stop overview of the traffic.

Typing the word 'http' into the filter will probably give you what you are looking for (i.e. the main traffic generated by users).

3
  • 1
    Would like to know why this was downvoted. Wireshark can read the interface on the fly and filter to just http traffic.
    – Kevin M
    Jan 22, 2011 at 17:10
  • 1
    @Kevin M, Well, I did not downvote your answer. But to be fair your answer is a bit incomplete and off-topic. 1) It misses details on how exactly wireshark should be used, i.e. that a filter should be used, the exact filter expression, etc. 2) it does not allow for command line usage like sketched in the question - even if I am ok with the GUI approach, the default view displays GET requests, where the domain name is not displayed side by side - with is not that convenient for the sketched use case.
    – maxschlepzig
    Jan 22, 2011 at 18:18
  • I mean: s/your answer/Phobia's answer/
    – maxschlepzig
    Jan 22, 2011 at 18:27
2

Another good option might be nethogs

On fedora is available among the core packages, and on centos you can get it through the epel repo.

1
  • How do you use nethogs to dump which URLs are being called?
    – Joril
    Jun 13 at 9:30
2

There is also the command line program urlsnarf which is part of the dsniff package (which is also packaged with e.g. Fedora 19).

Example:

# urlsnarf -i em1
urlsnarf: listening on em1 [tcp port 80 or port 8080 or port 3128]
jhost - - [29/May/2014:10:25:09 +0200] "GET http://unix.stackexchange.com/questions HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/ HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/layout/css/style-V5-2-2.css HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/layout/jscfg/http/global-V5-2-2.js HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/layout/js/http/javascript-V5-2-2.js HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/layout/js/http/interface-V5-2-2.js HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/layout/js/http/netmind-V5-2-2.js HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "GET http://www.spiegel.de/favicon.ico HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "POST http://ocsp.thawte.com/ HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0"
jhost - - [29/May/2014:10:25:36 +0200] "POST http://ocsp.thawte.com/ HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0
[..]

(when browsing first to SE and then to spiegel.de)

Limitations: dsnarf does not support IPv6. I can reproduce this bug report with 0.17 on Fedora 19. Also seems to be broken under Ubuntu trusty atm (works fine under lucid).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .