Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

For debugging purposes I want to monitor the http requests on a network interface.

Using a naive tcpdump command line I get too much low-level information and the information I need is not very clearly represented.

Dumping the traffic via tcpdump to a file and then using wireshark has the disadvantage that it is not on-the-fly.

I imagine a tool usage like this:

$ monitorhttp -ieth0 --only-get --just-urls
2011-01-23 20:00:01 GET http://foo.example.org/blah.js
2011-01-23 20:03:01 GET http://foo.example.org/bar.html
...

I am using Linux.

share|improve this question
    
There is same question answered on superuser.com/questions/67428/possible-to-catch-urls-in-linux –  AlexD Jan 23 '11 at 9:32
add comment

3 Answers

up vote 14 down vote accepted

Try tcpflow:

tcpflow -p -c -i eth0 port 80 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*'

Output is like this:

GET /search?q=stack+exchange&btnI=I%27m+Feeling+Lucky HTTP/1.1
Host: www.google.com

You can obviously add additional HTTP methods to the grep statement, and use sed to combine the two lines into a full URL.

share|improve this answer
    
An advantage of tcpflow is that it is already available in the default repositories in Ubuntu 10.04 (justsniffer, httpry are not). The package info states that IP fragments are not recorded properly - don't know, if this matters for this use case - perhaps justsniffer can handle them better. –  maxschlepzig Jan 22 '11 at 23:11
    
Since you're just grabbing the URL it doesn't seem like it'll matter. Tcpflow will display packets in the order they were received on the interface. Thus, if you were trying to capture file contents you can get packets that arrive out of order and will produce a corrupt file. But your use case listed in the question I think this will work for you. You can also widen your grep (or remove the -o) to see more of the packet data for sorting or whatnot later. –  bahamat Jan 23 '11 at 0:01
add comment

You can use httpry or Justniffer to do that.

httpry is available e.g. via the Fedora package repository.

Example call:

# httpry -i em1

(where em1 denotes an network interface name)

Example output:

2013-09-30 21:35:20    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/6281/editor-heartbeat/edit    HTTP/1.1
2013-09-30 21:35:20    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:35:49    192.168.0.1     198.252.206.16    >    POST    unix.stackexchange.com    /posts/validate-body                 HTTP/1.1
2013-09-30 21:35:49    198.252.206.16  192.168.0.1       < HTTP/1.1   200    OK
2013-09-30 21:33:33    192.168.0.1      92.197.129.26    >    GET     cdn4.spiegel.de    /images/image-551203-breitwandaufmacher-fgoe.jpg    HTTP/1.1

(output is a little bit shortened)

share|improve this answer
    
+1 for httpry. Hadn't heard about it. –  0xC0000022L Mar 11 '11 at 14:11
add comment

I think Wireshark is capable of doing what you want

On the plus side, it's very powerful, you can install it via apt-get, and it comes with a GUI.

However, the filter system is complicated - but there are good tutorials built in, and it will give you a live or start/stop overview of the traffic.

Typing the word 'http' into the filter will probably give you what you are looking for (i.e. the main traffic generated by users).

share|improve this answer
    
Would like to know why this was downvoted. Wireshark can read the interface on the fly and filter to just http traffic. –  Kevin M Jan 22 '11 at 17:10
    
@Kevin M, Well, I did not downvote your answer. But to be fair your answer is a bit incomplete and off-topic. 1) It misses details on how exactly wireshark should be used, i.e. that a filter should be used, the exact filter expression, etc. 2) it does not allow for command line usage like sketched in the question - even if I am ok with the GUI approach, the default view displays GET requests, where the domain name is not displayed side by side - with is not that convenient for the sketched use case. –  maxschlepzig Jan 22 '11 at 18:18
    
I mean: s/your answer/Phobia's answer/ –  maxschlepzig Jan 22 '11 at 18:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.