sudo tshark -i ppp0 'tcp port 80   \
and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'   \
-R'http.request.method == "GET" && http.request.uri contains "/ABC/XYZ"' \
-T fields -e http.host -e http.request.uri

I am using the above lines of tshark filtering the output based on /ABC/XYZ. it outputs dynamically random data for example like where i find duplicate lines example :


Is there any way to remove the duplicates without creating any file may be using some kind of buffer or pipe.

or can tshark itself do it

EDIT : I am not sure why but i get what i want after i prefix tshark with stdbuf -o L as suggested by meuh

  • @Pandya i am not sure why it does not seem to work – munish Apr 2 '16 at 9:44
  • it doesn't work.I believe it is not exaclty duplicate.Please dont delete – munish Apr 2 '16 at 9:59
  • for that, question should clearly say about facing difficulties while processing (piping) on tshark output. This will not be deleted though – Pandya Apr 2 '16 at 10:21

If you have a stop condition on your tshark you can simply pipe the output into |sort -u. Alternatively, pipe continuous output into

awk '{if(!seen[$0]++)print}'

You may need to have tshark not buffer its stdout: try prefixing the tshark with stdbuf -o L.

  • do you know why stdbuf -o L helps here – munish Apr 2 '16 at 10:14
  • 1
    Some programs use the standard library to print data and the library detects if the output is to the terminal or to a file or pipe. If output is to a terminal, each print is output immediately, but if to a file the output is optimised and only written when there is a full buffer. stdbuf is a utility to make the standard library never buffer output. – meuh Apr 2 '16 at 10:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.