0

This question already has an answer here:

sudo tshark -i ppp0 'tcp port 80   \
and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'   \
-R'http.request.method == "GET" && http.request.uri contains "/ABC/XYZ"' \
-T fields -e http.host -e http.request.uri

I am using the above lines of tshark filtering the output based on /ABC/XYZ. it outputs dynamically random data for example like where i find duplicate lines example :

1bcdJOSHhijklmnopqrstuvwxyz
1bcdefghijklmnopqrstuvwxyz
1bcdefghijklmnopqrstuvwxyz
3bcdefghijklmnopqrstuvwxyz
2bcdefghijklmnopqrstuvwxyz
1bcdJOSHhijklmnopqrstuvwxyz
3bcdefghijklmnopqrstuvwxyz

Is there any way to remove the duplicates without creating any file may be using some kind of buffer or pipe.

or can tshark itself do it

EDIT : I am not sure why but i get what i want after i prefix tshark with stdbuf -o L as suggested by meuh

marked as duplicate by Pandya, Jakuje, jimmij, garethTheRed, Archemar Apr 2 '16 at 10:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • @Pandya i am not sure why it does not seem to work – munish Apr 2 '16 at 9:44
  • it doesn't work.I believe it is not exaclty duplicate.Please dont delete – munish Apr 2 '16 at 9:59
  • for that, question should clearly say about facing difficulties while processing (piping) on tshark output. This will not be deleted though – Pandya Apr 2 '16 at 10:21
3

If you have a stop condition on your tshark you can simply pipe the output into |sort -u. Alternatively, pipe continuous output into

awk '{if(!seen[$0]++)print}'

You may need to have tshark not buffer its stdout: try prefixing the tshark with stdbuf -o L.

  • do you know why stdbuf -o L helps here – munish Apr 2 '16 at 10:14
  • 1
    Some programs use the standard library to print data and the library detects if the output is to the terminal or to a file or pipe. If output is to a terminal, each print is output immediately, but if to a file the output is optimised and only written when there is a full buffer. stdbuf is a utility to make the standard library never buffer output. – meuh Apr 2 '16 at 10:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.