1

My file (tcp dump from networked sensors) has this shape:

20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68)
    e108-193.eduroam.tugraz.ac.at.34225 > e158-093.eduroam.tugraz.ac.at.personal-agent: [udp sum ok] UDP, length 40
    0x0000:  4500 0044 68f7 4000 4011 25c7 8083 d0bf  E..Dh.@.@.%.....
    0x0010:  8083 da24 85b1 15b3 0030 1e8b 3132 3033  ...$.....0..1203
    0x0020:  3132 2e37 3836 3036 2c20 332c 2020 2030  12.78606,.3,...0
    0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
    0x0040:  2e39 3630                                .960
20:06:57.113591 IP (tos 0x0, ttl 64, id 26872, offset 0, flags [DF], proto UDP (17), length 68)
    e108-193.eduroam.tugraz.ac.at.34225 > e158-093.eduroam.tugraz.ac.at.personal-agent: [udp sum ok] UDP, length 40
    0x0000:  4500 0044 68f8 4000 4011 25c6 8083 d0bf  E..Dh.@.@.%.....
    0x0010:  8083 da24 85b1 15b3 0030 148f 3132 3033  ...$.....0..1203
    0x0020:  3132 2e38 3336 3131 2c20 332c 2020 2d30  12.83611,.3,..-0
    0x0030:  2e31 3533 2c20 2d30 2e34 3630 2c20 2039  .153,.-0.460,..9
    0x0040:  2e39 3630                                .960
20:06:57.188105 IP (tos 0x0, ttl 64, id 26873, offset 0, flags [DF], proto UDP (17), length 68)
    e108-193.eduroam.tugraz.ac.at.34225 > e158-093.eduroam.tugraz.ac.at.personal-agent: [udp sum ok] UDP, length 40
    0x0000:  4500 0044 68f9 4000 4011 25c5 8083 d0bf  E..Dh.@.@.%.....
    0x0010:  8083 da24 85b1 15b3 0030 1b80 3132 3033  ...$.....0..1203
    0x0020:  3132 2e38 3836 3135 2c20 332c 2020 2d30  12.88615,.3,..-0
    0x0030:  2e31 3533 2c20 2d30 2e33 3036 2c20 2039  .153,.-0.306,..9
    0x0040:  2e38 3037                                .807                                 
20:06:57.200719 IP (tos 0x0, ttl 64, id 26874, offset 0, flags [DF], proto UDP (17), length 68)
    e108-193.eduroam.tugraz.ac.at.34225 > e158-093.eduroam.tugraz.ac.at.personal-agent: [udp sum ok] UDP, length 40
    0x0000:  4500 0044 68fa 4000 4011 25c4 8083 d0bf  E..Dh.@.@.%.....
    0x0010:  8083 da24 85b1 15b3 0030 2884 3132 3033  ...$.....0(.1203
    0x0020:  3132 2e39 3336 3135 2c20 332c 2020 2030  12.93615,.3,...0
    0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
    0x0040:  2e38 3037                                .807

and I need to convert it to this or similar shape:

20:06:57.049686  26871, 120312.78606,.0.153,.-0.153,..9.960
20:06:57.113591  26872, 120312.83611,-0.153,.-0.460,..9.960
20:06:57.188105  26873, 120312.88615,-0.153,.-0.306,..9.807
20:06:57.200719  26874, 120312.93615,.0.000,.-0.153,..9.807

So far a vb script was doing the job just fine, but now we need this data to be displayed live, so there has to be a script that will use grep/cut/awk whatever commands to do this as the log is being updated, hence the opening the file when its completed and using the vbscript in libre-office is not good anymore.

Any suggestions?

  • I think the vbscript might actually be broken. If you look the last block there gets you a 0.000 in field 3 which breaks with the pattern of output for every other block. My suspicion is that this occurs because you reference that val as a float and attempt to assign it the data on the right which - and only in the last block - is prefaced with a ( there. – mikeserv Oct 24 '14 at 4:27
2

I rethought my approach and decided using Hold space was both easier and more robust. As you mention the file delimits on tabs rather than spaces the simple fix is to use a character class rather than the literal space character. The thing about classes though is they can tend to get a little bulky, and so I generally prefer to wrap them in a substitution when I use them. I'll also pack it up into a single simple command by wrapping it in a shell function.

Here it is again, different again:

#!/bin/sh
robot() (s=[:blank:];LC_ALL=C \
    sed "s/^[$s].*[$s]//;s/\.\.*/./g
    /[$s][^i$]*[^1-9d]*[d$s]*/!{H;\$!d
    };   s//,/;x;s///;s/\n//g
         s/,\.\([-1-9]*\)0*//
         s//,\1/g;s//,+.\1/g" "$@" 
)
robot "$@"            

If the top and bottom line is removed from the above, it can be copy/pasted into a running POSIX-compatible shell that includes a POSIX-compatible sed in $PATH. Else the above above can be written verbatim to an executable file in $PATH. Either way, I can call it just like INPUT| robot or robot <infile or robot infile1 infile[2-9] -. Like:

robot <<\DATA
20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68)
    e108-193.eduroam.tugraz.ac.at.34225 > e158-093.eduroam.tugraz.ac.at.personal-agent: [udp sum ok] UDP, length 40
    0x0000:  4500 0044 68f7 4000 4011 25c7 8083 d0bf  E..Dh.@.@.%.....
    0x0010:  8083 da24 85b1 15b3 0030 1e8b 3132 3033  ...$.....0..1203
    0x0020:  3132 2e37 3836 3036 2c20 332c 2020 2030  12.78606,.3,...0
    0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
    0x0040:  2e39 3630                                .960
DATA

Whereas the older version printed...

20:06:57.049686,26871,.0.120312.78606,.3,.0.153,-0.153,.9.960

This one prints...

20:06:57.049686,26871,120312.78606,+.153,-.153,9.960

It now strips leading .dots or 0zeroes from the various fields and removes the spurious .3 field it got before that was not included in your output. It also adds + signs to even the output for fields that might lead with a -dash.

Maybe it does too much - I don't honestly know how all of those fields should come together and this only amounts to my best guess. I try to break it down below, but if you'd like to see it do something different for any reason don't hesitate to ask.

The character class [:blank:] matches any horizontal whitespace character as defined by locale - and so it can be a space or a tab and it will make no difference. And thinking about the locale I decided to explicitly specify the C locale here as it is always the most sensible thing to do when doing text processing - in the C locale each input byte is guaranteed to match a single character (which can mean a whole lot to sed when it does stuff like .*). Any state defined in the function is local to the function only, though, and will not affect the execution environment except to print to stdout.

FLOW:

In the first line of the script sed will remove a sequence of characters from the first through last occurring [[:blank:]] if the line begins with one. It then also replaces any series of .dots with a single .dot.

The next line is a little hairy. It is a single address fashioned to handle multiply staged s///ubstitutions using an empty address to refer to the last non-empty in the script.

/[$s][^i$]*[^d1-9]*[d$s]*/

The only definite match in the address is at least one blank - all other sequences within match zero* or more occurrences - and so it cannot match a line which contains no blanks at all. This means that it only matches block headings because we just removed all of the blanks from any line beginning with one.

So the first thing I do is compare it against non-header lines and place them in Hold space if they do !not match it and $!delete all of those that are !not the $last from output. And so the $last line here falls through to be lumped in with all block headings for the remainder of the script, because, from here on, those are the only lines to which sed will apply any rules.

Consider that address compared to a header line:

20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68)
  • [$s] - matches the first blank - just before IP.
  • [^i$]* - matches up the 64,[[:blank:]].
  • [^d1-9]* - matches thei`.
  • [d$s]* - matches the d and the [[:blank:]]

So when I do s//,/ all of that is replaced with a comma. This doesn't affect the last line which we just let fall through because it cannot match the leading [[:blank:]]. And even if it did, any change would be irrelevant because, for the last line, the only copy ever printed is that waiting for us in Hold space which we only operate on following an exchange. Which is the next command - and applies universally - and so we're now working with a pattern space that looks something like...

20:06:57.049686,26871, offset 0, flags [DF], proto UDP (17), length 6\
8)\n40\nE.Dh.@.@.%.\n.$.0.1203\n12.78606,.3,.0\n.153,.-0.153,.9\n.960$

...which is representative of the edits already applied to the previous block heading and all lines which have been Held since - as you can see delimited with \newline escapes there.

Let's compare that again to our long address:

  • [$s] - matches the first [[:blank:]] occurring immediately before offset.
  • [^i$]* - matches all the way up to but not including the first occurring $ character.
  • [^d1-9]* - matches .0..
  • [d$s]* - matches nothing at all.

So, as before, s/// wipes all of that out. After also removing all \newlines like s/\n//g and now pattern space looks like:

20:06:57.049686,26871,120312.78606,.3,.0.153,.-0.153,.9.960

...which is pretty much where the old version left it. But I was having fun with the recursive matches and so I added a little. You should not though, that, while I probably could make it better as I find holes in it this last bit is a thing probably better done in a second filter with a fresh script targeting the csv's. Still, it definitely works with the example input and probably will with more, but I thought I just might disclaim it anyway in case it doesn't or in case it does but does not conform to your expectiations. The last two lines can easily be removed to obtain output in line with that which you accepted when you checked this answer - as shown above.

s/,\.\([-1-9]*\)0*//

That removes the first occurring sequence of a ,comma followed by a .dot, then zero or more -dash or numbers which are not zero and any trailing zeroes. That gets the ,.3 field removed from output.

Applied a second time with a global flag and saving the \1 from being removed like s//,\1/ and it transforms:

20:06:57.049686,26871,120312.78606,.0.153,.-0.153,.9.960

...into...

20:06:57.049686,26871,120312.78606,.153,-.153,9.960

One more time applied globally with an added +. like s//,+./g and it further modifies pattern space to read:

20:06:57.049686,26871,120312.78606,+.153,-.153,9.960

...which is what is then auto-printed to stdout before sed begins the next line cycle.

if any reader has actually made it this far and is curious about what I mean by the old version, you'll find it and a detailed explanation of how it works in the edit history

Feeding this script your entire example input gets:

20:06:57.049686,26871,120312.78606,+.153,-.153,9.960
20:06:57.113591,26872,120312.83611,-.153,-.460,9.960
20:06:57.188105,26873,120312.88615,-.153,-.306,9.807
20:06:57.200719,26874,120312.93615,+.153,-.153,9.807
  • it works for one line, but if instead of the data in the command I feed the file I get different things the reason is that that I have tabs instead of spaces e.g. '-TAB-0x0000:' – A robot from Mars Oct 24 '14 at 20:44
  • @ArobotfromMars - Yeah - that would explain a lot. To make it a little more robust, you can also employ the shell. I like to do it in a function. I'll do an update in a moment to show you. – mikeserv Oct 24 '14 at 20:53
  • @ArobotfromMars - ok. I put up a modified version that will handle whitespace of any kind gracefully. – mikeserv Oct 24 '14 at 23:45
  • ok, I am trying robot() (s=[:blank:];LC_ALL=C \ sed "s/^[$s].*[$s]//;s/\.\.*/./g /[$s][^i$]*[^1-9d]*[d$s]*/!{H;\$!d }; s//,/;x;s///;s/\n//g s/,\.([-1-9]*)0*// s//,\1/g;s//,+.\1/g" "$@" ) <<\DATA (the data there) DATA echo $DATA -----and it comes out totally blank. I went through your logic and it seems sound. But I am trying to understand why it deletes it all! Maybe I will try to debug it somehow. – A robot from Mars Oct 28 '14 at 14:48
  • @ArobotfromMars - because of echo "$data" - what is that? This doesn't set any variables. You need to ensure that you format the heredocs properly - they must be delimited with newlines. But you don't have to use heredocs - you can just feed it your file. – mikeserv Oct 28 '14 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.