Linux log file structure change, abnormal (?) sed behaviour

Question

Ok, so I tried to get this

0x0000:  4500 0044 68f7 4000 4011 25c7 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 1e8b 3132 3033  ...$.....0..1203
0x0020:  3132 2e37 3836 3036 2c20 332c 2020 2030  12.78606,.3,...0
0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
0x0040:  2e39 3630                                .960

0x0000:  4500 0044 68f8 4000 4011 25c6 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 148f 3132 3033  ...$.....0..1203
0x0020:  3132 2e38 3336 3131 2c20 332c 2020 2d30  12.83611,.3,..-0
0x0030:  2e31 3533 2c20 2d30 2e34 3630 2c20 2039  .153,.-0.460,..9
0x0040:  2e39 3630                                .960

0x0000:  4500 0044 68f9 4000 4011 25c5 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 1b80 3132 3033  ...$.....0..1203
0x0020:  3132 2e38 3836 3135 2c20 332c 2020 2d30  12.88615,.3,..-0
0x0030:  2e31 3533 2c20 2d30 2e33 3036 2c20 2039  .153,.-0.306,..9
0x0040:  2e38 3037                                .807                                 

0x0000:  4500 0044 68fa 4000 4011 25c4 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 2884 3132 3033  ...$.....0(.1203
0x0020:  3132 2e39 3336 3135 2c20 332c 2020 2030  12.93615,.3,...0
0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
0x0040:  2e38 3037                                .807

to

E..Dh.@.@.%.....
  ...$.....0..1203
  12.78606,.3,...0
.153,.-0.153,..9
.960

etc,

with this command

sed -u -e 's_0x0000:  4500 0044 68f7 4000 4011 25c7 8083 d0bf  __g;s_0x0010:  8083 da24 85b1 15b3 0030 1e8b 3132 3033__g;s_0x0030:  2e31 3533 2c20 2d30 2e34 3630 2c20 2039__g;s_0x0020:  3132 2e37 3836 3036 2c20 332c 2020 2030__g;s_0x0040:  2e39 3630__g' <tcpdump_log_sample_capture2.txt >out2

and I got

E..Dh.@.@.%.....
  ...$.....0..1203
  12.78606,.3,...0
0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
                                .960

0x0000:  4500 0044 68f8 4000 4011 25c6 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 148f 3132 3033  ...$.....0..1203
0x0020:  3132 2e38 3336 3131 2c20 332c 2020 2d30  12.83611,.3,..-0
  .153,.-0.460,..9
                                .960

0x0000:  4500 0044 68f9 4000 4011 25c5 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 1b80 3132 3033  ...$.....0..1203
0x0020:  3132 2e38 3836 3135 2c20 332c 2020 2d30  12.88615,.3,..-0
0x0030:  2e31 3533 2c20 2d30 2e33 3036 2c20 2039  .153,.-0.306,..9
0x0040:  2e38 3037                                .807                                 

0x0000:  4500 0044 68fa 4000 4011 25c4 8083 d0bf  E..Dh.@.@.%.....
0x0010:  8083 da24 85b1 15b3 0030 2884 3132 3033  ...$.....0(.1203
0x0020:  3132 2e39 3336 3135 2c20 332c 2020 2030  12.93615,.3,...0
0x0030:  2e31 3533 2c20 2d30 2e31 3533 2c20 2039  .153,.-0.153,..9
0x0040:  2e38 3037                                .807

so, in the first 5 lines, it worked in every line but the fourth starting with 0x0030:

on the second set it worked for the last two lines BUT not for the first 3, and for the 3rd and 4th set, it worked for none!

can someone have a look and tell me what is going on? it is supposed to be recursive with the g option!

(I got sed (GNU sed) 4.2.2)

What are you expecting it to do? The byte sequences are different in the successive stanzas, your expressions simply don't match them. FYI the g option relates to multiple instances in a single pattern space (line). — steeldriver, Oct 23, 2014 at 23:00
@mikeserv That won't work reliably. See my comments below your answer for an example of how it can break. — CodeGnome, Oct 24, 2014 at 6:35

CodeGnome · Accepted Answer · 2014-10-24 00:22:33Z

1

Using GNU Sed with Fixed-Width Fields

Since the format of your output appears to be formatted into fixed-width fields, you can get the data you want by simply stripping out the first 50 characters from each line. For example:

sed -r 's/^.{50}//' /tmp/corpus

answered Oct 24, 2014 at 0:22

CodeGnome

7,7301 gold badge18 silver badges25 bronze badges

It's the good code.
– PersianGulf
Oct 24, 2014 at 4:25

Add a comment |

terdon · Accepted Answer · 2014-10-24 19:14:14Z

0

Whenever you see a text file with data in columns, think awk. This is trivially done in awk:

$ awk '{print $NF}' file 
E..Dh.@.@.%.....
...$.....0..1203
12.78606,.3,...0
.153,.-0.153,..9
.960

E..Dh.@.@.%.....
...$.....0..1203
12.83611,.3,..-0
.153,.-0.460,..9
.960

E..Dh.@.@.%.....
...$.....0..1203
12.88615,.3,..-0
.153,.-0.306,..9
.807

E..Dh.@.@.%.....
...$.....0(.1203
12.93615,.3,...0
.153,.-0.153,..9
.807

The special variable NF is the number of fields of the current line. Therefore, $NF is the last field of the line. This awk scriptlet just goes through each line of the file and prints the last field.

You could do the same in perl if you prefer:

perl -lane 'print $F[$#F]' file

Or even with GNU grep:

grep -oP '.+\s\s*\K[^\s]+' file1

Or, if you really want a sed approach for some reason, use GNU sed (or any other version that accepts extended regular expressions) and:

 sed -r 's/.* +([^ ]+) *$/\1/' file

The command above looks for the longest string ending with one or more spaces (.* +), then the longest string of non-spaces ([^ ]+) then 0 or more spaces ( *) and replaces the whole thing with the captured pattern (that's what the parentheses do). Still, this will fail if the file has non-space whitespace for example. Awk is by far the best tool for this.

answered Oct 24, 2014 at 19:14

terdon♦

235k66 gold badges447 silver badges671 bronze badges

awk fails for the same reason sed does in your example - the last record must be delimited on something. here there are no spaces, so the job is as simple as s/.* //, still, even if there were, you can select fields in sed as well: s/ */\n/2 replaces the second occurring two+ space span with a \newline. i disagree in every respect about awk being the best tool, but that is mainly because the best tool is likely one you know how to use. ive yet to encounter the awk statement that i couldnt easily replicate w/ sed, but im pretty good w/ sed and so it is the best tool for me
– mikeserv
Oct 24, 2014 at 20:33
@mikeserv how does awk fail? I tested on the OP's example and it works as expected, as do the rest of my suggestions. The sed (or anything that uses a regex) is more complex because of the spaces after the last field on some lines. That's where your 1st sed approach also fails. Awk, on the other hand, deals with that kind of thing gracefully since any field delimiters (spaces here) after the last field are simply ignored. Plus, it can deal with tabs which your sed will choke on. Anyway, I never said it's the best tool, I said it's the best tool for dealing with data in columns.
– terdon ♦
Oct 24, 2014 at 20:40
I meant only in the case of spaces in the input. And no - sed never chokes - it always does exactly as it is told. Assumption free. If there were spaces in the data it would fail in the same way awk does. To handle \tabs you just tell it to - same for columns s/.//[num]. And there are no \tabs here - or trailing spaces - this is formatted input so sed couldn't fail where aren't any - not if the data has not been otherwise mangled. And if that is the case, the use of any filter is uncertain. In any case, my point was a tool is only ever as good as the hand that wields it.
– mikeserv
Oct 24, 2014 at 20:46
@mikeserv of course, my point was that the simple awk '{print $NF} will print the last field irrespective of whether you have tab or space delimited data or whether you have extra space after the last field. The almost as simple sed 's/.* // will not work if the whitespace is not a space and it will also not work if you have trailing spaces. Of course sed can deal with that, but as you say, you need to think about it while awk behaves that way by default. Which is why it's easier to use when dealing with columnar data. You still haven't told me how awk fails by the way.
– terdon ♦
Oct 24, 2014 at 20:52
It fails in the same way s/.* // might - (though here neither do). If the the last field contains spaces. sed would swallow part of the last field, awk would do the same. That's why I suggested the second two+space match - that always occurs before the last field. While others may find it convenient, it is precisely the assumptions that awk makes that has always left in wanting in my respect - I don't like uppity computers. I put mine in the closet occasionally just to remind it of that fact.
– mikeserv
Oct 24, 2014 at 20:58

| Show 2 more comments

mikeserv · Accepted Answer · 2014-10-24 20:27:17Z

Don't use sed -u - it doesn't do what you think. Any buffering sed does will be to your benefit in that it will help to increase sed's processing speed rather than otherwise. sed -u is really only useful in situations in which you want to quit input at a very specific point and ensure that sed does not consume input beyond it.

For example:

printf %s\\n line1 line2 | {
    sed -u =\;1q
    sed =
}

...which prints...

1
line1
1
line2

...but if the -u flag were not used would only print...

1
line1

...because the first sed will attempt to fill its buffer with every read() call and consume the entire |pipe before the second sed ever gets a chance to look at it. Given input of any reasonable length, sed -u will significantly reduce sed's processing speed because it will have to do a read() per line.

This, however, is not your problem.

Your problem is that you are working far too hard. Do:

sed 's/.* //' <infile >outfile

That will remove everything on a line up to and including the last occurring space. In that way you will get just the output you want.

@CodeGnome is right about this not being a reliable way to go. While it will work for the data you show, another, more robust, way might be:

sed 's/   */\n/2;s/.*\n//'

That replaces the second occurrence of two or more consecutive spaces with a \newline character and then removes everything before it. There is only one way to get a \newline character in sed pattern space, and that is by putting it there.

sed 's/.* //' won't do what you expect if there are literal spaces in the ASCII dump section. It works for the OP's posted corpus, but it's not reliable unless you know the data will never contains spaces. — CodeGnome, Oct 24, 2014 at 6:29
For example, echo 'foo bar' | xxd | sed 's/.* //' will strip out the hex but will incorrectly return bar. — CodeGnome, Oct 24, 2014 at 6:33
@CodeGnome - i get it - i use od all the time (why is it still sometimes called xxd? is it a debian thing?) but thats not this. these are tcp headers i think. — mikeserv, Oct 24, 2014 at 9:58
od is from coreutils. xxd comes from vim-common. Either way, what makes you think a tcp dump can't include space characters? Think telnet or an http stream, for example. — CodeGnome, Oct 24, 2014 at 13:53
@CodeGnome - youve got a point about the data - i was working on the other question and had it mind - it wasnt just a tcpdump - it was sensor data. degrees and whatever — mikeserv, Oct 24, 2014 at 20:18

PersianGulf · Accepted Answer · 2014-10-24 00:07:00Z

You have to do it:

paste  -d ""  <( awk {'print $10'}  yourfile )  <( cut -b 51 yourfile ) <( cut -b 52 yourfile ) <( cut -b 53 yourfile )  <( cut -b 54 yourfile )

Output is:

E..Dh.@.@.%.....E..D
...$.....0..1203...$
12.78606,.3,...012.7
.153,.-0.153,..9.153
.960

E..Dh.@.@.%.....E..D
...$.....0..1203...$
12.83611,.3,..-012.8
.153,.-0.460,..9.153
.960

E..Dh.@.@.%.....E..D
...$.....0..1203...$
12.88615,.3,..-012.8
.153,.-0.306,..9.153
.807

E..Dh.@.@.%.....E..D
...$.....0(.1203...$
12.93615,.3,...012.9
.153,.-0.153,..9.153
.807

Stack Exchange Network

Linux log file structure change, abnormal (?) sed behaviour

4 Answers 4

Using GNU Sed with Fixed-Width Fields

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
text-processing
sed
editors
.

Linked

Hot Network Questions

Linux log file structure change, abnormal (?) sed behaviour

4 Answers 4

Using GNU Sed with Fixed-Width Fields

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged text-processingsededitors.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
text-processing
sed
editors
.