I have 90 gig of data culled from 13.5 Terabytes.

I have tried sort -u | uniq on data that has been awk'd from the 13.5T of syslog data.

Some malformed data was apparent so I reran the parse with awk and 'seen' like so:

 awk -F, '!seen[$1]++' inputfile > outputfile

This turned out to be the most time efficient means but also included some malformed data... maybe there are malformed log entries or in sorting uniq'ing and awk'ing some lines got munged. I do not care if there is a more/better way of parsing the original data, since I have a large enough sample size - meaning losing a little data out of 13.5T is OK.

There are 3 IP addresses per valid line.

Since there are 3 periods in an IP address, I need something that will parse out only lines that have 9 "."'s.

  • 2
    Do you care how valid the IP's appear? In other words, are you OK selecting a line like: foo.bar.baz 300.300.300.300 ? – Jeff Schaller Nov 29 '17 at 20:06

Let's take this as a test file:

$ cat testfile  Keep 9.10.11     Bad: Missing 1 Bad: Extra period

Using grep

To select lines with exactly nine periods:

$ grep -E '^([^.]*\.){9}[^.]*$' testfile  Keep

[^.]*\. matches any number of non-period characters followed by a ([^.]*\.){9} matches exactly nine sequences of zero or more non-period characters followed by a period. The ^ at the beginning requires that the regex match starting at the beginning of the line. The [^.]*$ means that, between the end of the nine sequences and the end of the line, only non-period characters are allowed.

Using sed

$ sed -En '/^([^.]*\.){9}[^.]*$/p' testfile  Keep

The -n option tells sed not to print unless we explicitly ask it to. The p following the regex explicitly asks sed to print those lines which match the regex.

Using awk

$ awk '/^([^.]*\.){9}[^.]*$/' testfile  Keep

Or, using awk's ability to define a character to separate fields (hat tip: Jeff Schaller):

$ awk -F. 'NF==10' testfile  Keep
|improve this answer|||||
  • how about awk -F. NF==10 ? – Jeff Schaller Nov 29 '17 at 20:01
  • @JeffSchaller Yes, excellent. (I had too look up in the docs to verify that, when FS is not treated as a regex when it is a single character.) – John1024 Nov 29 '17 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.