0

I have text file like this.

2015-11-24 12:59:37.112 128.206.6.136 source
2014-11-24 12:59:36.920 8.8.8.8 source
2014-11-24 14:59:38.112 23.234.22.106 destination
2014-11-24 13:59:37.113 23.234.22.106 source
2014-11-24 12:59:29.047 74.125.198.141 source
2014-12-25 12:59:36.920 74.125.198.148 destination

If a particular Ip address is tagged as source as well destination, then I want to tag that Ip as both. In this case, Ip 23.234.22.106 is source as well as destination. So, I want to tag it as both.

My desired output should be like this

2015-11-24 12:59:37.112 128.206.6.136 source
2014-11-24 12:59:36.920 8.8.8.8 source
2014-11-24 14:59:38.112 23.234.22.106 both
2014-11-24 12:59:29.047 74.125.198.141 source
2014-12-25 12:59:36.920 74.125.198.148 destination

This is what I have tried.

cat input.txt | awk '{print $3}' | sort | uniq | while read line

do 
grep $line input.txt | sort -r -k1 | head -1
done

But, I don't understand how to tag a particular Ip as both if it is source as well as destination. In this case, 23.234.22.106.

How can I do it using awk? Any help with this would be appreciated. Thank you

3
  • 1
    In your text file, will all destination and source entries with the same IP be next to each other?
    – Peschke
    Jun 1 '16 at 18:13
  • Yes. They are going to be next to each other Jun 1 '16 at 18:18
  • Ok, that makes things easier. The program only needs to remember the previous lines IP. I will see what I can do.
    – Peschke
    Jun 1 '16 at 18:20
5

Try with sed

sed '
    N    #add next line
    s/\([0-9.]\+\)\s\S\+\n.*\s\1\s\S\+$/\1 both/
    P    #print first line from two
    D    #remove first line, return to start
    ' input.txt
  • [0-9.]\+ group of numbers and dots
  • \s space or tab
  • \S\+ group of non-space symbols
  • \n new line
  • .* any symbols
  • \1 back reference for group in parethesis \(...\)
  • $ pattern end

(modified: remove t command (tnx 2jthill) and add \space before group to check full address)

3
  • It works. That is some hardcore use of regex. Can you explain the code please? Thank you Jun 1 '16 at 18:39
  • I have a bit similar problem like this but there is major change in input text. Can you please help me with that. Here is the link unix.stackexchange.com/questions/288227/… Thank you! Jun 7 '16 at 17:44
  • You don't even need the t, the P; D pair has the same effect on a one-single-line buffer.
    – jthill
    Jun 9 '16 at 1:08
0

With perl:

#! /usr/bin/perl

use strict;

my @lines=();

while(<>) {
  chomp;
  s/#.*//g;        # strip comments
  s/^\s*|\s*$//g;  # strip leading and trailing spaces
  next if (/^$/);  # skip blank lines

  if (! scalar @lines) {
    # store the first line of the file in the array
    # we can't do anything else yet, so skip to the next line.
    push @lines, $_;
    next;
  } else {
    push @lines, $_;

    # split both lines into space-separated fields.
    my @a = split /\s+/, $lines[0];
    my @b = split /\s+/, $lines[1];

    # if 3rd fields are the same, change to "both"
    if ($a[2] eq $b[2]) {
      @lines = map { $_ =~ s/(source|destination)$/both/oi; $_} @lines;
    }
  }
  print $lines[0],"\n";
  shift @lines;
}
print $lines[0],"\n";

The idea here is to use an array (@lines) to hold the current line and the previous line. If the 3rd field (perl-arrays are zero-based) of both lines are the same then change the strings "source" or "destination" to "both".

Print the previous line, whether they were changed or not. Then delete the previous line from the array, using shift so that on the next pass through the loop, the current line will be the previous line.

Finally, after the loop has finished, print the last input line.

Output:

$ ./swatesh.pl <swatesh.txt 
2015-11-24 12:59:37.112 128.206.6.136 source
2014-11-24 12:59:36.920 8.8.8.8 source
2014-11-24 14:59:38.112 23.234.22.106 both
2014-11-24 13:59:37.113 23.234.22.106 both
2014-11-24 12:59:29.047 74.125.198.141 source
2014-12-25 12:59:36.920 74.125.198.148 destination

Some Notes:

The sed script works well, so why might you choose to use perl instead? What are the differences?

@Costas' sed version is faster, which might be significant if you have millions of lines to process.

This perl version explicitly checks if field 3 on both lines is exactly the same, while the sed version only checks to see if a pattern that looks similar to an IP address is repeated later in the same joined double-line (which is not necessarily a problem - for your sample input, the sed version works perfectly. It's optimised for exactly your sample).

The perl version is probably easier to adapt to different input.

The code at the start of the loop is a useful bit of code I use in many perl scripts to skip blank lines and to support # comments in text files. I often do the same in sed scripts but the longer a sed script becomes, the less readable it is...and I like to write code that I can understand at a glance in 6 months time.

Aside from these relatively minor details, both scripts use a very similar algorithm.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.