4

I'm working with a logfile with the following format:

Oct 12 01:28:26 server program: 192.168.1.105 text for 1.105 
Oct 12 01:30:00 server program: 192.168.1.104 text for 1.104 
Oct 12 01:30:23 server program: 192.168.1.103 text for 1.103
Oct 12 01:32:39 server program: 192.168.1.101 text for 1.101 
Oct 12 02:28:26 server program: 192.168.1.105 text for 1.105 
Oct 12 02:30:00 server program: 192.168.1.104 text for 1.104
Oct 12 02:30:23 server program: 192.168.1.103 text for 1.103 
Oct 12 02:32:39 server program: 192.168.1.101 text for 1.101

I need to achieve this:

Oct 12 02:28:26 server program: 192.168.1.105 text for 1.105 
Oct 12 02:30:00 server program: 192.168.1.104 text for 1.104
Oct 12 02:30:23 server program: 192.168.1.103 text for 1.103
Oct 12 02:32:39 server program: 192.168.1.101 text for 1.101

How can I send the new output to a file? I have tried this:

awk '!_[$6]++ {a=$6} END{print a}' logfile

But it does not give me the results expected. How can I use awk or sed to give me only the unique lines with last time the string match was seen or based on date/time?

Improve this question
5
  • Can you edit your example to include multiple different programs and multiple different messages for the same program on the same server, so I can see in your expected output how you want to handle those? Or is there only one program per server? (You can make up server and program names—use fruits or whatever—but when the example data is all monotonous it makes it hard to understand your actual use case.)
    – Wildcard
    Nov 3, 2016 at 22:52
  • both server and program will always be the same. The messages after the IP will only be different if the IP is different.
    – user53029
    Nov 3, 2016 at 22:56
  • I made some updates.
    – user53029
    Nov 3, 2016 at 23:00
  • @don_crissti, simple as it is, that is a very good Answer. ;)
    – Wildcard
    Nov 4, 2016 at 0:04
  • @don_crissti, "it can be done without any sorting", yes, but not with my approach (since uniq requires input to be sorted and I'm ignoring the timestamp), and AFAIK Awk arrays do not guarantee sequence.
    – Wildcard
    Nov 4, 2016 at 0:14

4 Answers 4

Reset to default
5

If you're going to do a second pass (which you pretty well have to), you may as well only store line numbers rather than full records. It makes the logic easier.

awk 'NR == FNR {if (z[$6]) y[z[$6]]; z[$6] = FNR; next} !(FNR in y)' logfile logfile

Proof of correctness:

At the end of processing each line, every line number processed so far is either a value in z, or an index (not value) in y, but never both.

The lines represented by values in z are, at the end of each iteration, exactly and only the latest records so far seen for each IP address.

The indices of y are, therefore, the exact lines which we wish not to print.

Improve this answer
1
  • 1
    Similar concept in one pass: awk 'p[$6]{delete k[p[$6]]}{p[$6]=NR;k[NR]=$0} END{for(i=1;i<=NR;i++)if(i in k)print k[i]}' logfile . (k=keep; p=posn_to_delete) With GNU awk can use END{PROC_INFO["sorted_in"]="@ind_num_asc";for(i in k)print k[i]}
    – dave_thompson_085
    Nov 4, 2016 at 12:59
3

Save the entire line (using $6 as array index) and in END iterate over the elements of the array: 

awk '{z[$6]=$0};END{for (i in z) print z[i]}' logfile

The result won't be sorted though... You could do something like:

awk '{z[$6]=NR" "$0};END{for (i in z) print z[i]}' logfile | sort -k1,1n | cut -f2-
### this space ^ is a literal TAB

which saves the line no. plus the line content to be able to then sort by line no.

Other ways involve a 2nd pass to get it sorted by date (since this is a log) but will print duplicate entries if the input contains duplicate lines (entire lines, that is) - e.g. with grep:

awk '{z[$6]=$0};END{for (var in z) print z[var]}' logfile | grep -Fxf- logfile

or only with awk:

awk 'NR==FNR{z[$6]=$0;next}
FNR==1{for (var in z) y[z[var]]}
$0 in y' logfile logfile
Improve this answer
4
  • 1
    Nice idea, using Awk for two passes. :) If you're going to do that, though, you don't even need to store lines. I decided to post the alternate approach as an answer rather than a comment since the explanation was a bit long.
    – Wildcard
    Nov 4, 2016 at 2:32
  • Don, I noticed that for some reason your command is leaving some duplicates. If you run your command and re-direct to a new file the do something like "grep program | awk '{print $6}' | sort | uniq -c | sort -nr | less" you will hopefully see what I'm talking about.
    – user53029
    Nov 4, 2016 at 16:08
  • record-keeping purposes. You can still see the anomaly when not re-directing its just easier that way for me to verify that there were no duplicates. I did not mean to imply that a re-direct was a possible cause of the results I was seeing.
    – user53029
    Nov 4, 2016 at 16:32
  • 1
    @user53029 - you do realise I don't have access to your data so I can't replicate whatever happens on your side hence I cannot see any anomaly (I don't see how the solutions above could print duplicate entries anyway - it's impossible "by design" - unless you have duplicate lines like in, entire lines) ?
    – don_crissti
    Nov 4, 2016 at 16:38
2

If you have lines only from the same day you can handle this like so:

sort -k6 -k3r logfile | uniq -f3 | sort -k3

If you have lines for more than one day you can still use this basic approach, but your sorting will have to get a lot fancier. The above command can only handle one day's records because it uses the time portion of the timestamp (e.g. 02:28:26) as a proxy for the entire timestamp.

Improve this answer
2

Logic becomes simpler by reversing the file linewise

$ tac logfile | awk '!seen[$6]++' | tac
Oct 12 02:28:26 server program: 192.168.1.105 text for 1.105 
Oct 12 02:30:00 server program: 192.168.1.104 text for 1.104
Oct 12 02:30:23 server program: 192.168.1.103 text for 1.103 
Oct 12 02:32:39 server program: 192.168.1.101 text for 1.101
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .