I am using awk on Ubuntu to corelate references between two log files (In.log and Out.log). It is working as expected, but went the size of the Out.log increase (~K lines), the processing time increase also ! So I was wondering, to reduce disk access, if I can load this Out.log file in memory once before the loop, using the MAPFILE command or something similar, but I do not see useful examples. Any suggestions?


2016-10-20 19:20:00,IN,DEU00000000159560100
2016-10-22 19:22:20,IN,DEU00000000159560511
2016-10-24 19:24:20,IN,DEU00000000159560382
2016-10-26 19:26:20,IN,DEU00000000159560609
2016-10-28 19:28:20,IN,DEU00000000159560809


2016-10-20 19:20:30,OUT,DEU00000000159560100
2016-10-21 19:21:30,OUT,DEU00000000159560510
2016-10-22 19:22:30,OUT,DEU00000000159560511
2016-10-23 19:23:30,OUT,DEU00000000159560381
2016-10-24 19:24:30,OUT,DEU00000000159560382
2016-10-25 19:25:30,OUT,DEU00000000159560195
2016-10-26 19:26:30,OUT,DEU00000000159560609
2016-10-27 19:27:30,OUT,DEU00000000159560433
2016-10-28 19:28:30,OUT,DEU00000000159560809
2016-10-29 19:29:30,OUT,DEU00000000159560694

Current script:

cat IN.LOG | while read -r Line
    Reference=$(cut -f3 -d',' <<< $Line)
    TimeStampIn=$(cut -f1 -d',' <<< $Line)
    TimeStampOut=$(awk -F',' -v Ref=$Reference '$3==Ref {print $1; exit} ' OUT.LOG)
    echo "$Reference,$TimeStampIn,$TimeStampOut"

1 Answer 1


Just one awk command like this:

awk -v reflog=OUT.LOG '
      while((getline <reflog) > 0) refs[$3]=$1;
      print $3,$1,refs[$3] ? refs[$3] : "not found"
    }' IN.LOG

Or using join which is exactly made for use cases like this:

join -t, -j 3 -o 0,1.1,2.1 -a 1 -e "not found" \
  <(sort -t, -k3 IN.LOG) <(sort -t, -k3 OUT.LOG) \
  | sort -t, -k2

Note, join needs sorted input files (by "join column"). That's why we sort before both log files by reference column and afterwards we sort the output again by date.

Both soutions behave differently regarding non-unique references in OUT.LOG. awk would ignore duplicate refs in OUT.LOG (last one wins) but join would print all matches. Moreover if you add -a 2 to the join command then it would print all un-pairable lines.

  • Many thanks, i just needs to implement with the rest of the business, both solution are quite fast!
    – Fdv
    Nov 13, 2016 at 14:21
  • Hi, it works perfectly ! Performance a very good Many thx What I still try to do is to output as 4th element, the difference in second between the two timestamps.
    – Fdv
    Nov 15, 2016 at 21:41
  • @user3489729 To add a date diff column I would use another awk (mktime) command on the current output, like awk 'BEGIN{FS=OFS=",";}{a=$2; b=$3; gsub("[:-]"," ",a); gsub("[:-]"," ",b); a=mktime(a); b=mktime(b); diff= (a>0 && b>0) ? b-a : "-"; print $0,diff}'
    – rudimeier
    Nov 16, 2016 at 7:02
  • Hi, indeed, it was I read from several proposals, but mktime is not defined in AWK, I will probably need to use gawk, but this is not yet available on the server.
    – Fdv
    Nov 16, 2016 at 15:40
  • @user3489729 Maybe you should post the timediff thing as a separate question.
    – rudimeier
    Nov 16, 2016 at 17:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.