How to use AWK to query a file loaded with mapfile [closed]

Question

I am using awk on Ubuntu to corelate references between two log files (In.log and Out.log). It is working as expected, but went the size of the Out.log increase (~K lines), the processing time increase also ! So I was wondering, to reduce disk access, if I can load this Out.log file in memory once before the loop, using the MAPFILE command or something similar, but I do not see useful examples. Any suggestions?

IN.LOG:

2016-10-20 19:20:00,IN,DEU00000000159560100
2016-10-22 19:22:20,IN,DEU00000000159560511
2016-10-24 19:24:20,IN,DEU00000000159560382
2016-10-26 19:26:20,IN,DEU00000000159560609
2016-10-28 19:28:20,IN,DEU00000000159560809

OUT.LOG:

2016-10-20 19:20:30,OUT,DEU00000000159560100
2016-10-21 19:21:30,OUT,DEU00000000159560510
2016-10-22 19:22:30,OUT,DEU00000000159560511
2016-10-23 19:23:30,OUT,DEU00000000159560381
2016-10-24 19:24:30,OUT,DEU00000000159560382
2016-10-25 19:25:30,OUT,DEU00000000159560195
2016-10-26 19:26:30,OUT,DEU00000000159560609
2016-10-27 19:27:30,OUT,DEU00000000159560433
2016-10-28 19:28:30,OUT,DEU00000000159560809
2016-10-29 19:29:30,OUT,DEU00000000159560694

Current script:

cat IN.LOG | while read -r Line
do
    Reference=$(cut -f3 -d',' <<< $Line)
    TimeStampIn=$(cut -f1 -d',' <<< $Line)
    TimeStampOut=$(awk -F',' -v Ref=$Reference '$3==Ref {print $1; exit} ' OUT.LOG)
    echo "$Reference,$TimeStampIn,$TimeStampOut"
done

Answer 1

Just one awk command like this:

awk -v reflog=OUT.LOG '
    BEGIN{
      FS=OFS=",";
      while((getline <reflog) > 0) refs[$3]=$1;
      close(reflog)
    }{
      print $3,$1,refs[$3] ? refs[$3] : "not found"
    }' IN.LOG

Or using join which is exactly made for use cases like this:

join -t, -j 3 -o 0,1.1,2.1 -a 1 -e "not found" \
  <(sort -t, -k3 IN.LOG) <(sort -t, -k3 OUT.LOG) \
  | sort -t, -k2

Note, join needs sorted input files (by "join column"). That's why we sort before both log files by reference column and afterwards we sort the output again by date.

Both soutions behave differently regarding non-unique references in OUT.LOG. awk would ignore duplicate refs in OUT.LOG (last one wins) but join would print all matches. Moreover if you add -a 2 to the join command then it would print all un-pairable lines.