compare two files based on a column and print it

Question

I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted file.

file 1:
  name   values
  aaa    10
  aab    acc
  aac    30
  aac    abc

file2:
  aaa
  aac
  aac
  aad

since the file contains 400,000 lines it takes time to process.

My current solution is like this

#!/bin/ksh
while read line
do
var=`echo $line `
grep "$var" file1 >> /dev/null
if [ $? -eq 0 ]
then
grep "$var" file1 >> present
else
echo " $line missing " > missing
fi
done < "file2"

Since I am using grep here, the value may be present some where in the file1 other than the intended column1, I don't want that to happen.

My expected solution:

1. compare the second file only with the column 1 of first file (even if we do this way it takes long time).
2. Using a perl script with file pointer compare two columns of the files. If the string matches print it. Else if the column 1 of first file is greater than that of second file increment the file 2 AND COMPARE. If it is VICE VERSA increment the column 1 of file 1 and compare.

Answer 1

join file1 file2

by default it will use column 1 for each file, and omit lines missing on either on them, which is what you want. Also, the files need to be sorted, which is already the case.

Answer 2

If the number of unique elements in file2 is not too large, then a feasible solution might be the classic approach of processing both files with awk, first creating an array of the unique elements in column 1 of file2, and then testing column 1 of file1 for membership in the array i.e.

awk 'FNR==NR {a[$1]++}; FNR!=NR && a[$1]' file2 file1

An equivalent approach using bash 4+ associative arrays might be something like

#!/bin/bash

declare -A a

while read col1 _ ; do
  ((a[$col1]++))
done < file2

while IFS= read -r line; do
  # compare only with 1st column of second file
  read -r col1 _ <<< "$line"
  [[ -n "${a[$col1]}" ]] && printf "$line\n"
done < file1

Answer 3

Is this anything like what you are looking for? I use cut to split the list into arrays, each containing one column. This assumes the columns are delimited by a tab character. You can change the delimiter cut uses by specifying the -d option. To split at underscore: cut -d '_'.

    #!/bin/bash

    FILE1='somefile'
    FILE2='someotherfile'

    # File 1, column 1
    f1c1=($(cut -f1 -s $FILE1))
    # File 1, column 2
    #f1c2=($(cut -f2 -s $FILE1))

    # File 2, column 1
    f2c1=($(cut -f1 -s $FILE2))
    # File 2, column 2
    #f2c2=($(cut -f2 -s $FILE2))

    # Looping through all items in file 1 column 1
    for x in "${f1c1[@]}"
    do
        # For each item in f1c1, check all items in f2c1 for a match
        for y in "${f2c1[@]}"
        do
            if [[ $x == $y ]]
            then
                # The items matched!
                echo $x
                # Breaking out of the loop (no need to check for more than one
                # match, right?)
                break
            fi
        done
    done