I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted file.

file 1:
  name   values
  aaa    10
  aab    acc
  aac    30
  aac    abc


since the file contains 400,000 lines it takes time to process.

My current solution is like this

while read line
var=`echo $line `
grep "$var" file1 >> /dev/null
if [ $? -eq 0 ]
grep "$var" file1 >> present
echo " $line missing " > missing
done < "file2"

Since I am using grep here, the value may be present some where in the file1 other than the intended column1, I don't want that to happen.

My expected solution:

  1. compare the second file only with the column 1 of first file (even if we do this way it takes long time).
  2. Using a perl script with file pointer compare two columns of the files. If the string matches print it. Else if the column 1 of first file is greater than that of second file increment the file 2 AND COMPARE. If it is VICE VERSA increment the column 1 of file 1 and compare.
  • It is not clear to me. Do you want to see of any of the lines of file2 occur anywhere in file1, or do you want to compare them row-wise.
    – Bernhard
    Commented Jun 3, 2014 at 10:04
  • The simplified version of your script would be grep -f file2 file1
    – Bernhard
    Commented Jun 3, 2014 at 10:11
  • Are both files ~400,000 lines, or just file1? What is the approximate number of unique elements in file2? Are they really strings like aac,xyz (i.e. a maximum of ~26^3 unique strings)? Commented Jun 3, 2014 at 12:24
  • @bernhard i want to see any of lines of file2 occur in column1 of file1.if it so then i need to write it to a file .
    – user68365
    Commented Jun 3, 2014 at 14:29
  • Both are 400000 lines . @steeldriver
    – user68365
    Commented Jun 3, 2014 at 14:30

3 Answers 3

join file1 file2

by default it will use column 1 for each file, and omit lines missing on either on them, which is what you want. Also, the files need to be sorted, which is already the case.

  • Oh, nice one! But don't both lists need to be sorted for this?
    – arnefm
    Commented Jun 3, 2014 at 10:33
  • i am using solaris 10 and join doest work .
    – user68365
    Commented Jun 3, 2014 at 11:06

If the number of unique elements in file2 is not too large, then a feasible solution might be the classic approach of processing both files with awk, first creating an array of the unique elements in column 1 of file2, and then testing column 1 of file1 for membership in the array i.e.

awk 'FNR==NR {a[$1]++}; FNR!=NR && a[$1]' file2 file1

An equivalent approach using bash 4+ associative arrays might be something like


declare -A a

while read col1 _ ; do
done < file2

while IFS= read -r line; do
  # compare only with 1st column of second file
  read -r col1 _ <<< "$line"
  [[ -n "${a[$col1]}" ]] && printf "$line\n"
done < file1
  • The awk gives out an error like this awk: syntax error near line 1 awk: bailing out near line 1
    – user68365
    Commented Jun 3, 2014 at 14:56
  • I am using Ksh here .
    – user68365
    Commented Jun 3, 2014 at 14:56
  • Sorry I am not familiar with the capabilities of your platform's version of awk - in case it does not support the FNR extension, you could try testing the file name instead i.e. awk 'FILENAME=="file2" {a[$1]++}; FILENAME=="file1" && a[$1]' file2 file1 Commented Jun 3, 2014 at 15:13
  • i am using solaris 10 ,even after trying the second method same persists
    – user68365
    Commented Jun 3, 2014 at 15:49

Is this anything like what you are looking for? I use cut to split the list into arrays, each containing one column. This assumes the columns are delimited by a tab character. You can change the delimiter cut uses by specifying the -d option. To split at underscore: cut -d '_'.



    # File 1, column 1
    f1c1=($(cut -f1 -s $FILE1))
    # File 1, column 2
    #f1c2=($(cut -f2 -s $FILE1))

    # File 2, column 1
    f2c1=($(cut -f1 -s $FILE2))
    # File 2, column 2
    #f2c2=($(cut -f2 -s $FILE2))

    # Looping through all items in file 1 column 1
    for x in "${f1c1[@]}"
        # For each item in f1c1, check all items in f2c1 for a match
        for y in "${f2c1[@]}"
            if [[ $x == $y ]]
                # The items matched!
                echo $x
                # Breaking out of the loop (no need to check for more than one
                # match, right?)
  • I think a double loop will be extreeeeeemely slow for larger files.
    – Bernhard
    Commented Jun 3, 2014 at 10:02
  • I have no idea, never tried it on a large file :) No one had answered so I gave it a shot.
    – arnefm
    Commented Jun 3, 2014 at 10:22
  • @arnefm , I need to print all the matches .
    – user68365
    Commented Jun 3, 2014 at 11:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .