1

I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted file.

file 1:
  name   values
  aaa    10
  aab    acc
  aac    30
  aac    abc

file2:
  aaa
  aac
  aac
  aad

since the file contains 400,000 lines it takes time to process.

My current solution is like this

#!/bin/ksh
while read line
do
var=`echo $line `
grep "$var" file1 >> /dev/null
if [ $? -eq 0 ]
then
grep "$var" file1 >> present
else
echo " $line missing " > missing
fi
done < "file2"

Since I am using grep here, the value may be present some where in the file1 other than the intended column1, I don't want that to happen.

My expected solution:

  1. compare the second file only with the column 1 of first file (even if we do this way it takes long time).
  2. Using a perl script with file pointer compare two columns of the files. If the string matches print it. Else if the column 1 of first file is greater than that of second file increment the file 2 AND COMPARE. If it is VICE VERSA increment the column 1 of file 1 and compare.
5
  • It is not clear to me. Do you want to see of any of the lines of file2 occur anywhere in file1, or do you want to compare them row-wise.
    – Bernhard
    Commented Jun 3, 2014 at 10:04
  • The simplified version of your script would be grep -f file2 file1
    – Bernhard
    Commented Jun 3, 2014 at 10:11
  • Are both files ~400,000 lines, or just file1? What is the approximate number of unique elements in file2? Are they really strings like aac,xyz (i.e. a maximum of ~26^3 unique strings)? Commented Jun 3, 2014 at 12:24
  • @bernhard i want to see any of lines of file2 occur in column1 of file1.if it so then i need to write it to a file .
    – user68365
    Commented Jun 3, 2014 at 14:29
  • Both are 400000 lines . @steeldriver
    – user68365
    Commented Jun 3, 2014 at 14:30

3 Answers 3

1
join file1 file2

by default it will use column 1 for each file, and omit lines missing on either on them, which is what you want. Also, the files need to be sorted, which is already the case.

2
  • Oh, nice one! But don't both lists need to be sorted for this?
    – arnefm
    Commented Jun 3, 2014 at 10:33
  • i am using solaris 10 and join doest work .
    – user68365
    Commented Jun 3, 2014 at 11:06
0

If the number of unique elements in file2 is not too large, then a feasible solution might be the classic approach of processing both files with awk, first creating an array of the unique elements in column 1 of file2, and then testing column 1 of file1 for membership in the array i.e.

awk 'FNR==NR {a[$1]++}; FNR!=NR && a[$1]' file2 file1

An equivalent approach using bash 4+ associative arrays might be something like

#!/bin/bash

declare -A a

while read col1 _ ; do
  ((a[$col1]++))
done < file2

while IFS= read -r line; do
  # compare only with 1st column of second file
  read -r col1 _ <<< "$line"
  [[ -n "${a[$col1]}" ]] && printf "$line\n"
done < file1
4
  • The awk gives out an error like this awk: syntax error near line 1 awk: bailing out near line 1
    – user68365
    Commented Jun 3, 2014 at 14:56
  • I am using Ksh here .
    – user68365
    Commented Jun 3, 2014 at 14:56
  • Sorry I am not familiar with the capabilities of your platform's version of awk - in case it does not support the FNR extension, you could try testing the file name instead i.e. awk 'FILENAME=="file2" {a[$1]++}; FILENAME=="file1" && a[$1]' file2 file1 Commented Jun 3, 2014 at 15:13
  • i am using solaris 10 ,even after trying the second method same persists
    – user68365
    Commented Jun 3, 2014 at 15:49
0

Is this anything like what you are looking for? I use cut to split the list into arrays, each containing one column. This assumes the columns are delimited by a tab character. You can change the delimiter cut uses by specifying the -d option. To split at underscore: cut -d '_'.

    #!/bin/bash

    FILE1='somefile'
    FILE2='someotherfile'

    # File 1, column 1
    f1c1=($(cut -f1 -s $FILE1))
    # File 1, column 2
    #f1c2=($(cut -f2 -s $FILE1))

    # File 2, column 1
    f2c1=($(cut -f1 -s $FILE2))
    # File 2, column 2
    #f2c2=($(cut -f2 -s $FILE2))

    # Looping through all items in file 1 column 1
    for x in "${f1c1[@]}"
    do
        # For each item in f1c1, check all items in f2c1 for a match
        for y in "${f2c1[@]}"
        do
            if [[ $x == $y ]]
            then
                # The items matched!
                echo $x
                # Breaking out of the loop (no need to check for more than one
                # match, right?)
                break
            fi
        done
    done
3
  • I think a double loop will be extreeeeeemely slow for larger files.
    – Bernhard
    Commented Jun 3, 2014 at 10:02
  • I have no idea, never tried it on a large file :) No one had answered so I gave it a shot.
    – arnefm
    Commented Jun 3, 2014 at 10:22
  • @arnefm , I need to print all the matches .
    – user68365
    Commented Jun 3, 2014 at 11:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .