Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted file.

      file 1: 
          name   values
          aaa    10
          aab    acc
          aac    30
          aac    abc 

      file2:
          aaa
          aac 
          aac
          aad 

since the file contains 400,000 lines it takes time to process.

My current solution is like this

    #!/bin/ksh
   while read line
   do
   var=`echo $line `
   grep "$var" file1 >> /dev/null
   if [ $? -eq 0 ]
   then
   grep "$var" file1 >> present
   else
   echo " $line missing " > missing 
   fi
   done < "file2"

Since I am using grep here, the value may be present some where in the file1 other than the intended column1, I don't want that to happen.

My expected solution:

  1. compare the second file only with the column 1 of first file (even if we do this way it takes long time).
  2. Using a perl script with file pointer compare two columns of the files. If the string matches print it. Else if the column 1 of first file is greater than that of second file increment the file 2 AND COMPARE. If it is VICE VERSA increment the column 1 of file 1 and compare.
share|improve this question
    
It is not clear to me. Do you want to see of any of the lines of file2 occur anywhere in file1, or do you want to compare them row-wise. –  Bernhard Jun 3 at 10:04
    
The simplified version of your script would be grep -f file2 file1 –  Bernhard Jun 3 at 10:11
    
Are both files ~400,000 lines, or just file1? What is the approximate number of unique elements in file2? Are they really strings like aac,xyz (i.e. a maximum of ~26^3 unique strings)? –  steeldriver Jun 3 at 12:24
    
@bernhard i want to see any of lines of file2 occur in column1 of file1.if it so then i need to write it to a file . –  user68365 Jun 3 at 14:29
    
Both are 400000 lines . @steeldriver –  user68365 Jun 3 at 14:30

3 Answers 3

Is this anything like what you are looking for? I use cut to split the list into arrays, each containing one column. This assumes the columns are delimited by a tab character. You can change the delimiter cut uses by specifying the -d option. To split at underscore: cut -d '_'.

    #!/bin/bash

    FILE1='somefile'
    FILE2='someotherfile'

    # File 1, column 1
    f1c1=($(cut -f1 -s $FILE1))
    # File 1, column 2
    #f1c2=($(cut -f2 -s $FILE1))

    # File 2, column 1
    f2c1=($(cut -f1 -s $FILE2))
    # File 2, column 2
    #f2c2=($(cut -f2 -s $FILE2))

    # Looping through all items in file 1 column 1
    for x in "${f1c1[@]}"
    do
        # For each item in f1c1, check all items in f2c1 for a match
        for y in "${f2c1[@]}"
        do
            if [[ $x == $y ]]
            then
                # The items matched!
                echo $x
                # Breaking out of the loop (no need to check for more than one
                # match, right?)
                break
            fi
        done
    done

Hope this helps or at least gets you closer to a solution.

share|improve this answer
    
I think a double loop will be extreeeeeemely slow for larger files. –  Bernhard Jun 3 at 10:02
    
I have no idea, never tried it on a large file :) No one had answered so I gave it a shot. –  arnefm Jun 3 at 10:22
    
@arnefm , I need to print all the matches . –  user68365 Jun 3 at 11:07
join file1 file2

by default it will use column 1 for each file, and omit lines missing on either on them, which is what you want. Also, the files need to be sorted, which is already the case.

share|improve this answer
    
Oh, nice one! But don't both lists need to be sorted for this? –  arnefm Jun 3 at 10:33
    
i am using solaris 10 and join doest work . –  user68365 Jun 3 at 11:06

If the number of unique elements in file2 is not too large, then a feasible solution might be the classic approach of processing both files with awk, first creating an array of the unique elements in column 1 of file2, and then testing column 1 of file1 for membership in the array i.e.

awk 'FNR==NR {a[$1]++}; FNR!=NR && a[$1]' file2 file1

An equivalent approach using bash 4+ associative arrays might be something like

#!/bin/bash

declare -A a

while read col1 _ ; do
  ((a[$col1]++))
done < file2

while IFS= read -r line; do
  # compare only with 1st column of second file
  read -r col1 _ <<< "$line"
  [[ -n "${a[$col1]}" ]] && printf "$line\n"
done < file1
share|improve this answer
    
The awk gives out an error like this awk: syntax error near line 1 awk: bailing out near line 1 –  user68365 Jun 3 at 14:56
    
I am using Ksh here . –  user68365 Jun 3 at 14:56
    
Sorry I am not familiar with the capabilities of your platform's version of awk - in case it does not support the FNR extension, you could try testing the file name instead i.e. awk 'FILENAME=="file2" {a[$1]++}; FILENAME=="file1" && a[$1]' file2 file1 –  steeldriver Jun 3 at 15:13
    
i am using solaris 10 ,even after trying the second method same persists –  user68365 Jun 3 at 15:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.