Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line from the first file into a new file. I had come up with a solution where i read the file using 'cat $filename | while read LINE'. but it's taking around 8 mins to complete. is there any other way like using 'awk' to reduce this process time.

my code

cat $filename | while read LINE
do
  #read 115 to 125 and then remove trailing spaces and leading zeroes
  vid=`echo "$LINE" | cut -c 115-125 | sed 's,^ *,,; s, *$,,' | sed 's/^[0]*//'`
  exist=0
  #match vid with entire line in id.txt
  exist=`grep -x "$vid" $file_dir/id.txt | wc -l`
  if [[ $exist -gt 0 ]]; then
    echo "$LINE" >> $dest_dir/id.txt
  fi
done
share|improve this question
    
Just so we're clear, you want to take cols 115-125 of file A, and if those 11 characters match any full line in file B, print the full file A line? –  Kevin Apr 24 '13 at 15:37
    
Yes thatz right, also I want to remove the trailing spaces and leading zero's from the cols 115-125 which is taken from file A and compare it with full line in file B. –  user37774 Apr 24 '13 at 15:52
add comment

1 Answer 1

up vote 3 down vote accepted

The following should work, updated to strip white space:

#!/usr/bin/awk -f
# NR is the current line number (doesn't reset between files)
# FNR is the line number within the current file
# So NR == FNR  takes only the first file
NR == FNR {
    # Mark the current line as existing, via an associative array.
    found[$0]=1

    # Skip to the next line, so we don't go through the next block
    next
}
{
    # Take the columns we're looking for
    cols = substr($0,115,11)

    # Strip whitespace (space and tab) from the beginning (^) and end ($) 
    gsub(/^[ \t]+/,"", cols)
    gsub(/[ \t]+$/,"", cols)

    # Check the associative array to see if this was in the first file
    # If so, print the full line
    if(found[cols]) print;
}       

Put it in a file and call with one of the following

awk -f script.awk patterns.txt full.txt
./script.awk patterns.txt full.txt
share|improve this answer
    
Thank you somuch Kevin, it worked perfectly –  user37774 Apr 24 '13 at 16:38
    
Your script took just 3 seconds to complete where mine taken over 17 minutes. Thank you somuch Kevin. –  user37774 Apr 24 '13 at 16:48
    
Kevin, Can you please explain the code and its iteration especially if(found[cols]) print; and NR == FNR { found[$0]=1 next } –  user37774 Apr 24 '13 at 17:07
1  
Thank you for detailed explanation. –  user37774 Apr 24 '13 at 17:30
1  
When you do found[cols] you're actually allocating and setting (to nothing) found[cols]. Doing found[$0] (without =1), and then test with if(cols in found) is a lot more efficient (memory and cpu-wise). –  Stéphane Chazelas Apr 24 '13 at 18:31
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.