I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line from the first file into a new file. I had come up with a solution where i read the file using 'cat $filename | while read LINE'. but it's taking around 8 mins to complete. is there any other way like using 'awk' to reduce this process time.

my code

cat $filename | while read LINE
  #read 115 to 125 and then remove trailing spaces and leading zeroes
  vid=`echo "$LINE" | cut -c 115-125 | sed 's,^ *,,; s, *$,,' | sed 's/^[0]*//'`
  #match vid with entire line in id.txt
  exist=`grep -x "$vid" $file_dir/id.txt | wc -l`
  if [[ $exist -gt 0 ]]; then
    echo "$LINE" >> $dest_dir/id.txt
  • Just so we're clear, you want to take cols 115-125 of file A, and if those 11 characters match any full line in file B, print the full file A line? – Kevin Apr 24 '13 at 15:37
  • Yes thatz right, also I want to remove the trailing spaces and leading zero's from the cols 115-125 which is taken from file A and compare it with full line in file B. – user37774 Apr 24 '13 at 15:52

The following should work, updated to strip white space:

#!/usr/bin/awk -f
# NR is the current line number (doesn't reset between files)
# FNR is the line number within the current file
# So NR == FNR  takes only the first file
NR == FNR {
    # Mark the current line as existing, via an associative array.

    # Skip to the next line, so we don't go through the next block
    # Take the columns we're looking for
    cols = substr($0,115,11)

    # Strip whitespace (space and tab) from the beginning (^) and end ($) 
    gsub(/^[ \t]+/,"", cols)
    gsub(/[ \t]+$/,"", cols)

    # Check the associative array to see if this was in the first file
    # If so, print the full line
    if(found[cols]) print;

Put it in a file and call with one of the following

awk -f script.awk patterns.txt full.txt
./script.awk patterns.txt full.txt
  • Your script took just 3 seconds to complete where mine taken over 17 minutes. Thank you somuch Kevin. – user37774 Apr 24 '13 at 16:48
  • Kevin, Can you please explain the code and its iteration especially if(found[cols]) print; and NR == FNR { found[$0]=1 next } – user37774 Apr 24 '13 at 17:07
  • @user37774 Sure, I've added explanation to the code. – Kevin Apr 24 '13 at 17:17
  • 2
    When you do found[cols] you're actually allocating and setting (to nothing) found[cols]. Doing found[$0] (without =1), and then test with if(cols in found) is a lot more efficient (memory and cpu-wise). – Stéphane Chazelas Apr 24 '13 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.