I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line from the first file into a new file. I had come up with a solution where i read the file using 'cat $filename | while read LINE'. but it's taking around 8 mins to complete. is there any other way like using 'awk' to reduce this process time.

my code

cat $filename | while read LINE
  #read 115 to 125 and then remove trailing spaces and leading zeroes
  vid=`echo "$LINE" | cut -c 115-125 | sed 's,^ *,,; s, *$,,' | sed 's/^[0]*//'`
  #match vid with entire line in id.txt
  exist=`grep -x "$vid" $file_dir/id.txt | wc -l`
  if [[ $exist -gt 0 ]]; then
    echo "$LINE" >> $dest_dir/id.txt
  • Just so we're clear, you want to take cols 115-125 of file A, and if those 11 characters match any full line in file B, print the full file A line?
    – Kevin
    Commented Apr 24, 2013 at 15:37
  • Yes thatz right, also I want to remove the trailing spaces and leading zero's from the cols 115-125 which is taken from file A and compare it with full line in file B.
    – user37774
    Commented Apr 24, 2013 at 15:52

1 Answer 1


The following should work, updated to strip white space:

#!/usr/bin/awk -f
# NR is the current line number (doesn't reset between files)
# FNR is the line number within the current file
# So NR == FNR  takes only the first file
NR == FNR {
    # Mark the current line as existing, via an associative array.

    # Skip to the next line, so we don't go through the next block
    # Take the columns we're looking for
    cols = substr($0,115,11)

    # Strip whitespace (space and tab) from the beginning (^) and end ($) 
    gsub(/^[ \t]+/,"", cols)
    gsub(/[ \t]+$/,"", cols)

    # Check the associative array to see if this was in the first file
    # If so, print the full line
    if(found[cols]) print;

Put it in a file and call with one of the following

awk -f script.awk patterns.txt full.txt
./script.awk patterns.txt full.txt
  • Your script took just 3 seconds to complete where mine taken over 17 minutes. Thank you somuch Kevin.
    – user37774
    Commented Apr 24, 2013 at 16:48
  • Kevin, Can you please explain the code and its iteration especially if(found[cols]) print; and NR == FNR { found[$0]=1 next }
    – user37774
    Commented Apr 24, 2013 at 17:07
  • @user37774 Sure, I've added explanation to the code.
    – Kevin
    Commented Apr 24, 2013 at 17:17
  • 2
    When you do found[cols] you're actually allocating and setting (to nothing) found[cols]. Doing found[$0] (without =1), and then test with if(cols in found) is a lot more efficient (memory and cpu-wise). Commented Apr 24, 2013 at 18:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .