Bash: Why is read returning a nonzero exit status while reading my file?

Question

I'm trying to read two files line-by-line in Bash and do something to each of their lines. Here is my Bash script:

#!/usr/bin/env bash

die()
{
    echo "$@" >&2
    exit 1
}

extract_char()
{
    echo "$1" | sed "s/.*'\([^']*\)'.*/\1/g"
}

file1=$1 # old
file2=$2 # new
counter=0

win_count=0
lose_count=0

test ! -z "$file1" || die "Please enter 2 files."
test ! -z "$file2" || die "Please enter 2 files."

while read -r line1 && read -r line2 <&3
do
    let counter++
    index=$(expr index "$line1" "'")
    if [ $index -ne 0 ]; then
        char=$(extract_char "$line1")
        char2=$(extract_char "$line2")
        test "$char" = "$char2" || die "Chars in line1 and line2 were not the same."
    elif [ "${line1#char.}" != "$line1" ]; then
        test "${line2#char.}" != "$line2" || die "Method signature found in line1, but not line2."
        method=${line1%:}
        method=${method#char.}
    elif ! grep -q '[^[:space:]]'; then
        # benchmark times
        if [ $(date --date="$line1" +%s%N) -gt $(date --date="$line2" +%s%N) ]; then
            echo "$char $method $counter: $line1 is greater than $line2"
            let lose_count++
        else
            let win_count++
        fi
    fi
done < "$file1" 3< "$file2"

echo
echo "Lines where this made an improvement: $win_count"
echo "Lines where this made a regression: $lose_count"

Its usage is like this:

./compare.sh oldresults.txt newresults.txt

Where oldresults.txt and newresults.txt are two files containing benchmark results. Here's an example file:

Test results for '\u0020':

char.IsUpper:
00:00:00.1231231
00:00:00:4564564

char.IsLower:
00:00:00:3453455
00:11:22:4444444

Tests for '\u1234':

# and so on

For some reason, it seems that read is returning a nonzero exit status before it's finished reading the file. Here is the output when I debug my script (via bash --debug -x compare.sh [args]):

+ file1=oldresults.txt
+ file2=newresults.txt
+ counter=0
+ win_count=0
+ lose_count=0
+ test '!' -z oldresults.txt
+ test '!' -z newresults.txt
+ read -r line1
+ read -r line2
+ let counter++
++ expr index 'Test results for '\''\u0020'\'':
' \'
+ index=18
+ '[' 18 -ne 0 ']'
++ extract_char 'Test results for '\''\u0020'\'':
'
++ echo 'Test results for '\''\u0020'\'':
'
++ sed 's/.*'\''\([^'\'']*\)'\''.*/\1/g'
+ char='\u0020'
++ extract_char 'Test results for '\''\u0020'\'':
'
++ echo 'Test results for '\''\u0020'\'':
'
++ sed 's/.*'\''\([^'\'']*\)'\''.*/\1/g'
+ char2='\u0020'
+ test '\u0020' = '\u0020'
+ read -r line1
+ read -r line2
+ let counter++
++ expr index $'\r' \'
+ index=0
+ '[' 0 -ne 0 ']'
+ '[' $'\r' '!=' $'\r' ']'
+ grep -q '[^[:space:]]'
+ read -r line1 # exits the loop here
+ echo

+ echo 'Lines where this made an improvement: 0'
Lines where this made an improvement: 0
+ echo 'Lines where this made a regression: 0'
Lines where this made a regression: 0

As you can see, the script iterates over two lines: first the "Test results for..." line where it extracts \u0020 from in between the quotes, and then a carriage return. After that, read -r line1 mysteriously seems to fail and it exits the loop.

Why does this happen, and what can I do to fix it? Thanks.

Answer 1

What is happening is that grep -q '[^[:space:]]' is processing the remaining lines in standard input (which is what grep does by default if you haven't given it any input), leaving nothing for the next read - the file pointer is at EOF. What you'll want is grep -q '[^[:space:]]' <<< "$line1".

A simple way to avoid this kind of error is to always use a non-default file descriptor if your loop code is non-trivial. There are lots of ways to end up swallowing all of stdin in a single command, but I've yet to encounter any programs which will try to read FD 3 and higher by default.