3

I'm trying to read two files line-by-line in Bash and do something to each of their lines. Here is my Bash script:

#!/usr/bin/env bash

die()
{
    echo "$@" >&2
    exit 1
}

extract_char()
{
    echo "$1" | sed "s/.*'\([^']*\)'.*/\1/g"
}

file1=$1 # old
file2=$2 # new
counter=0

win_count=0
lose_count=0

test ! -z "$file1" || die "Please enter 2 files."
test ! -z "$file2" || die "Please enter 2 files."

while read -r line1 && read -r line2 <&3
do
    let counter++
    index=$(expr index "$line1" "'")
    if [ $index -ne 0 ]; then
        char=$(extract_char "$line1")
        char2=$(extract_char "$line2")
        test "$char" = "$char2" || die "Chars in line1 and line2 were not the same."
    elif [ "${line1#char.}" != "$line1" ]; then
        test "${line2#char.}" != "$line2" || die "Method signature found in line1, but not line2."
        method=${line1%:}
        method=${method#char.}
    elif ! grep -q '[^[:space:]]'; then
        # benchmark times
        if [ $(date --date="$line1" +%s%N) -gt $(date --date="$line2" +%s%N) ]; then
            echo "$char $method $counter: $line1 is greater than $line2"
            let lose_count++
        else
            let win_count++
        fi
    fi
done < "$file1" 3< "$file2"

echo
echo "Lines where this made an improvement: $win_count"
echo "Lines where this made a regression: $lose_count"

Its usage is like this:

./compare.sh oldresults.txt newresults.txt

Where oldresults.txt and newresults.txt are two files containing benchmark results. Here's an example file:

Test results for '\u0020':

char.IsUpper:
00:00:00.1231231
00:00:00:4564564

char.IsLower:
00:00:00:3453455
00:11:22:4444444

Tests for '\u1234':

# and so on

For some reason, it seems that read is returning a nonzero exit status before it's finished reading the file. Here is the output when I debug my script (via bash --debug -x compare.sh [args]):

+ file1=oldresults.txt
+ file2=newresults.txt
+ counter=0
+ win_count=0
+ lose_count=0
+ test '!' -z oldresults.txt
+ test '!' -z newresults.txt
+ read -r line1
+ read -r line2
+ let counter++
++ expr index 'Test results for '\''\u0020'\'':
' \'
+ index=18
+ '[' 18 -ne 0 ']'
++ extract_char 'Test results for '\''\u0020'\'':
'
++ echo 'Test results for '\''\u0020'\'':
'
++ sed 's/.*'\''\([^'\'']*\)'\''.*/\1/g'
+ char='\u0020'
++ extract_char 'Test results for '\''\u0020'\'':
'
++ echo 'Test results for '\''\u0020'\'':
'
++ sed 's/.*'\''\([^'\'']*\)'\''.*/\1/g'
+ char2='\u0020'
+ test '\u0020' = '\u0020'
+ read -r line1
+ read -r line2
+ let counter++
++ expr index $'\r' \'
+ index=0
+ '[' 0 -ne 0 ']'
+ '[' $'\r' '!=' $'\r' ']'
+ grep -q '[^[:space:]]'
+ read -r line1 # exits the loop here
+ echo

+ echo 'Lines where this made an improvement: 0'
Lines where this made an improvement: 0
+ echo 'Lines where this made a regression: 0'
Lines where this made a regression: 0

As you can see, the script iterates over two lines: first the "Test results for..." line where it extracts \u0020 from in between the quotes, and then a carriage return. After that, read -r line1 mysteriously seems to fail and it exits the loop.

Why does this happen, and what can I do to fix it? Thanks.

  • 1
    Use one more file descriptor other than 1 for for read then the problem will go away. – cuonglm May 15 '16 at 2:56
  • and what is the grep supposed to read? – Law29 May 15 '16 at 5:55
3

What is happening is that grep -q '[^[:space:]]' is processing the remaining lines in standard input (which is what grep does by default if you haven't given it any input), leaving nothing for the next read - the file pointer is at EOF. What you'll want is grep -q '[^[:space:]]' <<< "$line1".

A simple way to avoid this kind of error is to always use a non-default file descriptor if your loop code is non-trivial. There are lots of ways to end up swallowing all of stdin in a single command, but I've yet to encounter any programs which will try to read FD 3 and higher by default.

  • Aha, thank you; I hadn't seem my simple mistake. Thanks for the tip as well. – James Ko May 15 '16 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.