5

I want to read a large text data file, line by line. Sample code:

lines=0
while IFS=$' \t\n'
do
    lines=$(($lines+1))
    read val
    echo "lines=$lines val=$val"
done < myfile

Problem: the loop reads past the end of the file !

$ wc -l myfile
41788 myfile

It goes fine at the beginning, I can see the data printed, but the loop does not stop after 41788. The counter goes on running with empty lines and I have to CTRL-C'it to stop.

Checked the data file, nothing special. size=5088370 bytes, the fields are tab separated (1).

$ file myfile 
myfile: ISO-8859 text, with CRLF line terminators

Any clue of what is going on here ? Did I miss something ?

(1) The fields are actually read with read -a val, but I tried the simplified code above to reduce my MCWE.

11

You're saying as long as IFS was set to $' \t\n' successfully, your loop will run. It should be something like this instead:

while IFS=$' \t\n' read -r val
do
    ((lines++))
    echo "lines=$lines val=$val"
done < myfile

The default value of IFS should be $' \t\n' so setting it to that may be redundant. I also believe that read by default will read whole lines, and I think even setting IFS to $' ' will still cause it to read the whole line.

If you want read to delimit by spaces you can use the -d switch like this:

read -r -d' ' val
do
    ((lines++))
    echo "lines=$lines val=$val"
done < myfile

This will loop through every space delimited string in myfile.

  • Wow ! Was so simple... Thanks, had that in front of me, couldn't see it, as it did correctly read the lines! Bash is hard. – kebs Dec 23 '17 at 15:34
  • This is a typical SO syndrome: probably copy/pasted that snippet from somewhere without thinking about it. – kebs Dec 23 '17 at 15:35
  • 1
    yep, Bash's read has -d to set the "line" delimiter, standard read just reads (logical) lines, where I assume the "logical" part comes into play if you backslash-escape a newline. – ilkkachu Dec 23 '17 at 15:58
  • 4
    Uhhhm, : $((++lines)) or : $((lines++)) or lines=$((lines+1)) please. – jthill Dec 23 '17 at 20:50
  • 3
    ((lines++)) (without the $) would also work, and is perfectly idiomatic in bash. – Gordon Davisson Dec 24 '17 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.