How to loop over the lines of a file?

Question

Say I have this file:

hello
world
hello world

This program

#!/bin/bash

for i in $(cat $1); do
    echo "tester: $i"
done

outputs

tester: hello
tester: world
tester: hello
tester: world

I'd like to have the for iterate over each line individually ignoring whitespaces though, i.e. the last two lines should be replaced by

tester: hello world

Using quotes for i in "$(cat $1)"; results in i being assigned the whole file at once. What should I change?

Answer 1

With for and IFS:

#!/bin/bash

IFS=$'\n'       # make newlines the only separator
set -f          # disable globbing
for i in $(cat < "$1"); do
  echo "tester: $i"
done

Note however that it will skip empty lines as newline being an IFS-white-space character, sequences of it count as 1 and the leading and trailing ones are ignored. With zsh and ksh93 (not bash), you can change it to IFS=$'\n\n' for newline not to be treated specially, however note that all trailing newline characters (so that includes trailing empty lines) will always be removed by the command substitution.

Or with read (no more cat):

#!/bin/bash

while IFS= read -r line; do
  echo "tester: $line"
done < "$1"

There, empty lines are preserved, but note that it would skip the last line if it was not properly delimited by a newline character.

Answer 2

(9 years later:)
Both provided answers would fail on files without a newline at the end, this will effectively skip the last line, produce no errors, would lead to disaster (learned hard way:).

The best concise solution I found so far that "Just Works" (in both bash and sh):

while IFS='' read -r LINE || [ -n "${LINE}" ]; do
    echo "processing line: ${LINE}"
done < /path/to/input/file.txt

For more in-depth discussion see this StackOverflow discussion: How to use "while read" (Bash) to read the last line in a file if there’s no newline at the end of the file?

Beware: this approach adds an additional newline to the last line if there is none already.

Answer 3

If you can avoid it, don't especially if it's to process text.

Most text utilities are already designed to process text one line at a time, and, at least for the GNU implementations, do it efficiently, correctly and handle error conditions nicely. Piping one to another which runs them in parallel also means you can leverage more than one processor to do the job.

Here:

<input.txt sed 's/^/tester /' > output.txt

Or:

<input.txt awk '{print "tester", $0}' > output.txt

More on that at: Why is using a shell loop to process text considered bad practice?

If it's not about text processing and you do need to run some command per line of a file, also note GNU xargs where you can do:

xargs -rd'\n' -I@ -a input.txt cp -- @ @.back

for instance.

With the bash shell, you can get each line of a file into an array with the readarray builtin:

readarray -t lines < input.txt &&
  for line in "${lines[@]}"; do
    do-some-non-text-processing-you-cannot-easily-do-with-xargs "$line" || break
  done

POSIXly, you can use IFS= read -r line to read one line off some input, but beware that if you redirect the whole while read loop with the input file on stdin, then commands inside the loop will also have their stdin redirected to the file, so best is to use a different fd which you close inside the loop:

while
  IFS= read -r line <&3 ||
    [ -n "$line" ] # to cover for an unterminated last line.
do
  {
    do-some-non-text-processing-you-cannot-easily-do-with-xargs "$line" ||
      break # abort upon failure if relevant
  } 3<&-
done 3< input.txt > output.txt

read -r line removes leading and trailing whitespace characters from the line that it reads provided they are in the $IFS variable, though only the yash shell honours that POSIX requirement. With most shells, that's limited to space and tab. ksh93 and recent versions of bash do it for all single-byte characters considered as whitespace in the locale.

So to read a line and also strip leading and trailing blanks, you can do: IFS=$' \t' read -r line. With ksh93, yash¹ or recent versions of bash. IFS=$' \t\r' would also strip the trailing CR character found in text files from the Microsoft world.

---

^{¹ though yash doesn't support the $'...' syntax yet, you'd need IFS=$(printf ' \t\r') there.}

Answer 4

For what it is worth, I need to do that quite often, and can never remember the exact way of using while IFS= read..., so I defined the following function in my bash profile:

# iterate the line of a file and call input function
iterlines() {
    (( $# < 2 )) && { echo "Usage: iterlines <File> <Callback>"; return; }
    local File=$1
    local Func=$2
    n=$(cat "$File" | wc -l)
    for (( i=1; i<=n; i++ )); do
        "$Func" "$(sed "${i}q;d" "$File")"
    done
}

This function first determines the number of lines in the file, then uses sed to extract line after line, and passes each line as a single string argument to any given function. I suppose this might get really inefficient with large files, but that hasn't been a problem for me so far (suggestions on how to improve this welcome of course).

The usage is pretty sweet IMO:

>> cat example.txt # note the use of spaces, whitespace, etc.
a/path

This is a sentence.
"wi\th quotes"
$End
>> iterlines example.txt echo # preserves quotes, $ and whitespace
a/path

This is a sentence.
"wi\th quotes"
$End
>> x() { echo "$#"; }; iterlines example.txt x # line always passed as single input string
1
1 
1
1
1

Answer 5

To read all the lines, regardless of whether they are ended with a new line or not:

cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/

Answer 6

There is an answer here that may be useful to people who this doesn't work for

loop through file by row in tcsh