Say for example I have a text file called hello.txt that contains "Hello". If I have a while loop for reading it:

while IFS= read -r line
    echo $line > hello.txt # whatever that inserts/ edits something in the text file
done < hello.txt

Will the while loop read the changes/ cause an infinite loop? or will the while loop not read the changes made to the text file? Is it possible to force it to read the updated/modified text file?

  • 3
    What happens if you try it? :)
    – ilkkachu
    Nov 4 '20 at 17:14
  • Another interesting exercise would be to change > into >>.
    – Kusalananda
    Nov 4 '20 at 17:26

You'll have both the input to the loop and read, and the output from echo connected to the same file, yes.

In that particular case, every time the output redirection is set up, for each echo invocation individually, the file is truncated. The read position of the input will be left where it was, exactly after the latest line read. Here, you're printing the exact same line back, so the read position will be right on the new EOF, and the loop will terminate after one iteration. You'll be left with the original first line of the file as the only contents of the file.

However, if e.g. you output a longer text, the loop may read parts of the previously output contents.

Consider e.g.

$ cat hello.txt
$ cat test.sh
while IFS= read -r line
    echo "$line"  # to terminal, to see what is read
    echo "something something $line" > "$1"
done < "$1"

Now, running the script on hello.txt leaves it with this one line:

something something g something abc

After the first line, the file contains something something abc, and the read position is at the first t (because abc and a newline were read). The loop reads thing something abc and prints something something thing something abc (truncating the file). The read position is still in the middle of that, so the loop repeats until eventually it reaches the EOF.

Note that read in particular leaves the read position exactly at the end of the line. That's the same with e.g. (a standards conforming) head -n1, but many other utilities will just happily read a full block and leave the read position there, probably at the original end-of-file for a shortish input. That would change the result here.

On the other hand, if you replace the output redirection with >> file, all writes will append to the file, the read position will never reach the end (since the loop reads and writes exactly one line in each iteration), and you'll get that unbounded loop.

Still again, if we remove the output file inside the loop, before the redirection, everything changes:

while IFS= read -r line
    rm "$1"
    echo "$line"  # to terminal, to see what is read
    echo "something something $line" > "$1"
done < "$1"

Now, the output redirection can't access the same file, as it no longer has a name. It'll create a new file instead. The read file handle is still connected to the file: it's only finally deleted when it's no longer held open. Running that on the original hello.txt leaves a new hello.txt with:

something something ghi

The loop reads the input to the end since the output file is independent of the input now. It just gets removed and recreated on every iteration.

Removing the file on each loop iteration isn't that useful, but to do it only once, we'd need to move redirections a bit. This would run through the whole file, and leave a new file with the same name, with all lines prefixed with something something :

exec < "$1"  # input redirection for the whole shell script
rm "$1"
exec > "$1"  # symmetrically, output, creating a new file
             # with the same name
while IFS= read -r line
    echo "$line"  >&2 # to terminal via stderr, to see what is read
    echo "something something $line"

Let's give it a try, with a bit of debugging info to look at. I'm not very familiar with kernel calls, but I hope anyone with more experience will correct any mistakes:

$ cd "$(mktemp --directory)"
$ cat > test.bash <<'EOF'
> while IFS= read -r line
> do
>     echo "$line" > hello.txt
> done < hello.txt
$ echo foo > hello.txt
$ strace bash --noprofile --norc test.bash 
[skipping the script setup for clarity, and annotating the rest]
# Open hello.txt for reading as file descriptor 3
openat(AT_FDCWD, "hello.txt", O_RDONLY) = 3
fcntl(0, F_GETFD)                       = 0
fcntl(0, F_DUPFD, 10)                   = 10
fcntl(0, F_GETFD)                       = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
# Duplicate FD 3 as FD 0
dup2(3, 0)                              = 0
# Close FD 3
close(3)                                = 0
ioctl(0, TCGETS, 0x7ffc7a087960)        = -1 ENOTTY (Inappropriate ioctl for device)
# Go to start of FD 0
lseek(0, 0, SEEK_CUR)                   = 0
# Read up to max 128 bytes from FD 0. Four bytes read, which is the full file.
# `read` strips the newline, so $line ends up containing just "foo"
read(0, "foo\n", 128)                   = 4
# Open hello.txt for writing as FD 3
openat(AT_FDCWD, "hello.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_GETFD)                       = 0
fcntl(1, F_DUPFD, 10)                   = 11
fcntl(1, F_GETFD)                       = 0
fcntl(11, F_SETFD, FD_CLOEXEC)          = 0
# Duplicate FD 3 as FD 1
dup2(3, 1)                              = 1
# Close FD 3
close(3)                                = 0
fstat(1, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
# Write what we read plus a newline to FD 1
write(1, "foo\n", 4)                    = 4
dup2(11, 1)                             = 1
fcntl(11, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(11)                               = 0
ioctl(0, TCGETS, 0x7ffc7a087960)        = -1 ENOTTY (Inappropriate ioctl for device)
# Go to current position in FD 0 (index 4, as returned by the previous `read`)
lseek(0, 0, SEEK_CUR)                   = 4
# Try to read again from FD 0. Encounters end of file (return value 0).
read(0, "", 128)                        = 0
# Shut down
dup2(10, 0)                             = 0
fcntl(10, F_GETFD)                      = 0x1 (flags FD_CLOEXEC)
close(10)                               = 0
read(255, "", 129)                      = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(0)                           = ?

Basically any read which encounters EOF will result in the while loop terminating. If you replace echo "$line" > hello.txt with echo "$line" >> hello.txt in the script it will run until storage fills up, because every time read runs there will be more content before the end of the file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.