Why does this 'while' loop not recognize the last line?

Question

We use the following script:

more test.sh
#!/bin/bash

while read -r line
do

echo $line

done < /tmp/file

This is the file:

kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,10240
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300000
ams-env,metrics_collector_heapsize,512
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,512
fefolp,hbase_master_xmn_size,192
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024
blolp-env,fefolp_heapsize,4096

Remark - after the last line - there are no space!

But the script prints only these lines (except the last line):

./test.sh
kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,140
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300
ams-env,metrics_collector_heapsize,51
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,542
fefolp,hbase_master_xmn_size,19
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024

Why does this happen?

Answer 1

Your input text contains an incomplete line as its last line. The last line is not terminated by a newline.

while IFS= read -r line || [ -n "$line" ]; do
    printf '%s\n' "$line"
done <file

The above loop will read unmodified lines¹ (without stripping whitespaces or interpreting backslashed control sequences) from the file called file and print them to standard output.

When an incomplete line is read, read will fail, but $line will still contain data. The extra -n test will detect this so that the loop body is allowed to output the incomplete line. In the iteration after that, read will fail again and $line will be an empty string, thus terminating the loop.

---

^{¹ assuming they don't contain NUL characters in shells other than zsh and assuming they don't contain sequences of bytes not forming part of valid characters in the yash shell, both of which shouldn't happen if the input is valid text, though that missing line delimiter on the last line already makes it invalid text.}

Answer 2

Input to the while read loop must be done like this :

cat /tmp/file | { cat ; echo ; } | while read -r line; do echo $line;  done

This will process all lines, including the last one.

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/

Answer 3

Following Your input text contains an incomplete line as its last line. The last line is not terminated by a newline. Leave the loop as is and simply make sure there is a return for the variable line.

while read -r i
do
#
line="$(echo "$i"| sed 's/$/@/'| tr '\n' '@'|sed 's/@@/@/' |tr '@' '\n')"
#
echo $line
#
done < /tmp/file

You can change the @ to anything you like.