7

We use the following script:

more test.sh
#!/bin/bash

while read -r line
do

echo $line

done < /tmp/file

This is the file:

kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,10240
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300000
ams-env,metrics_collector_heapsize,512
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,512
fefolp,hbase_master_xmn_size,192
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024
blolp-env,fefolp_heapsize,4096

Remark - after the last line - there are no space!

But the script prints only these lines (except the last line):

./test.sh
kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,140
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300
ams-env,metrics_collector_heapsize,51
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,542
fefolp,hbase_master_xmn_size,19
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024

Why does this happen?

5
  • because the last text line is not complete. you need a \n there. Nov 18 '18 at 11:50
  • do you have suggestion how to add \n on the last line in the file , so I will put thus in the scritp
    – yael
    Nov 18 '18 at 12:02
  • echo >> file as a poor mans solution. Nov 18 '18 at 12:36
  • Note POSIX defines "line" as a sequence of zero or more non- <newline> characters plus a terminating <newline> character. Your last line isn't even a line in this context. Nov 18 '18 at 16:24
  • Does this answer your question? Read a line-oriented file which may not end with a newline
    – AdminBee
    Sep 27 '21 at 10:16
14

Your input text contains an incomplete line as its last line. The last line is not terminated by a newline.

while IFS= read -r line || [ -n "$line" ]; do
    printf '%s\n' "$line"
done <file

The above loop will read unmodified lines¹ (without stripping whitespaces or interpreting backslashed control sequences) from the file called file and print them to standard output.

When an incomplete line is read, read will fail, but $line will still contain data. The extra -n test will detect this so that the loop body is allowed to output the incomplete line. In the iteration after that, read will fail again and $line will be an empty string, thus terminating the loop.


¹ assuming they don't contain NUL characters in shells other than zsh and assuming they don't contain sequences of bytes not forming part of valid characters in the yash shell, both of which shouldn't happen if the input is valid text, though that missing line delimiter on the last line already makes it invalid text.

7
  • the script is only example of the problem , actually I set the echo in the script to show this issue , but what we need is how to set the last line with "\n" with sed for example, since the the - file is already exists
    – yael
    Nov 18 '18 at 12:26
  • so if need to add "\n on the last line , then need sed/perl to add the "\" for this solution
    – yael
    Nov 18 '18 at 12:30
  • @yael You say in the question that you use a loop like the one you are showing. I gave a solution for how to fix that loop. It does not matter what the body of the loop looks like, $line will be each individual line of the file, including the last incomplete one. If you don't use a loop like this, what do you use?
    – Kusalananda
    Nov 18 '18 at 12:37
  • ok , so maybe I will post another question for how to add "\n" on the last line
    – yael
    Nov 18 '18 at 12:38
  • 1
    @yael How to add a newline to the end of a file?
    – Kusalananda
    Nov 18 '18 at 12:40
0

Input to the while read loop must be done like this :

cat /tmp/file | { cat ; echo ; } | while read -r line; do echo $line;  done

This will process all lines, including the last one.

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/

3
  • Just do { cat /tmp/file; echo; } if you're going to do that: unix.stackexchange.com/a/418081/70524
    – muru
    Oct 12 '21 at 3:25
  • We could use any outputting program instead of cat /tmp/file hence the given structure / syntax.
    – Nathan SR
    Oct 12 '21 at 3:53
  • Yes, you could use any program: { foo ; echo; } | ...
    – muru
    Oct 12 '21 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.