How to add a newline to the end of a file?

Question

Using version control systems I get annoyed at the noise when the diff says No newline at end of file.

So I was wondering: How to add a newline at the end of a file to get rid of those messages?

Answer 1

To recursively sanitize a project I use this oneliner:

git ls-files -z | while IFS= read -rd '' f; do tail -c1 < "$f" | read -r _ || echo >> "$f"; done

Explanation:

• git ls-files -z lists files in the repository. It takes an optional pattern as additional parameter which might be useful in some cases if you want to restrict the operation to certain files/directories. As an alternative, you could use find -print0 ... or similar programs to list affected files - just make sure it emits NUL-delimited entries.

• while IFS= read -rd '' f; do ... done iterates through the entries, safely handling filenames that include whitespace and/or newlines.

• tail -c1 < "$f" reads the last char from a file.

• read -r _ exits with a nonzero exit status if a trailing newline is missing.

• || echo >> "$f" appends a newline to the file if the exit status of the previous command was nonzero.

Answer 2

Here you go:

sed -i -e '$a\' file

And alternatively for OS X sed:

sed -i '' -e '$a\' file

This adds \n at the end of the file only if it doesn’t already end with a newline. So if you run it twice, it will not add another newline:

$ cd "$(mktemp -d)"
$ printf foo > test.txt
$ sed -e '$a\' test.txt > test-with-eol.txt
$ diff test*
1c1
< foo
\ No newline at end of file
---
> foo
$ echo $?
1
$ sed -e '$a\' test-with-eol.txt > test-still-with-one-eol.txt
$ diff test-with-eol.txt test-still-with-one-eol.txt
$ echo $?
0

Answer 3

Have a look:

$ echo -n foo > foo 
$ cat foo
foo$
$ echo "" >> foo
$ cat foo
foo

so echo "" >> noeol-file should do the trick. (Or did you mean to ask for identifying these files and fixing them?)

edit removed the "" from echo "" >> foo (see @yuyichao's comment) edit2 added the "" again (but see @Keith Thompson's comment)

Answer 4

Another solution using ed. This solution only affect the last line and only if \n is missing:

ed -s file <<< w

It essentially works opening the file for editing through a script, the script is the single w command, that write the file back to disk. It is based on this sentence found in ed(1) man page:

LIMITATIONS
       (...)

       If  a  text (non-binary) file is not terminated by a newline character,
       then ed appends one on reading/writing it.  In the  case  of  a  binary
       file, ed does not append a newline on reading/writing.

Answer 5

A simple, portable, POSIX-compliant way to add an absent, final newline to a would be text file:

[ -n "$(tail -c1 file)" ] && echo >> file

This approach does not need to read the entire file; it can simply seek to EOF and work from there.

This approach also does not need to create temp files behind your back (e.g. sed -i), so hardlinks aren't affected.

echo appends a newline to the file only when the result of the command substitution is a non-empty string. Note that this can only happen if the file is not empty and the last byte is not a newline.

If the last byte of the file is a newline, tail returns it, then command substitution strips it; the result is an empty string. The -n test fails and echo does not run.

If the file is empty, the result of the command substitution is also an empty string, and again echo does not run. This is desirable, because an empty file is not an invalid text file, nor is it equivalent to a non-empty text file with an empty line.

Answer 6

Add newline regardless:

echo >> filename

Here is a way to check if a newline exists at the end before adding one, by using Python:

f=filename; python -c "import sys; sys.exit(open(\"$f\").read().endswith('\n'))" && echo >> $f

Answer 7

The fastest solution is:

[ -n "$(tail -c1 file)" ] && printf '\n' >>file 

---

1. Is really fast.
   On a medium size file seq 99999999 >file this takes miliseconds.
   Other solutions take a long time:

   [ -n "$(tail -c1 file)" ] && printf '\n' >>file  0.013 sec
   vi -ecwq file                                    2.544 sec
   paste file 1<> file                             31.943 sec
   ed -s file <<< w                             1m  4.422 sec
   sed -i -e '$a\' file                         3m 20.931 sec
   

2. Works in ash, bash, lksh, mksh, ksh93, attsh and zsh but not yash.

3. Does not change file timestamp if there is no need to add a newline.
   All other solutions presented here change the timestamp of file.
4. All solutions above are valid POSIX.

If you need a solution portable to yash (and all other shells listed above), it may get a bit more complex:

f=file
if       [ "$(tail -c1 "$f"; echo x)" != "$(printf '\nx')" ]
then     printf '\n' >>"$f"
fi

Answer 8

The fastest way to test if the last byte of a file is a newline is to read only that last byte. That could be done with tail -c1 file. However, the simplistic way to test if the byte value is a new line, depending on the shell usual removal of a trailing new line inside a command expansion fails (for example) in yash, when the last character in the file is an UTF-8 value.

The correct, POSIX-compliant, all (reasonable) shells way to find if the last byte of a file is a new line is to use either xxd or hexdump:

tail -c1 file | xxd -u -p
tail -c1 file | hexdump -v -e '/1 "%02X"'

Then, comparing the output of above to 0A will provide a robust test.
It is useful to avoid adding a new line to an otherwise empty file.
File that will fail to provide a last character of 0A, of course:

f=file
a=$(tail -c1 "$f" | hexdump -v -e '/1 "%02X"')
[ -s "$f" -a "$a" != "0A" ] && echo >> "$f"

Short and sweet. This takes very little time as it just reads the the last byte (seek to EOF). It does not matter if the file is big. Then only add one byte if needed.

No temp files needed nor used. No hardlinks are affected.

If this test is run twice, it will not add another newline.

Answer 9

If you just want to quickly add a newline when processing some pipeline, use this:

outputting_program | { cat ; echo ; }

it's also POSIX compliant.

Then, of course, you can redirect it to a file.

Answer 10

Provided there are no nulls in input:

paste - <>infile >&0

...would suffice to always only append a newline to the tail end of an infile if it didn't have one already. And it need only read the input file through the one time to get it right.

Answer 11

At least in the GNU versions, simply grep '' or awk 1 canonicalizes its input, adding a final newline if not already present. They do copy the file in the process, which takes time if large (but source shouldn't be too large to read anyway?) and updates the modtime unless you do something like

 mv file old; grep '' <old >file; touch -r old file

(although that may be okay on a file you are checking-in because you modified it) and it loses hardlinks, nondefault permissions and ACLs etc unless you are even more careful.

Answer 12

Although it doesn't directly answer the question, here is a related script I wrote to detect files which do not end in newline. It is very fast.

find . -type f | # sort |        # sort file names if you like
/usr/bin/perl -lne '
   open FH, "<", $_ or do { print " error: $_"; next };
   $pos = sysseek FH, 0, 2;                     # seek to EOF
   if (!defined $pos)     { print " error: $_"; next }
   if ($pos == 0)         { print " empty: $_"; next }
   $pos = sysseek FH, -1, 1;                    # seek to last char
   if (!defined $pos)     { print " error: $_"; next }
   $cnt = sysread FH, $c, 1;
   if (!$cnt)             { print " error: $_"; next }
   if ($c eq "\n")        { print "   EOL: $_"; next }
   else                   { print "no EOL: $_"; next }
'

The perl script reads a list of (optionally sorted) file names from stdin and for every file it reads the last byte to determine if the file ends in a newline or not. It is very fast because it avoids reading the entire contents of each file. It outputs one line for each file it reads, prefixed with "error:" if some kind of error occurs, "empty:" if the file is empty (doesn't end with newline!), "EOL:" ("end of line") if the file ends with newline and "no EOL:" if the file doesn't end with newline.

Note: the script doesn't handle file names which contain newlines. If you're on a GNU or BSD system, you could handle all possible file names by adding -print0 to find, -z to sort, and -0 to perl, like this:

find . -type f -print0 | sort -z |
/usr/bin/perl -ln0e '
   open FH, "<", $_ or do { print " error: $_"; next };
   $pos = sysseek FH, 0, 2;                     # seek to EOF
   if (!defined $pos)     { print " error: $_"; next }
   if ($pos == 0)         { print " empty: $_"; next }
   $pos = sysseek FH, -1, 1;                    # seek to last char
   if (!defined $pos)     { print " error: $_"; next }
   $cnt = sysread FH, $c, 1;
   if (!$cnt)             { print " error: $_"; next }
   if ($c eq "\n")        { print "   EOL: $_"; next }
   else                   { print "no EOL: $_"; next }
'

Of course, you'd still have to come up with a way of encoding the file names with newlines in the output (left as an exercise for the reader).

The output could be filtered, if desired, to append a newline to those files which don't have one, most simply with

 echo >> "$filename"

Lack of a final newline can cause bugs in scripts since some versions of shell and other utilities will not properly handle a missing final newline when reading such a file.

In my experience, the lack of a final newline is caused by using various Windows utilities to edit files. I have never seen vim cause a missing final newline when editing a file, although it will report on such files.

Finally, there are much shorter (but slower) scripts which can loop over their file name inputs to print those files which do not end in newline, such as:

/usr/bin/perl -ne 'print "$ARGV\n" if /.\z/' -- FILE1 FILE2 ...

Answer 13

The vi/vim/ex editors automatically add <EOL> at EOF unless file already has it.

So try either:

vi -ecwq foo.txt

which is equivalent to:

ex -cwq foo.txt

Testing:

$ printf foo > foo.txt && wc foo.txt
0 1 3 foo.txt
$ ex -scwq foo.txt && wc foo.txt
1 1 4 foo.txt

To correct multiple files, check: How to fix 'No newline at end of file' for lots of files? at SO

Why this is so important? To keep our files POSIX compatible.

Answer 14

To apply the accepted answer to all files in the current directory (plus subdirectories):

$ find . -type f -exec sed -i -e '$a\' {} \;

This works on Linux (Ubuntu). On OS X you probably have to use -i '' (untested).

Answer 15

To fix all files in a git repo run

git ls-files --eol |\
 grep -e 'i/lf' |\
 grep -v 'attr/-text' |\
 sed 's/.*\t//' |\
 xargs -d '\n' sed -b -i -e '$a\'

• git ls-files --eol list all files tracked by git with their eol attribute
• grep -e 'i/lf' filter files checked into the index with LF
• grep -v 'attr/-text' skip files that are marked as binary or -text in .gitattributes
• sed 's/.*\t//' filter out everything but the paths
• xargs -d '\n' sed -b -i -e '$a\' add a newline at the end of the file
  • -b treat the file as binary (don't touch line endings)
  • -i edits the file in place
  • -e '$a\' add a newline at the end of the file but only if there's no newline at the end of the file and the file isn't empty.

Answer 16

You could write a fix-non-delimited-line script like:

#! /bin/zsh -
zmodload zsh/system || exit
ret=0
for file do
  if sysopen -rwu0 -- "$file"; then
    if sysseek -w end -1; then
      read -r x || print -u0
    else
      syserror -p "Can't seek in $file before the last byte: "
      ret=1
    fi
  else
    ret=1
  fi
done
exit $ret

Contrary to some of the solutions given here, it

• should be efficient in that it doesn't fork any process, only reads one byte for each file, and doesn't rewrite the file over (just appends a newline)
• will not break symlinks/hardlinks or affect metadata (also, the ctime/mtime are only updated when a newline is added)
• should work OK even if the last byte is a NUL or is part of a multi-byte character.
• should work OK regardless of what characters or non-characters the file names may contain
• Should handle correctly unreadable or unwritable or unseekable files (and report errors accordingly)
• Should not add a newline to empty files (but reports an error about an invalid seek in that case)

You can use it for instance as:

that-script *.txt

or:

git ls-files -z | xargs -0 that-script

POSIXly, you could do something functionally equivalent with

export LC_ALL=C
ret=0
for file do
  [ -s "$file" ] || continue
  {
    c=$(tail -c 1 | od -An -vtc)
    case $c in
      (*'\n'*) ;;
      (*[![:space:]]*) printf '\n' >&0 || ret=$?;;
      (*) ret=1;; # tail likely failed
    esac
  } 0<> "$file" || ret=$? # record failure to open
done

Answer 17

This works in AIX ksh:

lastchar=`tail -c 1 *filename*`
if [ `echo "$lastchar" | wc -c` -gt "1" ]
then
    echo "/n" >> *filename*
fi

In my case, if the file is missing the newline, the wc command returns a value of 2 and we write a newline.

Answer 18

Adding to Patrick Oscity's answer, if you just want to apply it to a specific directory, you could also use:

find -type f | while read f; do tail -n1 $f | read -r _ || echo >> $f; done

Run this inside the directory you would like to add newlines to.

Answer 19

echo $'' >> <FILE_NAME> will add a blank line to the end of the file.

echo $'\n\n' >> <FILE_NAME> will add 3 blank lines to the end of the file.

Answer 20

If your file is terminated with Windows line endings \r\n and you are in Linux you can use this sed command. It only adds \r\n to the last line if it is not already there:

sed -i -e '$s/\([^\r]\)$/\1\r\n/'

Explanation:

-i    replace in place
-e    script to run
$     matches last line of a file
s     substitute
\([^\r]\)$    search the last character in the line which is not a \r
\1\r\n    replace it with itself and add \r\n

If the last line already contains a \r\n then the search regexp will not match, therefore nothing will happen.