191

Using version control systems I get annoyed at the noise when the diff says No newline at end of file.

So I was wondering: How to add a newline at the end of a file to get rid of those messages?

20 Answers 20

45

To recursively sanitize a project I use this oneliner:

git ls-files -z | while IFS= read -rd '' f; do tail -c1 < "$f" | read -r _ || echo >> "$f"; done

Explanation:

  • git ls-files -z lists files in the repository. It takes an optional pattern as additional parameter which might be useful in some cases if you want to restrict the operation to certain files/directories. As an alternative, you could use find -print0 ... or similar programs to list affected files - just make sure it emits NUL-delimited entries.

  • while IFS= read -rd '' f; do ... done iterates through the entries, safely handling filenames that include whitespace and/or newlines.

  • tail -c1 < "$f" reads the last char from a file.

  • read -r _ exits with a nonzero exit status if a trailing newline is missing.

  • || echo >> "$f" appends a newline to the file if the exit status of the previous command was nonzero.

  • You can also do it like this if you want to just sanitize a subset of your files: find -name \*.java | while read f; do tail -n1 $f | read -r _ || echo >> $f; done – Per Lundberg Mar 22 at 9:54
  • @StéphaneChazelas good suggestions, will try to incorporate this into my answer. – Patrick Oscity Mar 22 at 12:57
  • @PerLundberg you can also pass a pattern to git ls-files which will still save you from editing files that are not tracked in version control. – Patrick Oscity Mar 22 at 12:58
  • @StéphaneChazelas adding the IFS= to unset the separator is good to preserve surrounding whitespace. The null terminated entries are only relevant if you have files or directories with a newline in their name, which seems kind of far fetched, but is the more correct way to handle the generic case, I agree. Just as a small caveat: the -d option to read is not available in POSIX sh. – Patrick Oscity Mar 22 at 13:26
  • Yes, hence my zsh/bash's. See also my use of tail -n1 < "$f" to avoid problems with file names that start with - (tail -n1 -- "$f" doesn't work for the file called -). You may want to clarify that the answer is now zsh/bash specific. – Stéphane Chazelas Mar 22 at 13:30
207

Here you go:

sed -i -e '$a\' file

And alternatively for OS X sed:

sed -i '' -e '$a\' file

This adds \n at the end of the file only if it doesn’t already end with a newline. So if you run it twice, it will not add another newline:

$ cd "$(mktemp -d)"
$ printf foo > test.txt
$ sed -e '$a\' test.txt > test-with-eol.txt
$ diff test*
1c1
< foo
\ No newline at end of file
---
> foo
$ echo $?
1
$ sed -e '$a\' test-with-eol.txt > test-still-with-one-eol.txt
$ diff test-with-eol.txt test-still-with-one-eol.txt
$ echo $?
0
  • 1
    @jwd: From man sed: $ Match the last line. But maybe it works only by accident. Your solution also works. – l0b0 Feb 20 '12 at 11:54
  • 1
    Your solution is also more elegant, and I've tested & committed it, but how can it work? If $ matches the last line, why doesn't it add another newline to a string which already contains a newline? – l0b0 Feb 20 '12 at 12:09
  • 27
    There are two different meanings of $. Inside a regex, such as with the form /<regex>/, it has the usual "match end of line" meaning. Otherwise, used as an address, sed gives it the special "last line in file" meaning. The code works because sed by default appends a newline to its output if it is not already there. The code "$a\" just says "match the last line of the file, and add nothing to it." But implicitly, sed adds the newline to every line it processes (such as this $ line) if it is not already there. – jwd Feb 22 '12 at 19:07
  • 1
    Regarding the manpage: The quote you are referring to is under the "Addresses" section. Putting it inside /regex/ gives it a different meaning. The FreeBSD manpages are a little more informative, I think: freebsd.org/cgi/man.cgi?query=sed – jwd Feb 22 '12 at 19:15
  • 2
    If the file already ends in a newline, this doesn't change it, but it does rewrite it and update its timestamp. That may or may not matter. – Keith Thompson Feb 19 '16 at 19:01
39

Have a look:

$ echo -n foo > foo 
$ cat foo
foo$
$ echo "" >> foo
$ cat foo
foo

so echo "" >> noeol-file should do the trick. (Or did you mean to ask for identifying these files and fixing them?)

edit removed the "" from echo "" >> foo (see @yuyichao's comment) edit2 added the "" again (but see @Keith Thompson's comment)

  • 4
    the "" is not necessary (at least for bash) and tail -1 | wc -l can be used to find out the file without a new line at the end – yuyichao Feb 17 '12 at 14:42
  • 5
    @yuyichao: The "" isn't necessary for bash, but I've seen echo implementations that print nothing when invoked without arguments (though none of the ones I can find now do this). echo "" >> noeol-file is probably slightly more robust. printf "\n" >> noeol-file is even more so. – Keith Thompson Feb 17 '12 at 17:17
  • 2
    @KeithThompson, csh's echo is the one known to output nothing when not passed any argument. But then if we're going to support non-Bourne-like shells, we should make it echo '' instead of echo "" as echo "" would ouput ""<newline> with rc or es for instance. – Stéphane Chazelas Feb 19 '16 at 11:49
  • 1
    @StéphaneChazelas: And tcsh, unlike csh, prints a newline when invoked with no arguments -- regardless of the setting of $echo_style. – Keith Thompson Feb 19 '16 at 18:55
16

Another solution using ed. This solution only affect the last line and only if \n is missing:

ed -s file <<< w

It essentially works opening the file for editing through a script, the script is the single w command, that write the file back to disk. It is based on this sentence found in ed(1) man page:

LIMITATIONS
       (...)

       If  a  text (non-binary) file is not terminated by a newline character,
       then ed appends one on reading/writing it.  In the  case  of  a  binary
       file, ed does not append a newline on reading/writing.
  • 1
    This does not add a newline for me. – Olhovsky Apr 12 '13 at 1:46
  • 4
    Works for me; it even prints "Newline appended" (ed-1.10-1 on Arch Linux). – Stefan Majewsky Mar 10 '15 at 10:00
12

A simple, portable, POSIX-compliant way to add an absent, final newline to a would be text file:

[ -n "$(tail -c1 file)" ] && echo >> file

This approach does not need to read the entire file; it can simply seek to EOF and work from there.

This approach also does not need to create temp files behind your back (e.g. sed -i), so hardlinks aren't affected.

echo appends a newline to the file only when the result of the command substitution is a non-empty string. Note that this can only happen if the file is not empty and the last byte is not a newline.

If the last byte of the file is a newline, tail returns it, then command substitution strips it; the result is an empty string. The -n test fails and echo does not run.

If the file is empty, the result of the command substitution is also an empty string, and again echo does not run. This is desirable, because an empty file is not an invalid text file, nor is it equivalent to a non-empty text file with an empty line.

  • 1
    Note that it doesn't work with yash if the last character in the file is a multi-byte character (in UTF-8 locales for instance), or if the locale is C and the last byte in the file has the 8th bit set. With other shells (except zsh), it would not add a newline if the file ended in a NUL byte (but then again, that would mean the input would be non-text even after a newline is added). – Stéphane Chazelas Feb 19 '16 at 11:39
  • 1
    @StéphaneChazelas A solution for yash has been added. – sorontar Jan 14 '17 at 23:25
  • 1
    Is it possible to run this for every file in a folder and subfolders? – Qwerty Mar 24 '17 at 11:28
12

Add newline regardless:

echo >> filename

Here is a way to check if a newline exists at the end before adding one, by using Python:

f=filename; python -c "import sys; sys.exit(open(\"$f\").read().endswith('\n'))" && echo >> $f
  • 1
    I wouldn't use the python version in any sort of loop because of the slow python startup time. Of course you could do the loop in python if you wanted. – Kevin Cox Nov 9 '13 at 14:03
  • 3
    The startup time for Python is 0.03 seconds here. Do you really consider that to be problematic? – Alexander Nov 10 '13 at 11:48
  • 3
    Startup time does matter if you call python in a loop, that is why I said consider doing the loop in python. Then you only incur the startup cost once. For me, half the cost the startup is more than half of the time of the whole snipit, I would consider that substantial overhead. (Again, irrelevant if only doing a small number of files) – Kevin Cox Nov 11 '13 at 16:35
  • 2
    echo "" seems more robust than echo -n '\n'. Or you could use printf '\n' – Keith Thompson Feb 19 '16 at 18:58
  • 2
    This worked fine for me – Daniel Gomez Rico Jan 18 at 22:58
8

The fastest solution is:

[ -n "$(tail -c1 file)" ] && printf '\n' >>file 

  1. Is really fast.
    On a medium size file seq 99999999 >file this takes miliseconds.
    Other solutions take a long time:

    [ -n "$(tail -c1 file)" ] && printf '\n' >>file  0.013 sec
    vi -ecwq file                                    2.544 sec
    paste file 1<> file                             31.943 sec
    ed -s file <<< w                             1m  4.422 sec
    sed -i -e '$a\' file                         3m 20.931 sec
    
  2. Works in ash, bash, lksh, mksh, ksh93, attsh and zsh but not yash.

  3. Does not change file timestamp if there is no need to add a newline.
    All other solutions presented here change the timestamp of file.
  4. All solutions above are valid POSIX.

If you need a solution portable to yash (and all other shells listed above), it may get a bit more complex:

f=file
if       [ "$(tail -c1 "$f"; echo x)" != "$(printf '\nx')" ]
then     printf '\n' >>"$f"
fi
7

The fastest way to test if the last byte of a file is a newline is to read only that last byte. That could be done with tail -c1 file. However, the simplistic way to test if the byte value is a new line, depending on the shell usual removal of a trailing new line inside a command expansion fails (for example) in yash, when the last character in the file is an UTF-8 value.

The correct, POSIX-compliant, all (reasonable) shells way to find if the last byte of a file is a new line is to use either xxd or hexdump:

tail -c1 file | xxd -u -p
tail -c1 file | hexdump -v -e '/1 "%02X"'

Then, comparing the output of above to 0A will provide a robust test.
It is useful to avoid adding a new line to an otherwise empty file.
File that will fail to provide a last character of 0A, of course:

f=file
a=$(tail -c1 "$f" | hexdump -v -e '/1 "%02X"')
[ -s "$f" -a "$a" != "0A" ] && echo >> "$f"

Short and sweet. This takes very little time as it just reads the the last byte (seek to EOF). It does not matter if the file is big. Then only add one byte if needed.

No temp files needed nor used. No hardlinks are affected.

If this test is run twice, it will not add another newline.

  • 1
    @crw I do believe that it adds useful information. – sorontar Jan 14 '17 at 23:17
  • 2
    Note that neither xxd nor hexdump are POSIX utilities. In the POSIX toolchest, there's od -An -tx1 to get the hex value of a byte. – Stéphane Chazelas May 15 '18 at 15:45
  • @StéphaneChazelas Please post that as an answer; I have come here looking for this comment too many times :) – kelvin Sep 19 at 21:15
  • @kelvin, I've updated my answer – Stéphane Chazelas Sep 20 at 8:27
  • Note that POSIX doesn't guarantee the value of LF to be 0x0a. There are still POSIX systems where it's not (EBCDIC based ones) though they're extremely rare these days. – Stéphane Chazelas Sep 20 at 8:29
4

You're better off correcting the editor of the user who last edited the file. If you are the last person to have edited the file - what editor are you using, I'm guessing textmate ..?

  • 2
    Vim is the editor in question. But in general, you are right, I should not only fix the symptons ;) – k0pernikus Feb 17 '12 at 13:46
  • 6
    for vim, you have to go out of your way and do the binary-file-on-save dance to get vim to not add a new line at the end of the file - just don't do that dance. OR, to simply correct existing files open them in vim and save the file and vim will 'fix' the missing newline for you (can be easily scripted for multiple files) – AD7six Feb 17 '12 at 13:50
  • 3
    My emacs do not add a newline at end of file. – enzotib Feb 20 '12 at 17:45
  • 2
    Thanks for the comment @AD7six, I keep getting phantom reports from diffs when I commit things, about how the original file does not have a newline at the end. No matter how I edit a file with vim i can't get it to not put a newline there. So it's just vim doing it. – Steven Lu Jun 21 '13 at 19:39
  • 1
    @enzotib: I have (setq require-final-newline 'ask) in my .emacs – Keith Thompson Feb 19 '16 at 18:59
3

If you just want to quickly add a newline when processing some pipeline, use this:

outputting_program | { cat ; echo ; }

it's also POSIX compliant.

Then, of course, you can redirect it to a file.

  • 2
    The fact that I can use this in a pipeline is helpful. This allows me to count the number of rows in a CSV file, excluding the header. And it helps get an accurate line count on windows files that don't end with a newline or carriage return. cat file.csv | tr "\r" "\n" | { cat; echo; } | sed "/^[[:space:]]*$/d" | tail -n +2 | wc -l – Kyle Tolle Dec 22 '15 at 16:54
3

Provided there are no nulls in input:

paste - <>infile >&0

...would suffice to always only append a newline to the tail end of an infile if it didn't have one already. And it need only read the input file through the one time to get it right.

  • That won't work like that as stdin and stdout share the same open-file description (so cursor within the file). You'd need paste infile 1<> infile instead. – Stéphane Chazelas Aug 18 '17 at 14:03
2

Although it doesn't directly answer the question, here is a related script I wrote to detect files which do not end in newline. It is very fast.

find . -type f | # sort |        # sort file names if you like
/usr/bin/perl -lne '
   open FH, "<", $_ or do { print " error: $_"; next };
   $pos = sysseek FH, 0, 2;                     # seek to EOF
   if (!defined $pos)     { print " error: $_"; next }
   if ($pos == 0)         { print " empty: $_"; next }
   $pos = sysseek FH, -1, 1;                    # seek to last char
   if (!defined $pos)     { print " error: $_"; next }
   $cnt = sysread FH, $c, 1;
   if (!$cnt)             { print " error: $_"; next }
   if ($c eq "\n")        { print "   EOL: $_"; next }
   else                   { print "no EOL: $_"; next }
'

The perl script reads a list of (optionally sorted) file names from stdin and for every file it reads the last byte to determine if the file ends in a newline or not. It is very fast because it avoids reading the entire contents of each file. It outputs one line for each file it reads, prefixed with "error:" if some kind of error occurs, "empty:" if the file is empty (doesn't end with newline!), "EOL:" ("end of line") if the file ends with newline and "no EOL:" if the file doesn't end with newline.

Note: the script doesn't handle file names which contain newlines. If you're on a GNU or BSD system, you could handle all possible file names by adding -print0 to find, -z to sort, and -0 to perl, like this:

find . -type f -print0 | sort -z |
/usr/bin/perl -ln0e '
   open FH, "<", $_ or do { print " error: $_"; next };
   $pos = sysseek FH, 0, 2;                     # seek to EOF
   if (!defined $pos)     { print " error: $_"; next }
   if ($pos == 0)         { print " empty: $_"; next }
   $pos = sysseek FH, -1, 1;                    # seek to last char
   if (!defined $pos)     { print " error: $_"; next }
   $cnt = sysread FH, $c, 1;
   if (!$cnt)             { print " error: $_"; next }
   if ($c eq "\n")        { print "   EOL: $_"; next }
   else                   { print "no EOL: $_"; next }
'

Of course, you'd still have to come up with a way of encoding the file names with newlines in the output (left as an exercise for the reader).

The output could be filtered, if desired, to append a newline to those files which don't have one, most simply with

 echo >> "$filename"

Lack of a final newline can cause bugs in scripts since some versions of shell and other utilities will not properly handle a missing final newline when reading such a file.

In my experience, the lack of a final newline is caused by using various Windows utilities to edit files. I have never seen vim cause a missing final newline when editing a file, although it will report on such files.

Finally, there are much shorter (but slower) scripts which can loop over their file name inputs to print those files which do not end in newline, such as:

/usr/bin/perl -ne 'print "$ARGV\n" if /.\z/' -- FILE1 FILE2 ...
1

The vi/vim/ex editors automatically add <EOL> at EOF unless file already has it.

So try either:

vi -ecwq foo.txt

which is equivalent to:

ex -cwq foo.txt

Testing:

$ printf foo > foo.txt && wc foo.txt
0 1 3 foo.txt
$ ex -scwq foo.txt && wc foo.txt
1 1 4 foo.txt

To correct multiple files, check: How to fix 'No newline at end of file' for lots of files? at SO

Why this is so important? To keep our files POSIX compatible.

0

To apply the accepted answer to all files in the current directory (plus subdirectories):

$ find . -type f -exec sed -i -e '$a\' {} \;

This works on Linux (Ubuntu). On OS X you probably have to use -i '' (untested).

  • 4
    Note that find . lists all files, including files in .git. To exclude: find . -type f -not -path './.git/*' -exec sed -i -e '$a\' {} \; – friederbluemle Jul 14 '15 at 7:59
  • Wish I would have read this comment/thought about it before I ran it. Oh well. – kstev Nov 9 '15 at 4:02
0

At least in the GNU versions, simply grep '' or awk 1 canonicalizes its input, adding a final newline if not already present. They do copy the file in the process, which takes time if large (but source shouldn't be too large to read anyway?) and updates the modtime unless you do something like

 mv file old; grep '' <old >file; touch -r old file

(although that may be okay on a file you are checking-in because you modified it) and it loses hardlinks, nondefault permissions and ACLs etc unless you are even more careful.

  • Or just grep '' file 1<> file, though that would still read and write the file fully. – Stéphane Chazelas Aug 18 '17 at 14:08
-1

This works in AIX ksh:

lastchar=`tail -c 1 *filename*`
if [ `echo "$lastchar" | wc -c` -gt "1" ]
then
    echo "/n" >> *filename*
fi

In my case, if the file is missing the newline, the wc command returns a value of 2 and we write a newline.

  • Feedback will come in forms of up- or downvotes, or you will be asked in comments to outline your answers/questions more, no point in asking for it in the answer's body. Keep it to the point, welcome to stackexchange! – k0pernikus Feb 12 '15 at 2:45
-1

Adding to Patrick Oscity's answer, if you just want to apply it to a specific directory, you could also use:

find -type f | while read f; do tail -n1 $f | read -r _ || echo >> $f; done

Run this inside the directory you would like to add newlines to.

-1

echo $'' >> <FILE_NAME> will add a blank line to the end of the file.

echo $'\n\n' >> <FILE_NAME> will add 3 blank lines to the end of the file.

-1

If your file is terminated with Windows line endings \r\n and you are in Linux you can use this sed command. It only adds \r\n to the last line if it is not already there:

sed -i -e '$s/\([^\r]\)$/\1\r\n/'

Explanation:

-i    replace in place
-e    script to run
$     matches last line of a file
s     substitute
\([^\r]\)$    search the last character in the line which is not a \r
\1\r\n    replace it with itself and add \r\n

If the last line already contains a \r\n then the search regexp will not match, therefore nothing will happen.

-1

You could write a fix-non-delimited-line script like:

#! /bin/zsh -
zmodload zsh/system || exit
ret=0
for file do
  if sysopen -rwu0 -- "$file"; then
    if sysseek -w end -1; then
      read -r x || print -u0
    else
      syserror -p "Can't seek in $file before the last byte: "
      ret=1
    fi
  else
    ret=1
  fi
done
exit $ret

Contrary to some of the solutions given here, it

  • should be efficient in that it doesn't fork any process, only reads one byte for each file, and doesn't rewrite the file over (just appends a newline)
  • will not break symlinks/hardlinks or affect metadata (also, the ctime/mtime are only updated when a newline is added)
  • should work OK even if the last byte is a NUL or is part of a multi-byte character.
  • should work OK regardless of what characters or non-characters the file names may contain
  • Should handle correctly unreadable or unwritable or unseekable files (and report errors accordingly)
  • Should not add a newline to empty files (but reports an error about an invalid seek in that case)

You can use it for instance as:

that-script *.txt

or:

git ls-files -z | xargs -0 that-script

POSIXly, you could do something functionaly equivalent with

export LC_ALL=C
ret=0
for file do
  [ -s "$file" ] || continue
  {
    c=$(tail -c 1 | od -An -vtc)
    case $c in
      (*'\n'*) ;;
      (*[![:space:]]*) printf '\n' >&0 || ret=$?;;
      (*) ret=1;; # tail likely failed
    esac
  } 0<> "$file" || ret=$? # record failure to open
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.