Using version control systems I get annoyed at the noise when the diff says No newline at end of file
.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this communityUsing version control systems I get annoyed at the noise when the diff says No newline at end of file
.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
To recursively sanitize a project I use this oneliner:
git ls-files -z | while IFS= read -rd '' f; do tail -c1 < "$f" | read -r _ || echo >> "$f"; done
Explanation:
git ls-files -z
lists files in the repository. It takes an optional pattern as additional parameter which might be useful in some cases if you want to restrict the operation to certain files/directories. As an alternative, you could use find -print0 ...
or similar programs to list affected files - just make sure it emits NUL
-delimited entries.
while IFS= read -rd '' f; do ... done
iterates through the entries, safely handling filenames that include whitespace and/or newlines.
tail -c1 < "$f"
reads the last char from a file.
read -r _
exits with a nonzero exit status if a trailing newline is missing.
|| echo >> "$f"
appends a newline to the file if the exit status of the previous command was nonzero.
find -name \*.java | while read f; do tail -n1 $f | read -r _ || echo >> $f; done
– Per Lundberg
Mar 22 '19 at 9:54
git ls-files
which will still save you from editing files that are not tracked in version control.
– Patrick Oscity
Mar 22 '19 at 12:58
IFS=
to unset the separator is good to preserve surrounding whitespace. The null terminated entries are only relevant if you have files or directories with a newline in their name, which seems kind of far fetched, but is the more correct way to handle the generic case, I agree. Just as a small caveat: the -d
option to read
is not available in POSIX sh.
– Patrick Oscity
Mar 22 '19 at 13:26
sed -i -e '$a\' file
And alternatively for OS X sed
:
sed -i '' -e '$a\' file
This adds \n
at the end of the file only if it doesn’t already end with a newline. So if you run it twice, it will not add another newline:
$ cd "$(mktemp -d)"
$ printf foo > test.txt
$ sed -e '$a\' test.txt > test-with-eol.txt
$ diff test*
1c1
< foo
\ No newline at end of file
---
> foo
$ echo $?
1
$ sed -e '$a\' test-with-eol.txt > test-still-with-one-eol.txt
$ diff test-with-eol.txt test-still-with-one-eol.txt
$ echo $?
0
man sed
: $ Match the last line.
But maybe it works only by accident. Your solution also works.
– l0b0
Feb 20 '12 at 11:54
$
. Inside a regex, such as with the form /<regex>/
, it has the usual "match end of line" meaning. Otherwise, used as an address, sed gives it the special "last line in file" meaning. The code works because sed by default appends a newline to its output if it is not already there. The code "$a\" just says "match the last line of the file, and add nothing to it." But implicitly, sed adds the newline to every line it processes (such as this $
line) if it is not already there.
– jwd
Feb 22 '12 at 19:07
sed '$q'
with GNU sed 4.4
, it didn't work. q
just quits without doing anything. a\
has some extra logic that will add a trailing newline if it doesn't exist.
– wisbucky
Sep 25 '19 at 23:31
Have a look:
$ echo -n foo > foo
$ cat foo
foo$
$ echo "" >> foo
$ cat foo
foo
so echo "" >> noeol-file
should do the trick. (Or did you mean to ask for identifying these files and fixing them?)
edit removed the ""
from echo "" >> foo
(see @yuyichao's comment)
edit2 added the ""
again (but see @Keith Thompson's comment)
""
is not necessary (at least for bash) and tail -1 | wc -l
can be used to find out the file without a new line at the end
– yuyichao
Feb 17 '12 at 14:42
""
isn't necessary for bash, but I've seen echo
implementations that print nothing when invoked without arguments (though none of the ones I can find now do this). echo "" >> noeol-file
is probably slightly more robust. printf "\n" >> noeol-file
is even more so.
– Keith Thompson
Feb 17 '12 at 17:17
csh
's echo
is the one known to output nothing when not passed any argument. But then if we're going to support non-Bourne-like shells, we should make it echo ''
instead of echo ""
as echo ""
would ouput ""<newline>
with rc
or es
for instance.
– Stéphane Chazelas
Feb 19 '16 at 11:49
tcsh
, unlike csh
, prints a newline when invoked with no arguments -- regardless of the setting of $echo_style
.
– Keith Thompson
Feb 19 '16 at 18:55
Another solution using ed
. This solution only affect the last line and only if \n
is missing:
ed -s file <<< w
It essentially works opening the file for editing through a script, the script is the single w
command, that write the file back to disk. It is based on this sentence found in ed(1)
man page:
LIMITATIONS (...) If a text (non-binary) file is not terminated by a newline character, then ed appends one on reading/writing it. In the case of a binary file, ed does not append a newline on reading/writing.
Add newline regardless:
echo >> filename
Here is a way to check if a newline exists at the end before adding one, by using Python:
f=filename; python -c "import sys; sys.exit(open(\"$f\").read().endswith('\n'))" && echo >> $f
echo ""
seems more robust than echo -n '\n'
. Or you could use printf '\n'
– Keith Thompson
Feb 19 '16 at 18:58
A simple, portable, POSIX-compliant way to add an absent, final newline to a would be text file:
[ -n "$(tail -c1 file)" ] && echo >> file
This approach does not need to read the entire file; it can simply seek to EOF and work from there.
This approach also does not need to create temp files behind your back (e.g. sed -i), so hardlinks aren't affected.
echo appends a newline to the file only when the result of the command substitution is a non-empty string. Note that this can only happen if the file is not empty and the last byte is not a newline.
If the last byte of the file is a newline, tail returns it, then command substitution strips it; the result is an empty string. The -n test fails and echo does not run.
If the file is empty, the result of the command substitution is also an empty string, and again echo does not run. This is desirable, because an empty file is not an invalid text file, nor is it equivalent to a non-empty text file with an empty line.
yash
if the last character in the file is a multi-byte character (in UTF-8 locales for instance), or if the locale is C and the last byte in the file has the 8th bit set. With other shells (except zsh), it would not add a newline if the file ended in a NUL byte (but then again, that would mean the input would be non-text even after a newline is added).
– Stéphane Chazelas
Feb 19 '16 at 11:39
The fastest solution is:
[ -n "$(tail -c1 file)" ] && printf '\n' >>file
Is really fast.
On a medium size file seq 99999999 >file
this takes miliseconds.
Other solutions take a long time:
[ -n "$(tail -c1 file)" ] && printf '\n' >>file 0.013 sec
vi -ecwq file 2.544 sec
paste file 1<> file 31.943 sec
ed -s file <<< w 1m 4.422 sec
sed -i -e '$a\' file 3m 20.931 sec
Works in ash, bash, lksh, mksh, ksh93, attsh and zsh but not yash.
If you need a solution portable to yash (and all other shells listed above), it may get a bit more complex:
f=file
if [ "$(tail -c1 "$f"; echo x)" != "$(printf '\nx')" ]
then printf '\n' >>"$f"
fi
The fastest way to test if the last byte of a file is a newline is to read only that last byte. That could be done with tail -c1 file
. However, the simplistic way to test if the byte value is a new line, depending on the shell usual removal of a trailing new line inside a command expansion fails (for example) in yash, when the last character in the file is an UTF-8 value.
The correct, POSIX-compliant, all (reasonable) shells way to find if the last byte of a file is a new line is to use either xxd or hexdump:
tail -c1 file | xxd -u -p
tail -c1 file | hexdump -v -e '/1 "%02X"'
Then, comparing the output of above to 0A
will provide a robust test.
It is useful to avoid adding a new line to an otherwise empty file.
File that will fail to provide a last character of 0A
, of course:
f=file
a=$(tail -c1 "$f" | hexdump -v -e '/1 "%02X"')
[ -s "$f" -a "$a" != "0A" ] && echo >> "$f"
Short and sweet. This takes very little time as it just reads the the last byte (seek to EOF). It does not matter if the file is big. Then only add one byte if needed.
No temp files needed nor used. No hardlinks are affected.
If this test is run twice, it will not add another newline.
xxd
nor hexdump
are POSIX utilities. In the POSIX toolchest, there's od -An -tx1
to get the hex value of a byte.
– Stéphane Chazelas
May 15 '18 at 15:45
You're better off correcting the editor of the user who last edited the file. If you are the last person to have edited the file - what editor are you using, I'm guessing textmate ..?
(setq require-final-newline 'ask)
in my .emacs
– Keith Thompson
Feb 19 '16 at 18:59
If you just want to quickly add a newline when processing some pipeline, use this:
outputting_program | { cat ; echo ; }
it's also POSIX compliant.
Then, of course, you can redirect it to a file.
cat file.csv | tr "\r" "\n" | { cat; echo; } | sed "/^[[:space:]]*$/d" | tail -n +2 | wc -l
– Kyle Tolle
Dec 22 '15 at 16:54
Provided there are no nulls in input:
paste - <>infile >&0
...would suffice to always only append a newline to the tail end of an infile if it didn't have one already. And it need only read the input file through the one time to get it right.
paste infile 1<> infile
instead.
– Stéphane Chazelas
Aug 18 '17 at 14:03
Although it doesn't directly answer the question, here is a related script I wrote to detect files which do not end in newline. It is very fast.
find . -type f | # sort | # sort file names if you like
/usr/bin/perl -lne '
open FH, "<", $_ or do { print " error: $_"; next };
$pos = sysseek FH, 0, 2; # seek to EOF
if (!defined $pos) { print " error: $_"; next }
if ($pos == 0) { print " empty: $_"; next }
$pos = sysseek FH, -1, 1; # seek to last char
if (!defined $pos) { print " error: $_"; next }
$cnt = sysread FH, $c, 1;
if (!$cnt) { print " error: $_"; next }
if ($c eq "\n") { print " EOL: $_"; next }
else { print "no EOL: $_"; next }
'
The perl script reads a list of (optionally sorted) file names from stdin and for every file it reads the last byte to determine if the file ends in a newline or not. It is very fast because it avoids reading the entire contents of each file. It outputs one line for each file it reads, prefixed with "error:" if some kind of error occurs, "empty:" if the file is empty (doesn't end with newline!), "EOL:" ("end of line") if the file ends with newline and "no EOL:" if the file doesn't end with newline.
Note: the script doesn't handle file names which contain newlines. If you're on a GNU or BSD system, you could handle all possible file names by adding -print0 to find, -z to sort, and -0 to perl, like this:
find . -type f -print0 | sort -z |
/usr/bin/perl -ln0e '
open FH, "<", $_ or do { print " error: $_"; next };
$pos = sysseek FH, 0, 2; # seek to EOF
if (!defined $pos) { print " error: $_"; next }
if ($pos == 0) { print " empty: $_"; next }
$pos = sysseek FH, -1, 1; # seek to last char
if (!defined $pos) { print " error: $_"; next }
$cnt = sysread FH, $c, 1;
if (!$cnt) { print " error: $_"; next }
if ($c eq "\n") { print " EOL: $_"; next }
else { print "no EOL: $_"; next }
'
Of course, you'd still have to come up with a way of encoding the file names with newlines in the output (left as an exercise for the reader).
The output could be filtered, if desired, to append a newline to those files which don't have one, most simply with
echo >> "$filename"
Lack of a final newline can cause bugs in scripts since some versions of shell and other utilities will not properly handle a missing final newline when reading such a file.
In my experience, the lack of a final newline is caused by using various Windows utilities to edit files. I have never seen vim cause a missing final newline when editing a file, although it will report on such files.
Finally, there are much shorter (but slower) scripts which can loop over their file name inputs to print those files which do not end in newline, such as:
/usr/bin/perl -ne 'print "$ARGV\n" if /.\z/' -- FILE1 FILE2 ...
The vi
/vim
/ex
editors automatically add <EOL>
at EOF unless file already has it.
So try either:
vi -ecwq foo.txt
which is equivalent to:
ex -cwq foo.txt
Testing:
$ printf foo > foo.txt && wc foo.txt
0 1 3 foo.txt
$ ex -scwq foo.txt && wc foo.txt
1 1 4 foo.txt
To correct multiple files, check: How to fix 'No newline at end of file' for lots of files? at SO
Why this is so important? To keep our files POSIX compatible.
At least in the GNU versions, simply grep ''
or awk 1
canonicalizes its input, adding a final newline if not already present. They do copy the file in the process, which takes time if large (but source shouldn't be too large to read anyway?) and updates the modtime unless you do something like
mv file old; grep '' <old >file; touch -r old file
(although that may be okay on a file you are checking-in because you modified it) and it loses hardlinks, nondefault permissions and ACLs etc unless you are even more careful.
grep '' file 1<> file
, though that would still read and write the file fully.
– Stéphane Chazelas
Aug 18 '17 at 14:08
To apply the accepted answer to all files in the current directory (plus subdirectories):
$ find . -type f -exec sed -i -e '$a\' {} \;
This works on Linux (Ubuntu). On OS X you probably have to use -i ''
(untested).
find .
lists all files, including files in .git
. To exclude: find . -type f -not -path './.git/*' -exec sed -i -e '$a\' {} \;
– friederbluemle
Jul 14 '15 at 7:59
This works in AIX ksh:
lastchar=`tail -c 1 *filename*`
if [ `echo "$lastchar" | wc -c` -gt "1" ]
then
echo "/n" >> *filename*
fi
In my case, if the file is missing the newline, the wc
command returns a value of 2
and we write a newline.
Adding to Patrick Oscity's answer, if you just want to apply it to a specific directory, you could also use:
find -type f | while read f; do tail -n1 $f | read -r _ || echo >> $f; done
Run this inside the directory you would like to add newlines to.
echo $'' >> <FILE_NAME>
will add a blank line to the end of the file.
echo $'\n\n' >> <FILE_NAME>
will add 3 blank lines to the end of the file.
If your file is terminated with Windows line endings \r\n
and you are in Linux you can use this sed
command. It only adds \r\n
to the last line if it is not already there:
sed -i -e '$s/\([^\r]\)$/\1\r\n/'
Explanation:
-i replace in place
-e script to run
$ matches last line of a file
s substitute
\([^\r]\)$ search the last character in the line which is not a \r
\1\r\n replace it with itself and add \r\n
If the last line already contains a \r\n
then the search regexp will not match, therefore nothing will happen.
You could write a fix-non-delimited-line
script like:
#! /bin/zsh -
zmodload zsh/system || exit
ret=0
for file do
if sysopen -rwu0 -- "$file"; then
if sysseek -w end -1; then
read -r x || print -u0
else
syserror -p "Can't seek in $file before the last byte: "
ret=1
fi
else
ret=1
fi
done
exit $ret
Contrary to some of the solutions given here, it
You can use it for instance as:
that-script *.txt
or:
git ls-files -z | xargs -0 that-script
POSIXly, you could do something functionally equivalent with
export LC_ALL=C
ret=0
for file do
[ -s "$file" ] || continue
{
c=$(tail -c 1 | od -An -vtc)
case $c in
(*'\n'*) ;;
(*[![:space:]]*) printf '\n' >&0 || ret=$?;;
(*) ret=1;; # tail likely failed
esac
} 0<> "$file" || ret=$? # record failure to open
done