43

I have trouble understanding a weird behavior: vi seems to add a newline (ASCII: LF, as it is a Unix (AIX) system) at the end of the file, when I did NOT specifically type it.

I edit the file as such in vi (taking care to not input a newline at the end):

# vi foo   ## Which I will finish on the char "9" and not input a last newline, then `:wq`
123456789
123456789
123456789
123456789
~
~
  ## When I save, the cursor is just above the last "9", and no newline was added.

I expect vi to save it "as is", so to have 39 bytes: 10 ASCII characters on each of the first three lines (numbers 1 to 9, followed by a newline (LF on my system)) and only 9 on the last line (characters 1 to 9, no terminating newline/LF).

But it appears when I save it it is 40 bytes (instead of 39), and od shows a terminating LF:

# wc foo
       4       4      40 foo  ## I expected 39 here! as I didn't add the last newline
# od -a toto
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9  lf
0000050
     ## An "lf" terminates the file?? Did vi add it silently?

If I create the file with a printf doing exactly what I did inside vi, it works as expected:

# ## I create a file with NO newline at the end:
# printf "123456789\n123456789\n123456789\n123456789" > foo2
# wc foo2  ## This one is as expected: 39 bytes, exactly as I was trying to do above with vi.
       3       4      39 foo  ## As expected, as I didn't add the last newline

  ## Note that for wc, there are only three lines!
  ## (So wc -l doesn't count lines; it counts the [newline] chars... Which is rather odd.)

# root@SPU0WMY1:~  ## od -a foo2
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9
0000047                                ## As expected, no added LF.

Both files (foo (40 characters) and foo2 (39 characters) appear exactly the same if I re-open them with vi...

And if I open foo2 (39 characters, no terminating newline) in vi and just do :wq without editing it whatsoever, it says it writes 40 chars, and the linefeed appears!

I can't have access to a more recent vi (I do this on AIX, vi (not Vim) version 3.10 I think? (no "-version" or other means of knowing it)).

# strings /usr/bin/vi | grep -i 'version.*[0-9]'
@(#) Version 3.10

Is it normal for vi (and perhaps not in more recent version? Or Vim?) to silently add a newline at the end of a file? (I thought the ~ indicated that the previous line did NOT end with a newline.)

--

Edit: some additional updates and a bit of a summary, with a big thanks to the answers below :

  • vi silently add a trailing newline at the moment it writes a file that lacked it (unless file is empty).

  • it only does so at the writing time! (ie, until you :w, you can use :e to verify that the file is still as you openened it... (ie: it still shows "filename" [Last line is not complete] N line, M character). When you save, a newline is silently added, without a specific warning (it does say how many bytes it saves, but this is in most cases not enough to know a newline was added) (thanks to @jiliagre for talking to me about the opening vi message, it helped me to find a way to know when the change really occurs)

  • This (silent correction) is POSIX behavior! (see @barefoot-io answer for references)

20
  • Just for completeness, which version of AIX (full version). Feb 17, 2016 at 15:52
  • 2
    I'm not aware of AIX's vi having this option -- appears vim-only
    – Jeff Schaller
    Feb 17, 2016 at 15:54
  • 1
    @JeffSchaller : thx for the link. Unfortunately native vi doesn't have ":set noeol" nor even the -b option to open in binary mode... Feb 17, 2016 at 16:27
  • 1
    You might be able to get vi version or at least a clue about its origin by running the :ve command.
    – jlliagre
    Feb 17, 2016 at 21:00
  • 1
    @ThomasDickey Indeed. For some reason, IBM stripped down the ex manual page where the :ver command is normally documented.
    – jlliagre
    Feb 18, 2016 at 11:19

4 Answers 4

54

POSIX requires this behavior, so it's not in any way unusual.

From the POSIX vi manual:

INPUT FILES

See the INPUT FILES section of the ex command for a description of the input files supported by the vi command.

Following the trail to the POSIX ex manual:

INPUT FILES

Input files shall be text files or files that would be text files except for an incomplete last line that is not longer than {LINE_MAX}-1 bytes in length and contains no NUL characters. By default, any incomplete last line shall be treated as if it had a trailing <newline>. The editing of other forms of files may optionally be allowed by ex implementations.

The OUTPUT FILES section of the vi manual also redirects to ex:

OUTPUT FILES

The output from ex shall be text files.

A pair of POSIX definitions:

3.397 Text File

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

3.206 Line

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

These definitions in the context of these manual page excerpts mean that while a conformant ex/vi implementation must accept a malformed text file if that file's only deformity is an absent final newline, when writing that file's buffer the result must be a valid text file.

While this post has referenced the 2013 edition of the POSIX standard, the relevant stipulations also appear in the much older 1997 edition.

Lastly, if you find ex's newline appension unwelcome, you will feel profoundly violated by Seventh Edition UNIX's (1979) intolerant ed. From the manual:

When reading a file, ed discards ASCII NUL characters and all characters after the last newline. It refuses to read files containing non-ASCII characters.

5
  • thanks, that does answer my question. i will just wait a few more days in case some better answer coles up, but right now I feel you can be the accepted answer. Feb 17, 2016 at 18:28
  • Very well done on the thoroughly documented answer, straight from the specs! :)
    – Wildcard
    Feb 18, 2016 at 9:10
  • 1
    @Wildcard, the behavior preceded the specs though.
    – jlliagre
    Feb 18, 2016 at 9:28
  • @jlliagre, unless you have a memoir from Bill Joy or perhaps the creator of ex (don't know his name), I think POSIX specs are as good as can be expected. ;) Closest to "original source" by this point, even though it's true they began as more or less descriptions of existing functionality.
    – Wildcard
    Feb 18, 2016 at 9:39
  • 3
    @Wildcard ex was co-written by Bill Joy and Chuck Alley ( web.cecs.pdx.edu/~kirkenda/joy84.html .) I do not question the POSIX specs and the fact current vi releases do follow it, I just state the behavior long predates it.
    – jlliagre
    Feb 18, 2016 at 10:01
33

This is the expected vi behavior.

Your file has an incomplete last line so strictly speaking (i.e. according to the POSIX standard), it is not a text file but a binary file.

vi which is a text file editor, not a binary one, gracefully fixes it when you save it.

This allows other text file tools like wc, sed and the likes to provide the expected output. Note that vi isn't silent about the issue:


$ printf "one\ntwo" >file     # Create a unterminated file
$ cat file                    # Note the missing newline before the prompt
one
two$ wc -l file               # wc ignores the incomplete last line
       1 file
$ sed '' file > file1
$ cat file1                   # so does a legacy sed
one
$ PATH=$(getconf PATH) sed  '' file
one                           # while a POSIX conformant sed warns you:
sed: Missing newline at end of file file.
two
$ vi file
one
two
~
~
~                             # vi tells you too about the issue
"file" [Incomplete last line] 2 lines, 7 characters

:w

"file" 2 lines, 8 characters  # and tells it writes two lines
                              # You'll even notice it writes one more
                              # character if you are a very shrewd observer :-)
:q
$ cat file                    # the file is now valid text
one
two
$ wc -l file                  # wc reports the expected number of lines
       2 file
$ sed '' file > file1         # sed works as expected
$ cat file1
one
two

Note, to get some clues about what vi version you are running, you can use the :ve command. It shows here I'm using a legacy SVR4 one here, definitely not vim:

:ve
Version SVR4.0, Solaris 2.5.0

Apparently, yours is stating:

:ve
Version 3.10

That likely means AIX vi is based on SVR3 source code.

In any case, this behavior, and the [Incomplete last line] warning message have been in the legacy Bill Joy's vi source code since at least 1979 and AFAIK, retained in all branches created from System V source code releases, from which proprietary Unix like AIX were built.

Chronologically speaking, this behavior is then not a consequence of POSIX conformance but more a consequence of Bill Joy's original decision to be helpful with users editing bogus text files, and then, a decade later, the POSIX committee decision to keep this tolerance.

If you use ed instead of vi, you'll notice that the former is more verbose about the issue, at least if your ed is from SVR3 or newer source branch:

$ ed file
'\n' appended
8
q

Note also that an empty file is a valid text file that happens to contain zero lines. As there is then no unterminated line to fix, vi doesn't append a newline when saving the file.

9
  • 1
    I believe you mistake vim for vi ;) legacy vi is very much less verbose than this... Feb 17, 2016 at 21:09
  • @OlivierDulac I am not confusing them. This test was done using the SVR4 legacy vi just like the OP does, although on a different Unix. This is not vim or another clone. Answer updated to clarify this.
    – jlliagre
    Feb 17, 2016 at 22:02
  • @OlivierDulac Hmm, I just noticed you are actually the OP. It seems AIX is using an older System V branch for its vi implementation. Possibly SVR3. Are you sure there is no [Incomplete last line] message when you open the file?
    – jlliagre
    Feb 17, 2016 at 22:06
  • 1
    It indeed says so when opening (easily missed), but what I don't like is that it silently changes this when writing !. Ie: printf "a" > foo ; vi foo displays "foo" [Last line is not complete] 1 line, 1 character. Then I just do: :w and it says: "foo" 1 line, 2 characters , which here is easy to spot, but sorely miss a "ADDING A NEWLINE" message (I rarely know exactly the number of bytes I had when opening, and after editing...)... I +1 you for all the research, however, and thank you for those hints. The behavior still (to me) lacks some warning (silently adding a char should be NOK) Feb 18, 2016 at 16:06
  • 1
    your answer is more complete than the top answer now, thanks to all your research and complements (ed behaviour, version info, etc). I still wait a bit but you took precendence now, I think. The posix citation of the top answer is a good thing though. Feb 18, 2016 at 16:22
1

Text that improperly lacks the ultimate newline run through a shell while loop results in the last line being silently discarded.

$ (echo transaction 1; echo -n transaction 2) \
  | while read line; do echo $line; done
transaction 1
$ 

Ensuring that there is an ultimate newline is the right and sane and proper default. The other option involves knowing to and having the time to audit all shell code that touches text lacking the ultimate newline, or to risk loosing the last line of the text.

0

I don't recall any other behaviour that a newline is added at the end of a file (using vi since the mid 80s).

The ~ indicates that a line on the screen that is not part of the text, not that the file doesn't end in a newline. ( You can get difficult to trace errors if you put a ~ on the last line of shell scripts ). If you load a short file with a newline at the end you will see the ~ yourself and disprove that your thought it indicates non-newline-ending text.

2
  • what surprises me is the addition of a newline... I expect vi to not add it silently, but it seems it does... I'm looking for an explanation of this attitude (the troubling fact is : I open foo2 (without trailing LF) and just :wq, it CHANGES its content... so it shows me something but saves another thing... weird, to say the least ^^ Feb 17, 2016 at 15:48
  • in its predecessor (ed) you would create lines and edit those, not by appending characters. I always thought about vi as a line oriented editor as well. But I understand your surprise.
    – Anthon
    Feb 17, 2016 at 16:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .