1

The Geostationary Operational Environmental Satellite (GOES)-R Product User Guide (PUG) from the National Oceanic and Atmospheric Administration (NOAA) contains the following rather wordy description of a plain text file (§4.3) (emphasis mine):

The Unix text file format is used in a small subset of the Level 1b and 2+ semi-static source data files. The Unix text file format, less the end-of-file character, is embedded in GRB metadata packets to store the XML-based netCDF Markup Language (NcML) representation of the netCDF file specifications, which includes the values for product metadata.

The Unix text file format is a sequence of lines (i.e., records), potentially variable in length, of electronic text. For the GOES-R ground system, the electronic text, newline, and end-of-file characters conform to the American Standard Code for Information Interchange (ASCII). At the end of each line is the newline character. At the end of file, there is an end-of-file character.

Is this an accurate description of the contents of a file? I thought that the end of file was a condition that the operating system or a library routine was returning when no more data can be read from a file (or other stream). Is this byte actually contained in the file?

7
  • I'd say they're completely wrong about an 'end-of-file' character. To start with, there is no EOF character. some apps use EOT (^D) to signal the end of the file (or input), but that's app-specific, far from universal or required. DOS/Windows uses (or used to use, i dunno any more) ^Z.
    – cas
    Aug 28 '19 at 11:27
  • @cas what about the file separator 0x1C if I'm not mistaken. They could of course also be eroniously naming the EOT as EOF as you point out. Aug 28 '19 at 22:15
  • 2
    Their definition is vacuously consistent, because they define "unix text file" as being terminated by a mysterious EOF character, but also specify that that character is omitted in the form used by them ("less the end-of-file character"). It's like defining "mouse" as "a rodent not wearing a hat".
    – mosvy
    Aug 28 '19 at 22:47
  • 4
    I suspect it's a legacy issue where NOAA or NASA had at some stage been using a non-Unix OS that uses EOF characters and it's possible that this sentence was accidentally copied from an earlier document. Aug 29 '19 at 10:03
4

The Unix text file format is a sequence of lines (i.e., records), potentially variable in length, of electronic text. At the end of each line is the newline character. At the end of file, there is an end-of-file character.

Is this an accurate description of the contents of a file?

Up to but excluding that last bolded part, yes. But I don't know of any Unixy systems that would use an end-of-file character, they all store the length of a file down to a byte, making such markers unnecessary.

Then again, it appears there have been systems that did use an end-of-file character. At least Wikipedia claims that:

The CP/M file system only recorded the lengths of files in multiples of 128-byte "records", so by convention a Control-Z character was used to mark the end of meaningful data if it ended in the middle of a record.

Having file lengths stored only up to a block would require some sort of custom to encode the end of the last line within the data stream. Any programs handling binary data would of course also have to deal with the more granular file sizes somehow. With binary files it might be easier to ignore the trailing "extra" bytes, though.

I think I've seen Control-Z used as an EOF marker on MS-DOS, but it wasn't necessary there either.

That quoted text seems to have a mistaken idea of text files in current systems. If we look at what the POSIX standard says, there's no mention of an end-of-file character or marker for text files, just that they contain no NUL bytes and consist of lines (ending in newlines).

See also: What's the last character in a file?

As for this part...

For the GOES-R ground system, [...] and end-of-file characters conform to the American Standard Code for Information Interchange (ASCII).

Like others have said in the comments, there's no character for end-of-file in ASCII, at least not with that name (*). Control-Z mentioned above is 26, or "substitute" (SUB), "used to indicate garbled or invalid characters". So, based on just that text, it would be hard to know what the EOF character would be, were it used.

(* There's "end of text" (ETX, code 3), "end of transmission" (EOT, code 4), "end of transmission block" (ETB, 23), "end of medium" (EOM, 25) and also "file separator" (FS, 28). Close, but not exact.)

I thought that the end of file was a condition that the operating system or a library routine was returning when no more data can be read from a file (or other stream).

That's what it is, indeed. The system call read() returns zero bytes (with no error) when the end of a file is reached, while some stdio functions (getchar()) have a return special value for it, unsurprisingly called EOF.

See also: Difference between EOT and EOF

2
  • IIRC, the POSIX definition of "text file" imposes a limit on the line length (LINE_MAX), which also has the unxpected consequence that shell scripts are not "text files", because a shell is supposed to accept lines of any length in scripts.
    – mosvy
    Aug 29 '19 at 15:10
  • @mosvy, yep, I ignored the length limitation on purpose here. But you are right, that difference does seem a bit surprising. FWIW, It's spelled out in the description of sh: "The input file shall be a text file, except that line lengths shall be unlimited."
    – ilkkachu
    Aug 29 '19 at 15:21
3

This looks to be something very specific to the file format they are discussing. As a general rule files don't NEED an EOF character. Non is added without a program explicitly writing one.

Checking an ASCII table I don't see an EOF character. They might be referring to an EOT or FS character, but that's not clear. https://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/ascii.html

It is however common in some file formats to out a marker at the end of the file. Particularly in simple file formats which are intended for communication. This protects against files being inadvertently truncated. If you know a file must end with a specific marker, and that marker only comes at the end, the. You can easily tell if you received the whole file or just par of it. As I read it, they are referring to this type of marker.

-1

The 'end of file' character they're referring to is probably a single newline occurring as the last character in the file. Most conventional text files on UNIX and UNIX-like systems end in such a way so that you can use the cat command (or something similar` to display the file contents and be certain that the next command prompt will be on it's own line.

Some poorly behaved applications actually fail to parse files correctly if they don't see this final newline. In this respect, it's a bit like the Unicode byte-order-mark in UTF-8 encoded text, it's not needed at all (actually, it's not even supposed to be there according to most standards), but some applications refuse to interpret things as UTF-8 without it.


From the perspective of the OS itself though, no such 'character' exists. The filesystem stores the correct size for the file, and when asked o read the file, the OS returns exactly that much data in total, so there's no point in even having such a concept, let alone having a character for it.

Some people confuse the EOT control code (^D) for this concept, as it's widely used on UNIX-like systems to signal the end of a stream of interactive input, but that's just a convention derived from the original usage (to signal the end of a transmission over some communication link). Note that this is significantly different from DOS systems, where ^Z was used to actually signal the end of a file both in interactive input and in actual files. The EOT control code doesn't actually show up in the data stream the application sees, it's interpreted by the terminal which signals an end-of-file condition to the application when it encounters ^D.

6
  • Hmm, interpreting EOT for this purpose seems risky for a binary file!
    – gerrit
    Aug 28 '19 at 21:18
  • 2
    This is misleading. 1) the standard is NOT referring to a trailing LF. 2) LF in *nix programs is usually a line terminator not separator. This makes processing simpler, and isn't realted to pretty "cat" output. Blank files therefore don't usually need a solo LF. Aug 29 '19 at 0:04
  • 3) There are two separate EOT byte sequences and both definitely do show up in a program's data stream. ASCII 0x04 is uncommon. "^D" is a keyboard combination translated by the terminal into a byte sequence often represented the same way. Programs reading directly from the TTY or a file will receive these bytes! It's the shell that strips them before passing to a subprocess. Aug 29 '19 at 0:05
  • @PhilipCouling And it's that interpretation of LF usage that leads to those misbehaving applications that have issues with files that don't have a LF on the last line. The assumption that the final line of a file must end with a newline of some sort is a recurring source of issues in software in general (not just software for UNIX-like systems). Most likely, the software in question here has some components that make that (incorrect) assumption. Aug 29 '19 at 14:30
  • 3
    "The 'end of file' character they're referring to is probably a single newline occurring as the last character in the file." -- doubtful, since the quoted text ends with "At the end of each line is the newline character. At the end of file, there is an end-of-file character." and that seems to imply that a) there is a newline at end of the last line (since it's each line) and b) the end-of-file is somehow different than a newline (since why else call it something else).
    – ilkkachu
    Aug 29 '19 at 14:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.