0

This question already has an answer here:

The Geostationary Operational Environmental Satellite (GOES)-R Product User Guide (PUG) from the National Oceanic and Atmospheric Administration (NOAA) contains the following rather wordy description of a plain text file (§4.3) (emphasis mine):

The Unix text file format is used in a small subset of the Level 1b and 2+ semi-static source data files. The Unix text file format, less the end-of-file character, is embedded in GRB metadata packets to store the XML-based netCDF Markup Language (NcML) representation of the netCDF file specifications, which includes the values for product metadata.

The Unix text file format is a sequence of lines (i.e., records), potentially variable in length, of electronic text. For the GOES-R ground system, the electronic text, newline, and end-of-file characters conform to the American Standard Code for Information Interchange (ASCII). At the end of each line is the newline character. At the end of file, there is an end-of-file character.

Is this an accurate description of the contents of a file? I thought that the end of file was a condition that the operating system or a library routine was returning when no more data can be read from a file (or other stream). Is this byte actually contained in the file?

marked as duplicate by muru, Anthony G - justice for Monica, Fabby, Kusalananda Sep 2 at 11:49

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • I'd say they're completely wrong about an 'end-of-file' character. To start with, there is no EOF character. some apps use EOT (^D) to signal the end of the file (or input), but that's app-specific, far from universal or required. DOS/Windows uses (or used to use, i dunno any more) ^Z. – cas Aug 28 at 11:27
  • @cas what about the file separator 0x1C if I'm not mistaken. They could of course also be eroniously naming the EOT as EOF as you point out. – Philip Couling Aug 28 at 22:15
  • 2
    Their definition is vacuously consistent, because they define "unix text file" as being terminated by a mysterious EOF character, but also specify that that character is omitted in the form used by them ("less the end-of-file character"). It's like defining "mouse" as "a rodent not wearing a hat". – mosvy Aug 28 at 22:47
  • 4
    I suspect it's a legacy issue where NOAA or NASA had at some stage been using a non-Unix OS that uses EOF characters and it's possible that this sentence was accidentally copied from an earlier document. – Anthony G - justice for Monica Aug 29 at 10:03
4

The Unix text file format is a sequence of lines (i.e., records), potentially variable in length, of electronic text. At the end of each line is the newline character. At the end of file, there is an end-of-file character.

Is this an accurate description of the contents of a file?

Up to but excluding that last bolded part, yes. But I don't know of any Unixy systems that would use an end-of-file character, they all store the length of a file down to a byte, making such markers unnecessary.

Then again, it appears there have been systems that did use an end-of-file character. At least Wikipedia claims that:

The CP/M file system only recorded the lengths of files in multiples of 128-byte "records", so by convention a Control-Z character was used to mark the end of meaningful data if it ended in the middle of a record.

Having file lengths stored only up to a block would require some sort of custom to encode the end of the last line within the data stream. Any programs handling binary data would of course also have to deal with the more granular file sizes somehow. With binary files it might be easier to ignore the trailing "extra" bytes, though.

I think I've seen Control-Z used as an EOF marker on MS-DOS, but it wasn't necessary there either.

That quoted text seems to have a mistaken idea of text files in current systems. If we look at what the POSIX standard says, there's no mention of an end-of-file character or marker for text files, just that they contain no NUL bytes and consist of lines (ending in newlines).

See also: What's the last character in a file?

As for this part...

For the GOES-R ground system, [...] and end-of-file characters conform to the American Standard Code for Information Interchange (ASCII).

Like others have said in the comments, there's no character for end-of-file in ASCII, at least not with that name (*). Control-Z mentioned above is 26, or "substitute" (SUB), "used to indicate garbled or invalid characters". So, based on just that text, it would be hard to know what the EOF character would be, were it used.

(* There's "end of text" (ETX, code 3), "end of transmission" (EOT, code 4), "end of transmission block" (ETB, 23), "end of medium" (EOM, 25) and also "file separator" (FS, 28). Close, but not exact.)

I thought that the end of file was a condition that the operating system or a library routine was returning when no more data can be read from a file (or other stream).

That's what it is, indeed. The system call read() returns zero bytes (with no error) when the end of a file is reached, while some stdio functions (getchar()) have a return special value for it, unsurprisingly called EOF.

See also: Difference between EOT and EOF

  • IIRC, the POSIX definition of "text file" imposes a limit on the line length (LINE_MAX), which also has the unxpected consequence that shell scripts are not "text files", because a shell is supposed to accept lines of any length in scripts. – mosvy Aug 29 at 15:10
  • @mosvy, yep, I ignored the length limitation on purpose here. But you are right, that difference does seem a bit surprising. FWIW, It's spelled out in the description of sh: "The input file shall be a text file, except that line lengths shall be unlimited." – ilkkachu Aug 29 at 15:21
3

This looks to be something very specific to the file format they are discussing. As a general rule files don't NEED an EOF character. Non is added without a program explicitly writing one.

Checking an ASCII table I don't see an EOF character. They might be referring to an EOT or FS character, but that's not clear. https://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/ascii.html

It is however common in some file formats to out a marker at the end of the file. Particularly in simple file formats which are intended for communication. This protects against files being inadvertently truncated. If you know a file must end with a specific marker, and that marker only comes at the end, the. You can easily tell if you received the whole file or just par of it. As I read it, they are referring to this type of marker.

-1

The 'end of file' character they're referring to is probably a single newline occurring as the last character in the file. Most conventional text files on UNIX and UNIX-like systems end in such a way so that you can use the cat command (or something similar` to display the file contents and be certain that the next command prompt will be on it's own line.

Some poorly behaved applications actually fail to parse files correctly if they don't see this final newline. In this respect, it's a bit like the Unicode byte-order-mark in UTF-8 encoded text, it's not needed at all (actually, it's not even supposed to be there according to most standards), but some applications refuse to interpret things as UTF-8 without it.


From the perspective of the OS itself though, no such 'character' exists. The filesystem stores the correct size for the file, and when asked o read the file, the OS returns exactly that much data in total, so there's no point in even having such a concept, let alone having a character for it.

Some people confuse the EOT control code (^D) for this concept, as it's widely used on UNIX-like systems to signal the end of a stream of interactive input, but that's just a convention derived from the original usage (to signal the end of a transmission over some communication link). Note that this is significantly different from DOS systems, where ^Z was used to actually signal the end of a file both in interactive input and in actual files. The EOT control code doesn't actually show up in the data stream the application sees, it's interpreted by the terminal which signals an end-of-file condition to the application when it encounters ^D.

  • Hmm, interpreting EOT for this purpose seems risky for a binary file! – gerrit Aug 28 at 21:18
  • 2
    This is misleading. 1) the standard is NOT referring to a trailing LF. 2) LF in *nix programs is usually a line terminator not separator. This makes processing simpler, and isn't realted to pretty "cat" output. Blank files therefore don't usually need a solo LF. – Philip Couling Aug 29 at 0:04
  • 3) There are two separate EOT byte sequences and both definitely do show up in a program's data stream. ASCII 0x04 is uncommon. "^D" is a keyboard combination translated by the terminal into a byte sequence often represented the same way. Programs reading directly from the TTY or a file will receive these bytes! It's the shell that strips them before passing to a subprocess. – Philip Couling Aug 29 at 0:05
  • @PhilipCouling And it's that interpretation of LF usage that leads to those misbehaving applications that have issues with files that don't have a LF on the last line. The assumption that the final line of a file must end with a newline of some sort is a recurring source of issues in software in general (not just software for UNIX-like systems). Most likely, the software in question here has some components that make that (incorrect) assumption. – Austin Hemmelgarn Aug 29 at 14:30
  • 3
    "The 'end of file' character they're referring to is probably a single newline occurring as the last character in the file." -- doubtful, since the quoted text ends with "At the end of each line is the newline character. At the end of file, there is an end-of-file character." and that seems to imply that a) there is a newline at end of the last line (since it's each line) and b) the end-of-file is somehow different than a newline (since why else call it something else). – ilkkachu Aug 29 at 14:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.