20

POSIX defines a text file as:

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

Source: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403

However, there are several things I find unclear:

  1. Must a text file be a regular file? In the above excerpt it does not explicitly say the file must be a regular file

  2. Can a file be considered a text file if contains one character and one character only (i.e., a single character that isn't terminated with a newline)? I know this question may sound nitpicky, but they use the word "characters" instead of "one or more characters". Others may disagree, but if they mean "one or more characters" I think they should explicitly say it

  3. In the above excerpt, it makes reference to "lines". I found four definitions with line in their name: "Empty Line", "Display Line", "Incomplete Line" and "Line". Am I supposed to infer that they mean "Line" because of their omission of "Empty", "Display" and "Incomplete"- or are all four of these definitions inclusive as being considered a line in the excerpt above?

All questions that come after this block of text depend on inferring that "characters" means "one or more characters":

  1. Can I safely infer that if a file is empty, it is not a text file because it does not contain one or more characters?

All questions that come after this block of text depend on inferring that in the above excerpt, a line is defined as a "Line", and that the other three definitions containing "Line" in their name should be excluded:

  1. Does the "zero" in "zero or more lines" mean that a file can still be considered a text file if it contains one or more characters that are not terminated with newline?

  2. Does "zero or more lines" mean that once a single "Line" (0 or more characters plus a terminating newline) comes into play, that it becomes illegal for the last line to be an "Incomplete Line" (one or more non-newline characters at the end of a file)?

  3. Does "none [no line] can exceed {LINE_MAX} bytes in length, including the newline character" mean that there a limitation to the number of characters allowed in any given "Line" in a text file (as an aside, the value of LINE_MAX on Ubuntu 18.04 and FreeBSD 11.1 is "2048")?

  • Good question, Harold ! Makes for a great discussion of terminology. Wish I could upvote the question extra times – Sergiy Kolodyazhnyy May 27 '18 at 4:59
21
  1. Must a text file be a regular file? In the above excerpt it does not explicitly say the file must be a regular file

    No; the excerpt even specifically notes standard input as a potential text file. Other standard utilities, such as make, specifically use the character special file /dev/null as a text file.

  2. Can a file be considered a text file if contains one character and one character only (i.e., a single character that isn't terminated with a newline)?

    That character must be a <newline>, or this isn't a line, and so the file it's in isn't a text file. A file containing exactly byte 0A is a single-line text file. An empty line is a valid line.

  3. In the above excerpt, it makes reference to "lines". I found four definitions with line in their name: "Empty Line", "Display Line", "Incomplete Line" and "Line". Am I supposed to infer that they mean "Line" because of their omission of "Empty", "Display" and "Incomplete"

    It's not really an inference, it's just what it says. The word "line" has been given a contextually-appropriate definition and so that's what it's talking about.

  4. Can I safely infer that if a file is empty, it is not a text file because it does not contain one or more characters?

    An empty file consists of zero (or more) lines and is thus a text file.

  5. Does the "zero" in "zero or more lines" mean that a file can still be considered a text file if it contains one or more characters that are not terminated with newline?

    No, these characters are not organised into lines.

  6. Does "zero or more lines" mean that once a single "Line" (0 or more characters plus a terminating newline) comes into play, that it becomes illegal for the last line to be an "Incomplete Line" (one or more non-newline characters at the end of a file)?

    It's not illegal, it's just not a text file. A utility requiring a text file to be given to it may behave adversely if given that file instead.

  7. Does "none [no line] can exceed {LINE_MAX} bytes in length, including the newline character" mean that there a limitation to the number of characters allowed in any given "Line" in a text file

    Yes.

This definition is just trying to set some bounds on what a text-based utility (for example, grep) will definitely accept — nothing more. They are also free to accept things more liberally, and quite often they do in practice. They are permitted to use a fixed-size buffer to process a line, to assume a newline appears before it's full, and so on. You may be reading too much into things.

  • 1
    Are you sure about point 2? The standard explicitly states "0 or more lines". So printf "a" > file would create a text file according to that definition. Your answer to 4 seems to be contradicting your answers to 2 and 5, as you suggest that touch file creates a text file while printf "a" > file does not. – terdon May 27 '18 at 11:31
  • 4
    @terdon: I don't see any contradiction in Michael's answer. Basically, he seems to be saying that a POSIX text file is any file whose contents matches the regexp (.{0,M}\n)* (implicitly anchored and both ends), where \n matches a newline and . matches any character that is not a newline, and M is a placeholder for the numeric value LINE_MAX-1. In particular, this implies that an empty file is a valid text file consisting of zero lines, but that any non-empty text file must end in a newline (since otherwise it would contain an incomplete line, and an incomplete line is not a line). – Ilmari Karonen May 27 '18 at 14:06
  • @Michael Homer Concerning the regular file thing, are there are other examples besides /dev/null? It's not really a text file since it contains one or more null characters. – Harold Fischer May 27 '18 at 18:20
  • 1
    @HaroldFischer /dev/null is an empty file. You’re thinking of /dev/zero. – Michael Homer May 27 '18 at 19:10
  • @HaroldFischer, no, /dev/null reads as empty, as in you get no data when you read it. I'm not sure it makes much sense to consider non-regular files here, since of many of them are dynamic in nature. That includes pipes, sockets, char devices, which are basically just transport interfaces to/from some other entity. They don't hold any static set of data, so it'd make more sense to consider the properties of the data that was transferred, instead of the properties of the file. – ilkkachu May 27 '18 at 19:21
7

As defined by POSIX:

Yes, a text file is (basically):

A file that contains characters organized into zero or more lines.

It would be useful to also include this definitions:

3.92 Character String

A contiguous sequence of characters terminated by and including the first null byte.

3.195 Incomplete Line

A sequence of one or more non- <newline> characters at the end of the file.

3.206 Line

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

3.243 Newline Character (<newline>)

A character that in the output stream indicates that printing should start at the beginning of the next line. It is the character designated by '\n' in the C language. It is unspecified whether this character is the exact sequence transmitted to an output device by the system to accomplish the movement to the next line.

3.247 NUL

A character with all bits set to zero.

Note that a "Text File" shall not contain NUL bytes.


So:

  1. Must a text file be a regular file?
    No, it does not need to be. A "text file" is defined in terms of what it contains when read. If a file contains "zero or more lines" it is a text file. Some file, like /dev/stdin, might contain a text file if read at one time and not on the next time it is read.
  2. Can a file be considered a text file if contains one character and one character only … ?
    No, that's an incomplete line (3.195).
    A text file shall have only non-"Incomplete Lines".
  3. Am I supposed to infer that they mean "Line" … ?
    Yes, you should.
  4. Can I safely infer that if a file is empty, it is not a text file … ?
    No, an empty file (zero characters) is a valid "text file".
    From above: …zero or more lines…. Zero lines (zero characters) is a valid "Text file".
  5. … considered a text file if it contains one or more characters that are not terminated with newline?
    No, an "Incomplete Line" in not (technically) a valid "line".
  6. Does the "zero" in "zero or more lines" mean that a file can still be considered a text file if it contains one or more characters that are not terminated with newline?
    No, an incomplete line is not a "Line". A text file shall not have incomplete lines.

  7. … there a limitation to the number of characters allowed in any given "Line" in a text file … ?
    Yes, no more than {LINE_MAX} bytes (as opposed to characters) shall be allowed in any given line of a valid "text file".
    The value of {LINE_MAX} is given in the file <limits.h>
    (also read Sensible line buffer size in C?):

    {LINE_MAX}
    Unless otherwise noted, the maximum length, in bytes, of a utility's input line (either standard input or another file), when the utility is described as processing text files. The length includes room for the trailing .
    Minimum Acceptable Value: {_POSIX2_LINE_MAX}

    For a GNU based system there is no set limit (except memory):

    Macro: int LINE_MAX
    The largest text line that the text-oriented POSIX.2 utilities can support. (If you are using the GNU versions of these utilities, then there is no actual limit except that imposed by the available virtual memory, but there is no way that the library can tell you this.)

    It seems to be defined in posix_lim.h to be 2048 (at least for 64 bit linux GNU systems):

    $ grep -ri 'POSIX2_LINE_MAX' /usr/include/ 
    
    /usr/include/x86_64-linux-gnu/bits/xopen_lim.h:#define NL_LANGMAX       _POSIX2_LINE_MAX
    /usr/include/x86_64-linux-gnu/bits/posix2_lim.h:#define _POSIX2_LINE_MAX                2048
    /usr/include/x86_64-linux-gnu/bits/posix2_lim.h:#define LINE_MAX                _POSIX2_LINE_MAX
    

    It may, also, be found using the POSIX utility getconf:

    $ getconf LINE_MAX
    2048
    

Related: Why should text files end with a newline?

locked by terdon Oct 12 '18 at 16:21

This post has been locked due to the high amount of off-topic comments generated. A link to a chat room will be posted in the comments below if the conversation was moved to chat.

Read more about locked posts here.

  • 2
    This answer is mostly correct, but the correct answer to “must a text file be a regular file” is no. Any kind of file can be a text file, it's a matter of content, the file type is irrelevant. The file utility only reports the file type for special files, but that's just how the utility works, use file - <… or (Linux) file -s … to see its heuristics on the file content for a special file. A special file can have different contents each time you open it, so it may or be a text file each time. /dev/null is always a text file because its content is always a text file. – Gilles May 27 '18 at 10:11
  • 1
    Rather than using grep on files, you can use getconf to get system conf values e.g. getconf LINE_MAX, which by the way returns 2048 (bytes) on my system (Ubuntu 16.04). – heemayl May 27 '18 at 16:07
  • I wanted to find the file where the variable was defined, thus grep was necessary, and did the job (quite quickly). But yes, getconf allows to read the present value of config. – Isaac May 28 '18 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.