3

One can obtain the value of of LINE_MAX by running getconf LINE_MAX. On FreeBSD 12 and Ubuntu 18.04 (and I suspect many others) this value is 2048 bytes.

In POSIX world, there are many shell utilties that are specifically designed to work with text files. A file that has any one line that exceeds LINE_MAX bytes in length (newline inclusive) is not a text file (as defined by POSIX anyway). A shell utility designed to take text files as input is not required to be able to properly handle this kind of file.

In practice, I seem unable to find a utility that is unable to deal with text files with really long lines. It is my understanding that GNU utilities do not have any sort of real limit to how long a line in a text file can be (I'm not sure about FreeBSD).

Is anyone aware of a shell utility on any modern or semi-modern operating system that chokes on lines that exceed LINE_MAX bytes in length? As an aside, are programming languages like Perl and Python affected by the value of LINE_MAX?

3

There’s at least one example of a shell utility which sticks to LINE_MAX, at least in some circumstances, on this site, OpenBSD cut; it doesn’t choke on lines longer than those, but some of its capabilities are limited to the first LINE_MAX characters of a line.

In addition to utilities which outright refuse to work with longer lines, there is a common pattern which will cause potentially unwanted behaviour: input is often processed using

fgets(line, sizeof line, fp);

where line is a LINE_MAX-sized character array (see for example pam_localuser). If line is processed every time fgets returns non-NULL, then the input will be processed with lines split at LINE_MAX if they’re too long. Thus in pam_localuser, a line in /etc/passwd longer than LINE_MAX will be handled as two lines.

The programming languages and parser generators I looked at don’t have a reference to LINE_MAX (other than pass-through references, to make the value available to programs written in those languages). This includes GCC, Perl, Python, and Lua.

  • You are mistaken, there are implementations that work fine for infinite line length even when using fgets(). The real problem from using fgets() is that it cannot support lines that contain NUL bytes as it does not return the line length. – schily Nov 29 '19 at 14:12
  • @schily I never said there weren’t. I said there are problems if the line is processed every time it’s retrieved using fgets, which is exactly what pam_localuser does. – Stephen Kitt Nov 29 '19 at 15:00
  • The GNU reimplementation may behave this way. Solaris started in the mid 1990s to fill a NUL byte into the last position of the buffer before calling fgets() and reconstructs longer lines in case that the NUL has been replaced by a non NL character. – schily Nov 29 '19 at 15:04
  • @schily and how does it deal with lines which are longer than the provided buffer? – Stephen Kitt Nov 29 '19 at 15:06
  • Simply call fgets() repeatedly until the last position in the buffer is not overwritten, grow the line buffer between calls and append the following reads to the previous content of the buffer. – schily Nov 29 '19 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.