4

Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, had this to say at the time [McIlroy78]:

(ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

Stringently columnar data sounds good to me, so I probably do not understand what he meant.

What does it mean and why is it bad?

5

I assume he meant aligned columns, not columnar data in general. That's how I would understand the stringently anyway. For example:

Bad:

1    200 3
100  3   400

Good:

1 200 3
100 3 400

In other words, make files that are easy for computers to read, not for humans. Adding spaces to align things makes them pretty and easier for you and me to understand but can confuse programs that need to parse them.

For example, if I were to use cut to get the second field of each of the above examples, it would fail on the first:

$ cut -d' ' -f 2 bad


$ cut -d' ' -f 2 good 
200
3

Because of the extra spaces, the 2nd field of the bad file is a space. However, it works as expected in the good file.

  • 3
    He means fixed-width records. These were widely used in the days of Fortran IV and punched cards. Their evil ways still live on with AIX. ;) – lcd047 Jun 25 '15 at 17:45
  • @lcd047 that's what I describe isn't it? Or are fixed-width records something else? – terdon Jun 25 '15 at 17:49
  • 4
    No, it's more like a wall of numbers, where columns 1-4 mean X, column 5 means Y, columns 6-12 is Z etc. No separators. Processing these is fine, but data entry is a pain in the rear. Remember that back then data entry wasn't interactive. Trust me, you'd run screaming the first time you'd have to deal with that. :) – lcd047 Jun 25 '15 at 17:54
0

Stringently Columnar

fixed with and ordered columns vs dynamic width and headers

Example

Fixed width

name (7 bytes) [space] age (2 bytes) [space] height (5 bytes)

alice   10 150cm 
bob     20 160cm
carol   30 170cm

if I now need to add Alexander age 40 height 180cm

alice   10 150cm 
bob     20 160cm
carol   30 170cm
Alexa~1 40 180cm

it doesn't fit in the stringently columnar structure you run out of space if you use a more flexible columnar structure

Space Separated Values

this pain can be avoided with a space separated values

alice 10 150cm 
bob 20 160cm
carol 30 170cm
Alexander 40 180cm

Space(s) Separated Values

or a format that divided columns with 1 or more space

alice     10 150cm 
bob       20 160cm
carol     30 170cm
Alexander 40 180cm

since the number of spaces doesn't matter we may format for human readability or for file density as suits our needs even better still in the eyes of the Unix philosophy is a format that doesn't force the order or presence of columns if the first row is the names of the columns and then the data space(s) delimited file

Name      Age Height
alice     10  150cm 
bob       20  160cm
carol     30  170cm
Alexander 40  180cm

with

Age Height Name      
10  150cm  alice
20  160cm  bob
30  170cm  carol
40  180cm  Alexander

just as valid as the first If we have extra columns we don't care about the headers give us more flexibility

Age Height t_shirt_size Name      
10  150cm  S            alice
20  160cm  M            bob
30  170cm  L            carol
40  180cm  XL           Alexander

we can still read out the Name, Age, and Height and just ignore the columns that have headers that we don't care about. still loads to the object

[{"Name":"alice",    "Age":10, "Height":"150cm"},
 {"Name":"bob",      "Age":20, "Height":"160cm"},
 {"Name":"carol",    "Age":30, "Height":"170cm"},
 {"Name":"Alexander","Age":40, "Height":"180cm"}]

These rules lead to a file format like CSV https://tools.ietf.org/html/rfc4180

Age,Height,t shirt size,Name      
10,150 cm,S,alice
20,160 cm,M,bob
30,170 cm,L,carol
40,180 cm,XL,Alexander

by avoiding stringently columnar data we make it more likely that the as yet unknown, program we expect the output to become the input to can work with our unmodified program.

If my program puts out data in CSV it can be read and graphed by countless programs I don't even know exist.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.