Why does the column command not align columns properly?

Question

In another question I asked on how to hide all .pyc files when using ls, Ignacio suggested the following:

This, as I mentioned above, doesn't work precisely, since the output is occasionally misaligned:

ceasarbautista@hse140:~/Desktop/Statistics/statistics/markov$ ls
README          __init__.pyc        markov.py       matrix2graph.pyc    pathfinder.pyc      priority_dict.pyc   spanning.py
__init__.py     graph.py        matrix2graph.py     pathfinder.py       priority_dict.py    space.py        vector.py
ceasarbautista@hse140:~/Desktop/Statistics/statistics/markov$ ls | grep -v '\.pyc$' | column
README      graph.py        matrix2graph.py priority_dict.py    spanning.py
__init__.py markov.py   pathfinder.py   space.py        vector.py

While I got a satisfactory answer, I'm curious: why does column do this (and can it be fixed to print correctly)?

Answer 1

If ls output is garbled, it indicates that there are non-printable characters within one or more filenames. I therefore recommend always to use the option -q with ls.

Any non-printable characters will then be printed out as a question mark.

So please try:

 ls -Cq

Answer 2

When called without any options, column makes each delimited strings align to the nearest TAB-STOP COLUMN. In a terminal, that is typically on every 8th character column.. Have a look at this example:

Create a file (ztxt) containing some tab-character \t delimited strings, spread over 3 lines which end with newline-character \n:

aaa1\taaaaaaaaaaaaaa2\taaaaaaaaaaaaaaaaaaa3\taaa4
bbbbbbb1\tbbb2\tbbb3
ccc1\tccc2

---

Output of: column ztxt - perfectly aligned to the nearest TAB-STOP

aaa1    aaaaaaaaaaaaaa2 aaaaaaaaaaaaaaaaaaa3    aaa4
bbbbbb1 bbb2    bbb3
ccc1    ccc2
|       |       |       |       |       |       |
|-------|-------|-------|-------|-------|-------|

---

To align the LSH of each successive field to the one above it, you need to use the -t option, eg. column -t ztxt

aaa1     aaaaaaaaaaaaaa2  aaaaaaaaaaaaaaaaaaa3  aaa4
bbbbbb1  bbb2             bbb3
ccc1     ccc2

---

If all your data is in one long stream without any newline breaks, you can uss a filter introduce them; eg one every 4 fields. sed can do it with this command.

sed -re 's/(([^\t]*\t){3}[^\t]*)\t/\1\n/g' 

By default, the column command will merge multiple adjacent delimiters into a single delimiter.. To cater for this in the sed filter, it also needs:

sed -re 's/\t+/\t/g;' 

So the command to split up a continuous stream of tab-delimited strings, as every 4th string is:

<ztxt sed -re 's/\t+/\t/g;s/(([^\t]*\t){3}[^\t]*)\t/\1\n/g' | column -t  

The output of such a contionuous input stream is (using the original sample input, but modified by replacing the original newlines with tab-characters--it must still keep its trailing \n):

aaa1      aaaaaaaaaaaaaa2  aaaaaaaaaaaaaaaaaaa3  aaa4
bbbbbbb1  bbb2             bbb3                  ccc1
ccc2

Answer 3

The problem with column is that it splits on \s+, I don't see how to make it split on a single tab. I wrote this much more flexible script to solve this problem, that anybody is free to use:

http://itmat.greg.s3.amazonaws.com/display.pl.gz

Here's the full usage:

  --------------------------------------------------------------------------
 | This script displays tab delimited text in properly aligned columns.     |
 |                                                                          |
 | Usage: perl display.pl  [num lines] [options]                      |
 |                                                                          |
 | Where  is tab delimited.                                           |
 |                                                                          |
 | If  is followed by a positive integer N, only the first N lines    |
 | will be returned.                                                        |
 |                                                                          |
 | If lines are long, this script works bests if piped to "less -S".        |
 |                                                                          |
 | [options]                                                                |
 |                                                                          |
 | -noheader :  A header line is assumed by default.                        |
 |                                                                          |
 | -nodots   :  By default if a column is wide and an entry in that column  |
 |              is short, dots are put after it to help line things up.     |
 |              Use this option to just print spaces, no dots.              |
 |                                                                          |
 | -dotcols  :  Use this to specify specific columns to have dots     |
 |                    (see the -nodots option).   must be a comma     |
 |                    separated list of positive integers w/o whitespace.   |
 |                                                                          |
 | -cols  :  Use this to specify a subset of columns to output.       |
 |                  is a comma separated list of positive integers    |
 |                 and/or spans of positive integers.  Examples of valid    |
 |                 lists are: 4,6,12 or 4-10 or 1-4,12,15,4-7.  Columns     |
 |                 can be repeated and don't have to be in numerical order. |
 |                                                                          |
 | NOTE: This script was hacked from a much more complicated script, so     |
 | the code is full of all kinds of irrelevant stuff.                       |
  --------------------------------------------------------------------------

Answer 4

To show non-printable characters (as C escape codes whenever possible) in your ls output you also can use the -b option to the ls command:

ls -Cb | grep -v '\.pyc$' | column -t