Why the output of "column" is misaligned with a ANSI colored input?

Question

I'm working on a commandline that retrieve some data (curl), extract the relevant fields (awk) and format it (column).

It work nice, although it's very ugly (but all my script begin from a "too long and ugly" one-liner) but when I try to have some colour column goes wrong.

This is the plain (reduced) version that work:

curl "http://webservices.rm.ingv.it/fdsnws/event/1/query?lat=42.35&lon=13.4&maxradius=5.0&starttime=2016-01-01T00:00:00&endtime=2016-12-31T23:59:59&minmag=5&format=text&orderby=time-asc" 2>/dev/null  \
 | awk 'BEGIN { FS= "|"; OFS= "|" }  {print  $1, $2, $5, $10, $11, $13}' \
 | column -t -s '|'

Now, I want to underline some fields, and then add some ANSI escape code in awk:

curl "http://webservices.rm.ingv.it/fdsnws/event/1/query?lat=42.35&lon=13.4&maxradius=5.0&starttime=2016-01-01T00:00:00&endtime=2016-12-31T23:59:59&minmag=5&format=text&orderby=time-asc" 2>/dev/null  \
 | awk 'BEGIN { FS= "|" ; OFS= "|" }  \
     $13~/Rieti/||/Perugia/ {$13="\033[1;31m"$13"\033[0m"} \
     $11~/[0-9]+/ && $11 > 5.8 {$11="\033[1;33m"$11"\033[0m"} 
     {print  $1, $2, $5, $10, $11, $13 }' \
 | column -t -s '|'

Now, the alignment is wrong (see the picture).

Why? And how can I fix it?

UPDATE

I already saw the question Issue with column command and color escape codes but does not solve my problem because his answers are applied and work in the case of a fully colored line.

In my case I can't apply or adapt the answers (or I'm not able to) because:

1. The problem is circumscribed to the case where the column $11 is colored, regardless of the subsequent column.
2. I can't see a good or elegant way to add color code after column.
   If I send column's output to awk for the test I don't know how to instruct awk to separate the fields correctly (if the fields were separated by more space I could use a regex but in some cases the separation is by a single space, and awk would not know how to recognize spaces between words and spaces as Fields Separators).

The only thing I can see is that if I move the reset color code from the assignment to the print block the first row was better spaced, like the plain output version (see below, the \033[0m underlined in second commandline):

Then, how can we fix it? There is another way, more elegant, to colorize as I did?

(I know, I can do it better with some lines of perl, but I'm curious about this problem)

Answer 1

There might be three simple ways to fix this:

1. always use escape sequences in those columns to maintain equal length
2. put the escapes in their own columns (4 extra columns), though this adds extra whitespace in the output
3. format after column, as you suggested

Some other considerations can be found here: A shell tool to “tablify” input data containing ANSI escape codes .

For the first option, instead of using just \033[1;31m for red use \033[31;1m for red, and \033[31;0m for "not red" or plain — the 0 code undoes any preceding code, even those in the same sequence. Then all the columns have the same length of escape codes.

BEGIN { FS=OFS="|" }  
function colour(ss,cc)    { return "\033[" cc ";1m" ss "\033[0m"; }
function notcolour(ss,cc) { return "\033[" cc ";0m" ss "\033[0m"; }
{  
  if ($13~/(RI|PG)/)    { $13=colour($13,31)    } 
                   else { $13=notcolour($13,31) }
  if (($11+0) > 5.8)    { $11=colour($11,33)    }
                   else { $11=notcolour($11,33) }
  print $1, $2, $5, $10, $11, $13 
}

(There are a number of minor simplifications and corrections applied in the above also, including one to match changes in source data.)

The problem with this approach is that it depends on your column and libc. (My column from util-linux-2.23.2) doesn't check the return code from wcswidth() which is -1 when non-printables are found, instead of the actual width; this really messes up the table formatting. The latest version from util-linux-2.30.1 uses a new libsmartcols which solves this problem, but it does that by replacing non-printables by a \x hex-encoded version — so you loose raw escapes altogether :/ That you can fix with the inelegant:

curl ... | awk ... | column -t -s '|' | while read -r line; do printf "$line\n"; done

where printf interprets the escapes. You could replace \033 with \\x1b in your own code for the same effect. I'm not sure if you're using Linux though.)

For the third option you will need a column that supports -o for setting the output separator, the default is two spaces. Set it to "|", then you can use this:

curl ... | column -t -s "|" -o "|" | awk '
BEGIN { FS="|" }  
function colour(ss,cc) { return sprintf("\033[%i;1m%s\033[0m",cc,ss) }
{  
  if ($13~/(RI|PG)/) { $13=colour($13,31) } 
  if (($11+0) > 5.8) { $11=colour($11,33) }
  print $1, $2, $5, $10, $11, $13 
}'

The trick here is we use column with pipe-delimited input and output, it fixes the widths and we can safely process that with awk, preserving the all important spaces. If your column doesn't support -o you can fake it with:

curl ... | sed -e 's/|/^|/g' | column -t -s^ | awk ...

This doubles up the separator to "^|", column uses ^ and awk uses |. This makes the assumption that ^ does not appear in the data of course. A hard tab may work instead.

I think you know the "why" now, but to be clear:

• column might count octets (or characters) naively with strlen()/wcslen(), this won't match the terminal rendered length
• column might count length using isprint(), also incorrect with terminal escapes
• column might give up (as mine does) on any column when non-printables are encountered

While stripping colour code sequences is a reasonably straightforward problem, there's no robust way around this without having a chunk of ANSI terminal emulator within column.

Answer 2

This snippet, modified from the OP:

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }

pl " Results, highlight:"
# Original code from post:
# curl "http://webservices.rm.ingv.it/fdsnws/event/1/query?lat=42.35&lon=13.4&maxradius=5.0&starttime=2016-01-01T00:00:00&endtime=2016-12-31T23:59:59&minmag=5&format=text&orderby=time-asc" 2>/dev/null  \
#  | awk 'BEGIN { FS= "|"; OFS= "|" }  {print  $1, $2, $5, $10, $11, $13}' \
#   | column -t -s '|'

# Codes my-highlight, my-hilite:
# https://unix.stackexchange.com/questions/46562/how-do-you-colorize-only-some-keywords-for-a-bash-script

SITE="http://webservices.rm.ingv.it/fdsnws/event/1/query?lat=42.35&lon=13.4&maxradius=5.0&starttime=2016-01-01T00:00:00&endtime=2016-12-31T23:59:59&minmag=5&format=text&orderby=time-asc"

curl "$SITE" > data1

awk 'BEGIN { FS= "|"; OFS= "|" }  {print  $1, $5, $10, $11, $13}' data1 |
tee f1 |
column -t -s '|' |
my-highlight -r "Norcia"

pl " Results, hilite:"
awk 'BEGIN { FS= "|"; OFS= "|" }  {print  $1, $5, $10, $11, $13}' data1 |
tee f2 |
column -t -s '|' |
my-hilite -f blue "Norcia"

produces:

-----
 Results, highlight:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   788    0   788    0     0   2566      0 --:--:-- --:--:-- --:--:--  2575
#EventID  Depth/Km  MagType  Magnitude  EventLocationName
7073641   8.1       Mw       6.0        1 km W Accumoli (RI)
7076161   8.0       Mw       5.3        5 km E Norcia (PG)
8663031   8.7       Mw       5.4        3 km SW Castelsantangelo sul Nera (MC)
8669321   7.5       Mw       5.9        3 km NW Castelsantangelo sul Nera (MC)
8788671   481.4     ML       5.8        Tirreno Meridionale (MARE)
8863681   9.2       Mw       6.5        5 km NE Norcia (PG)

-----
 Results, hilite:
#EventID  Depth/Km  MagType  Magnitude  EventLocationName
7073641   8.1       Mw       6.0        1 km W Accumoli (RI)
7076161   8.0       Mw       5.3        5 km E Norcia (PG)
8663031   8.7       Mw       5.4        3 km SW Castelsantangelo sul Nera (MC)
8669321   7.5       Mw       5.9        3 km NW Castelsantangelo sul Nera (MC)
8788671   481.4     ML       5.8        Tirreno Meridionale (MARE)
8863681   9.2       Mw       6.5        5 km NE Norcia (PG)

This uses the method 3 of mr.spuratic, colorize after columnize.

I eliminated one field to make it easier to read, then applied 2 scripts (renamed with a prefix my- here) from thread How do you colorize only some keywords for a bash script? -- both worked by coloring the string Norcia after running column. ( If anyone can point me to methods for showing colors in posts, I'd be grateful.)

I assumed that these strings would not appear elsewhere in the output, so specific fields are not an issue, the entire line is examined for a match. If that assumption is not the case, then this solution is of little value, other than calling attention to the scripts hilite and highlight.

This was done on a system like:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30

If you use script highlight, you'll need spc (in Debian package supercat); here are some details about that:

spc     colorize and print to standard output (man)
Path    : /usr/bin/spc
Package : supercat
Home    : http://supercat.nosredna.net/
Version : 2008
Type    : ELF64-bitLSBexecutable,x86-64,version1(SYSV ...)
Help    : probably available with -h,--help

Best wishes ... cheers, drl

Answer 3

Since everyone already explained why column is displaying misaligned output, i'll not explain it again.

Instead i'll show another solution.

For anyone that is interested to having a column-like output but with color codes and ANSI escape sequences, i made a version that has the same syntax as column (for the supported flags) and it renders the output as expected.

I found various solutions online, but they all used Bash for looping, which is very slow compared to Perl, which i used.

You can find the script here.

It can be executed directly (./column_ansi.sh [--options]) or sourced (source ./column_ansi.sh, in which case the command will be available as column_ansi)