Essentially, I am looking for an encoding-type agnostic version of the column -t command in linux.

I need to convert delimited files to flat and then write to an outfile. The specific delimiter varies (, | \t).

Sometimes, but not always, the fields contain text characters with diacriticals. Since column -t is only compatible with UTF-8 encoding type, it chokes on these "funny" characters:

I tried using commands like file -bi to determine encoding type so I could recode to UTF-8, but they only provide a "best guess" (which fails when the diacriticals start at row 100,000.)

So far I have come across the following:

This script in awk converts delimited to fixed length columns, but I need variable column widths:

$ awk -F@ '{for(i=1;i<=NF;i++){printf "%-20s", $i};printf "\n"}' input.csv

This script in perl finds max length for each column and sets each column to max length, but I also need the number of columns to be variable:

perl -F'\t' -lane '
    open I, "< inputfile";
    for (<I>) {
        @x = split /\t/;
        (length ($x[0]) > $f0) && ($f0 = length ($x[0]) + 1);
        (length ($x[1]) > $f1) && ($f1 = length ($x[1]) + 1);
        (length ($x[2]) > $f2) && ($f2 = length ($x[2]) + 1);
    close I;
print "$F[0]" . " " x ($f0 - length ($F[0])), "$F[1]" . " " x ($f1 - length     ($F[1])), "$F[2]" . " " x ($f2 - length ($F[2]));
' inputfile

Does anyone have a script that:

  • Is executable in the shell

  • Does what column -t does (deals with varying column widths, deals with varying numbers of columns, allows you to specify your delimiter)

  • AND is encoding-type agnostic?

closed as unclear what you're asking by Gilles, Stéphane Chazelas, Scott, Anthon, Stephen Kitt Dec 20 '15 at 12:59

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Can you provide some example of input that does not work for you? column should honour current locale (LC_ALL), but I didn't try it yet. – Jakuje Dec 18 '15 at 19:27
  • 1
    What do you mean by “encoding-type agnostic”? Since the task requires treating the input as a stream of characters, not a stream of bytes, it matters what the encoding of the input is. In what way is column not suitable? What makes you (incorrectly) believe that “column -t is only compatible with UTF-8 encoding”? – Gilles Dec 19 '15 at 22:11