53

If I execute the following simple script:

#!/bin/bash
printf "%-20s %s\n" "Früchte und Gemüse"   "foo"
printf "%-20s %s\n" "Milchprodukte"        "bar"
printf "%-20s %s\n" "12345678901234567890" "baz"

It prints:

Früchte und Gemüse foo
Milchprodukte        bar
12345678901234567890 baz

that is, text with umlauts (such as ü) is "shrunk" by one character per umlaut.

Certainly, I have some wrong setting somewhere, but I am not able to figure out which one that could be.

This occurs if the file's encoding is UTF-8.

If I change its encoding to latin-1, the alignment is correct, but the umlauts are rendered wrong:

Fr�chte und Gem�se   foo
Milchprodukte        bar
12345678901234567890 baz
  • 14
    You expect printf to be aware of UTF-8 and other multibyte charsets? – frostschutz Mar 9 '17 at 11:47
  • 16
    Looks like it's counting bytes rather than characters; see echo Früchte und Gemüse | wc -c -m for the difference. – Stephen Kitt Mar 9 '17 at 11:47
  • 7
    @frostschutz Zsh's printf is. – Stephen Kitt Mar 9 '17 at 11:48
  • 10
    Yes, I do expect printf do be aware of (at least) UTF-8. – René Nyffenegger Mar 9 '17 at 11:49
  • 12
    Well, it's not. Tough luck. ;-) – frostschutz Mar 9 '17 at 11:49
87

POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) and bash mailing lists).

The printf builtin of bash and most other POSIX shells honour that.

zsh ignores that silly requirement (even in sh emulation) so printf works as you'd expect there. Same for the printf builtin of fish (not a POSIX-like shell).

The ü character (U+00FC), when encoded in UTF-8 is made of two bytes (0xc3 and 0xbc), which explains the discrepancy.

$ printf %s 'Früchte und Gemüse' | wc -mcL
    18      20      18

That string is made of 18 characters, is 18 columns wide (-L being a GNU wc extension to report the display width of the widest line in the input) but is encoded on 20 bytes.

In zsh or fish, the text would be aligned correctly.

Now, there are also characters that have 0-width (like combining characters such as U+0308, the combining diaresis) or have double-width like in many Asiatic scripts (not to mention control characters like Tab) and even zsh wouldn't align those properly.

Example, in zsh:

$ printf '%3s|\n' u ü $'u\u308' $'\u1100'
  u|
  ü|
 ü|
  ᄀ|

In bash:

$ printf '%3s|\n' u ü $'u\u308' $'\u1100'
  u|
 ü|
ü|
ᄀ|

ksh93 has a %Ls format specification to count the width in terms of display width.

$ printf '%3Ls|\n' u ü $'u\u308' $'\u1100'
  u|
  ü|
  ü|
 ᄀ|

That still doesn't work if the text contains control characters like TAB (how could it? printf would have to know how far apart the tab stops are in the output device and what position it starts printing at). It does work by accident with backspace characters (like in the roff output where X (bold X) is written as X\bX) though as ksh93 considers all control characters as having a width of -1.

As other options, you could try:

printf '%s\t|\n' u ü $'u\u308' $'\u1100' | expand -t3

That works with some expand implementations (not GNU's though).

On GNU systems, you could use GNU awk whose printf counts in chars (not bytes, not display-widths, so still not OK for the 0-width or 2-width characters, but OK for your sample):

gawk 'BEGIN {for (i = 1; i < ARGC; i++) printf "%-3s|\n", ARGV[i]}
     ' u ü $'u\u308' $'\u1100'

If the output goes to a terminal, you can also use cursor positioning escape sequences. Like:

forward21=$(tput cuf 21)
printf '%s\r%s%s\n' \
  "Früchte und Gemüse"    "$forward21" "foo" \
  "Milchprodukte"         "$forward21" "bar" \
  "12345678901234567890"  "$forward21" "baz"
  • 2
    That is incorrect. The ü caracter can be composed as u + ¨, which is 3 bytes. In the case of the question, it is encoded as 2 characters, but not all ü are created equally. – Ismael Miguel Mar 9 '17 at 17:02
  • 6
    @IsmaelMiguel, u\u308 is two characters (in the Unix/wc -m sense at least) for one glyph/graphem/graphem-cluster and is already mentioned and included in this answer. – Stéphane Chazelas Mar 9 '17 at 17:27
  • "that makes little sense as printf is to print text" Well, one could argue that printf deals with C chars (bytes); it shouldn't deal with text locales, and it shouldn't have the burden of understand the (possibly multibyte) charset encoding. But this line of defense conflicts with the (ISO C99) requirements that "%s" byte truncation should not result in "invalid" texts (truncated chars). Glibc even fails in that case (it prints nothing). A real mess. postgresql.org/message-id/… – leonbloy Mar 10 '17 at 21:12
  • @leonbloy, that might make sense of C's printf(3) (little sense after that C99 requirement you're mentioning, thanks for that), but not the printf(1) utility as every shell operator or other text utility deal with characters (or were modified to also deal with characters like wc which got a -m (while -c stayed byte) or cut that got a -b after -c could mean something else than bytes). – Stéphane Chazelas Mar 10 '17 at 22:02
  • Even if it used characters rather than bytes, it still wouldn't be suitable for aligning columns. You need to know how many terminal cells each character occupies, which varies by character (0-2). – R.. Mar 11 '17 at 1:04
10

If I change its encoding to latin-1, the alignment is correct, but the umlauts are rendered wrong:

Fr�chte und Gem�se   foo
Milchprodukte        bar
12345678901234567890 baz

Actually, no, but your terminal doesn't speak latin-1, and therefore you get junk rather than umlauts.

You can fix this by using iconv:

printf foo bar | iconv -f ISO8859-1 -t UTF-8

(or just run the whole shell script piped into iconv)

  • 3
    This is a useful comment but does not answer the core question. – gerrit Mar 9 '17 at 14:29
  • 1
    @gerrit how so? If printf does the right thing when printing in latin1, then have it print in latin1 and convert it to UTF-8 later? Seems like a proper fix for the core question to me. – Wouter Verhelst Mar 9 '17 at 14:52
  • 1
    The core question is "Why is it shrinking umlaut", the answer (as in other answers) is "because it doesn't support utf-8". It's not asking why are the umlauts rendered wrong or how can I fix the umlaut rendering. Either way, your suggestion is useful for the subset of utf-8 that can be represented as iso8859-1 (only). – gerrit Mar 9 '17 at 15:18
  • 4
    @WouterVerhelst, yes though that can only apply to text that can be encoded in a single-byte charset. – Stéphane Chazelas Mar 9 '17 at 16:08
  • 3
    I too read the question as "how can I get the output right" rather than "I don't mind the faulty output, as long as I know why". – Mr Lister Mar 11 '17 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.