2

The ASCII character range is from 0 to 127, and within that range, awk's printf with the %c format specifier outputs one byte of data:

$ awk 'BEGIN{printf "%c", 97}'
a

$ awk 'BEGIN{printf "%c", 127}' | xxd
00000000: 7f

$ awk 'BEGIN{printf "%c", 127}' | xxd -b
00000000: 01111111

But for values greater than 127, it will print out multiple bytes:

$ awk 'BEGIN{printf "%c", 128}' | xxd
00000000: c280

$ awk 'BEGIN{printf "%c", 128}' | xxd -b
00000000: 11000010 10000000

What is the significance of 0xc280, and why does awk output that character instead of 0x80?

Improve this question

2 Answers 2

Reset to default
7

This is UTF-8 encoding. 11000010 starts a two-byte sequence (the first two bits set followed by a clear bit), and the significant bits are 00010000000 (the last five bits of the first byte, and the last six bits of the second byte), which is 128.

AWK is outputting this because your locale is set to use UTF-8; you can switch to a non-UTF-8 locale to see the difference:

$ LC_ALL=C awk 'BEGIN{printf "%c", 128}' | xxd -b
00000000: 10000000
Improve this answer
1
  • Failed on Termux, as en_US.UTF-8 is the only available locale and my AWK is refering GAWK; can be solved by either env -i awk to call AWK on Android itself, or gawk -b: with GAWK-specific option.
    – user451199
    Jan 27, 2021 at 8:26
-5

One trick to get awk to print out any arbitrary byte, regardless of whether your locale is UTF8 or POSIX or C, is to add a large multiple of 256 to the unsigned byte ordinance value to bring the new number above 0x10FFFF, the limit of Unicode 14 spec.

Here's a demo of how to access arbitrary bytes in gawk byte mode to print out UTF8-encoded characters. The same method can also be used in gawk unicode mode to access any byte:

 gawk -e 'BEGIN { printf("%c",50000) }' | od -baxco -t dC

0000000   354 215 220                                                    
           ?  8d  90                                                    
             8dec    0090                                                
          썐  **  **                                                    
           106754  000220                                                
          -20-115-112                                                    
0000003

% gawk -b -e 'BEGIN { printf("%c%c%c",
                             (-20)+8^8,
                            (-115)+8^8,
                            (-112)+8^8) }' | od -baxco -t dC
0000000   354 215 220                                                    
           ?  8d  90                                                    
             8dec    0090                                                
          썐  **  **                                                    
           106754  000220                                                
          -20-115-112                                                    
0000003

% gawk -e 'BEGIN { printf("%c%c%c%c",\
                                      \
                   0xAB+8^8, 0xBA+8^8, \
                   0xCA+8^8, 0xFE+8^8) }' \
 | god --endian=big -baxco -t dCxI

0000000  253  272  312  376
           +    :    J    ~
             abba      cafe
           ?    ?    ?    ?
           125672    145376
         -85  -70  -54   -2
                   abbacafe
0000004

This method works regardless of your locale settings.

For mawk-1, mawk2-beta, and nawk, you can also subtract 256 from the unsigned byte value and do printf("%c") using negative numbers. gawk used to also allow that, but recent versions might have disabled it.

Improve this answer
12
  • I probably have a slightly old version of gawk at hand, but with gawk 'BEGIN { printf("%c", 0 + 8^8) }' | od -tx1, I get the output bytes f9 80 80 80 80, i.e. it seems to print that UTF-8 too. Same-o up to 2^31-1 or so, I guess that's the since-prohibited upper range. Anyway, it means that one might want to use e.g. 2^32, or 8^11 (2^33) as the offset instead.
    – ilkkachu
    Jan 23, 2022 at 11:02
  • 2
    Also it might be worth mentioning that gawk actually documents that it falls back to using the low 8 bits if the conversion fails. Without knowing that it's documented, this might seem dangerously fiddly, as something that has a risk of breaking in a future version. Well, I guess it still could, if the locale definitions and/or the UTF-8 specification change (again).
    – ilkkachu
    Jan 23, 2022 at 11:04
  • actually, worse, gawk 'BEGIN { printf("%c", 255 + 2^32) }' | od -tx1 gives me c3 bf.
    – ilkkachu
    Jan 23, 2022 at 11:22
  • if you have the older variants of gawk, then subtract 8^8 instead of adding it. I'd advise against >2^31 as that may be subject to wrap-around : C3 BF is exactly U+00FF. Yes UTF8 can change, but it'd be quite some time before Unicode spec start bumping against this 8^8, seeing that it's some 15x larger than current upper limit : gawk -e 'BEGIN { printf("%c|%c|%c",255+8^8,255+8^11,255+2^31) }' | odview 377 | ÿ ** | 377 377 174 303 277 174 377 255 124 195 191 124 255 ff 7c c3 bf 7c ff
    – RARE Kpop Manifesto
    Jan 25, 2022 at 4:54
  • 2
    Those downvoting you (which now include me) do not consider this a solution to the question asked since it is instead giving a (useful and interesting) solution to a completley different question: "How can I print an arbitrary byte in awk". That isn't the question asked here. This question is asking "What is the significance of 0xc280, and why does awk output that character instead of 0x80?".
    – terdon ♦
    Jan 26, 2022 at 11:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .