12

Is it legal to print null bytes using awk's printf function according to POSIX? The POSIX standard of awk does not seem to explicitly mention it either way. Real world implementations differ in how they behave:

+$ gawk 'BEGIN { x = sprintf("\000"); print(length(x)); }'
1
+$ busybox awk 'BEGIN { x = sprintf("\000"); print(length(x)); }'
0
+$

and

+$ gawk 'BEGIN { printf("\000"); }' | xxd
00000000: 00                                       .
+$ busybox awk 'BEGIN { printf("\000"); }' | xxd
+$

Is this specified somewhere in the standard? If yes, is the behaviour required for variables (x = sprintf("\000")) and printf (printf("\000")) same?

  • 1
    I take it from the prompt’s you’re a vi-mode user in bash. Nice to find a friend :) – D. Ben Knoble Jan 26 at 23:40
17

There are at least 4 relevant pieces of text in the POSIX.2018 specification of awk:

Emphasis (bold text) is mine in all the quoted text below:

Input files to the awk program from any of the following sources shall be text files

That means that if the input contains NUL characters (which would make it non-text as per the POSIX definition of text), then the behaviour is unspecified.

\ddd : A <backslash> character followed by the longest sequence of one, two, or three octal-digit characters (01234567). If all of the digits are 0 (that is, representation of the NUL character), the behavior is undefined.

So \000 results in undefined behaviour.

About regexp matching:

However, in all awk ERE matching, the use of one or more NUL characters in the pattern, input record, or text string produces undefined results

About printf/sprintf:

7. For the c conversion specifier character: if the argument has a numeric value, the character whose encoding is that value shall be output. If the value is zero or is not the encoding of any character in the character set, the behavior is undefined.

So, that's another way to get a NUL character that leads to undefined behaviour.

So, to sum up, in awk, POSIX tells us you can't use the NUL character portably, whether it's for input, output or to store in its variables.

gawk (since at least 2.10 in 1989 which is the earliest version I could find where NUL support is documented) and @ThomasDickey's mawk (since version 20140914) are two implementations that can deal with NUL.

  • gawk still is NOT safe to use with data containing NUL-chars. When using string with NUL-chars as files or commands, they will SILENTLY decay into zero-terminated strings. printf 'foo\0bar' | gawk '{print $0 > $0}'. – pizdelect Jan 27 at 11:42
  • @pizdelect, yes, but in that case, it's not gawk's fault. File names or command arguments or environment variables (name and values) can't contain NUL characters. It's up to you the gawk user to deal with it as appropriate, there's nothing that gawk can do to magically fix it (though I'd agree with your SILENTLY objection, that it could be nice if it reported those cases). zsh can also deal with NUL, but in a similar way it will break when using APIs (execve(), open()...) that can't handle them. – Stéphane Chazelas Jan 27 at 11:44
-6

C-strings in general cannot contain nul bytes and awk basically is some kind of C interpreter. A POSIX utility that is able to print nul bytes would need to be explicitly mentioned in the POSIX standard. printf in awk is not such a case.

The following cases are explicitly listed in POSIX:

echo 'x\0000y'
printf 'x\000y\n'
printf '%b\n' 'x\0000y'

all three commands print four characters on a certified OS with POSIX UNIX branding.

Your awk examples are using unspecified behavior.

  • 2
    "awk basically is some kind of C interpreter" It is not. – Joseph Sible-Reinstate Monica Jan 26 at 23:35
  • 3
    The three usecases also print 4 characters in NetBSD with ksh . Besides the ambiguity of the "C interpreter" phrasing, why the six (!) downvotes? – schaiba Jan 27 at 12:36
  • These are most likely from Linux people that do not know that the XSI enhancements to echo are required for a POSIX UNIX branding and that platforms that do not implement XSI at best may get a POSIX certification for small embedded platforms. – schily Jan 27 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.