11

I would like to include a couple of non-ASCII characters in my POSIX shell script comments. Note this is in no way a duplicate of e.g. "Which character encodings are supported by posix?" as I care for the comments section only. Therefore I don't care if I can use Unicode for actual coding. I care if all shells which are POSIX-compatible will be able to read my file, or if some fail due to non-ASCII encoding.

So, my editor (VS Code), will save such a file with UTF-8 encoding.

Here are two files identified with the file utility (unsure if it cares for BOM):

$ file script1*
script1:     POSIX shell script, ASCII text executable
script1.utf: POSIX shell script, Unicode text, UTF-8 text executable

Question is, if POSIX shell scripts must be in ASCII only. Can't find anything relevant on this topic. Thank you.

0

3 Answers 3

24

POSIX specifies how tokens should be recognised, including comments:

If the current character is a '#', it and all subsequent characters up to, but excluding, the next <newline> shall be discarded as a comment. The <newline> that ends the line is not considered part of the comment.

You’re asking specifically about UTF-8; UTF-8 ensures that newlines are encoded as expected in ASCII, and that only newlines produce the corresponding byte value. So no non-ASCII UTF-8 character encoding can be mis-interpreted as a newline, which means that UTF-8 is safe for use in comments in POSIX-compliant shells.

Your question mentions BOMs in passing; those are not required in UTF-8, and files starting with a BOM are not backwards-compatible with ASCII. A shell script starting with a BOM is not POSIX-compliant and won’t behave as expected:

$ printf '\xEF\xBB\xBFecho Hello\n' > bomtest
$ file bomtest
bomtest: POSIX shell script, Unicode text, UTF-8 (with BOM) text executable
$ sh bomtest
bomtest: line 1: echo: command not found

The BOM is considered part of the first token, so the shell looks for a command matching “BOM echo” and doesn’t find one.

12
  • 3
    If the current character is a '#', it and all subsequent characters up to, but excluding, the next <newline> shall be discarded as a comment. But what if what follows don't form valid characters in the locale? For instance, non-ASCII UTF-8 encoded text is likely to be invalid in locale using BIG5-HKCSC to GB18030 charset. In effect, that would get you into unspecified territory. yash (one of the most strictly compliant POSIX sh implementations) will choke on that. Commented Jun 19 at 9:43
  • A file that starts with the UTF-8 encoding of U+FEFF followed by #!/bin/sh in a UTF-8 locale is more "POSIX" than one that starts with #!/bin/sh. The former asks the shell to run the sh command in the bin subdir of the <U+FEFF>#! directory, while the latter is unspecified. Commented Jun 19 at 9:47
  • 1
    @StéphaneChazelas I was considering only the POSIX locale... Commented Jun 19 at 9:48
  • 1
    @Toby indeed, I’ve reworked the example to clarify what the actual behaviour is. Commented Jun 19 at 11:23
  • 2
    re. the POSIX locale, if the answer on the other linked-to post is right and POSIX doesn't specify character encoding, then what does the POSIX locale even mean regarding this? Could a system using an oddball charset where the newline was encoded as e.g. 0xc3 be POSIX compliant? That would make the UTF-8 encoding of ä contain a newline, which should cause issues for comments... Of course, such a system or charset hopefully doesn't even exist, and ASCII is likely the base charset on many systems, but I just started wondering...
    – ilkkachu
    Commented Jun 19 at 15:50
12

The accepted answer is fine, but let me explain the same with a slightly different angle:

POSIX is very exact and complete in its handling of character encodings. That is, any conceivable effect of nontrivial character encodings is mentioned in the relevant pages. As shown in this answer it basically makes a minimum requirement about which characters are there, but does not say anything really limiting about the encoding of those characters. It does define what happens in certain difficult cases (e.g., invalid encoding in certain circumstances; occurrences of the special NUL byte; the requirement that a certain minimum amount of characters are contained in the character set and so on and forth). The relevant portion of the standard is POSIX Character Set.

Do note that in the overwhelming majority of places, POSIX then talks about characters, not bytes. In fact, if you look for "byte" in POSIX Shell Command Language, any mention of "byte" is always in the context of where the encoding could have gone wrong, or where RAM limits are involved (i.e. maximum path lengths) and so on and forth, or what should happen in case the user changes the encoding via setting the relevant environment variables within a shell. In all "normal" descriptions (i.e. of shell commands), it talks about characters, and characters only.

Specifically, the comment character is defined like this:

If the current character is a '#', it and all subsequent characters up to, but excluding, the next newline shall be discarded as a comment. The newline that ends the line is not considered part of the comment.

In the character set specification linked above, we find:

The encoded values associated with the members of the portable character set are each represented in a single byte.

(The # or <number-sign> is part of the Portable Character Set).

This latter is interesting. UTF-8, which you are asking about, contains 7-bit ASCII as a true subset in character definition and encoding, so the postulation is fulfilled. UTF-16 would in this case be difficult, and hence (and for many other reasons) UTF-16 is not POSIX compliant.

For your current question: all well-formed uses of UTF-8 are fine after the comment. The comment (number-sign) and the newline are completely well-defined and safe to in UTF-8; not by coincidence UTF-8 also makes sure that characters that encode to more than one byte do not contain 7-bit ASCII bytes themselves, so there can neither be an involuntary number-sign or new-line character in a random UTF-8 encoding.

All things that are not easy to specify are specifically specified as "unspecified" by POSIX. So what happens if your script contains invalid encodings is, well, undefined. For example if at the end of the comment, before the new-line, the last character is supposed to be a multi-byte encoding, and the new-line comes in the middle of that -> unspecified. Expect bugs and all kinds of hilarity to ensue.

That said, a well-behaved command will either not (need to) care about such cases, by basically treating everything as bytes, and only caring about specific 7-bit bytes (i.e., control characters).

Or, if the command does support UTF-8, the usual UTF-8-defined recovery methods for detecting and handling invalid encodings should apply. Specifically per RFC 3629 - UTF-8:

Implementations of the decoding algorithm above MUST protect against decoding invalid sequences.

But POSIX says nothing about that, in the context of this question.

0

If you are worrying about really POSIX compliant shells, the other answers are fine. But the Real World™ is not POSIX compliant. You'll run into shells that handle Latin-1, or UTF-8, or other exotic encodings without a hitch. And (most probable in older, limited or just only somewhat-Unix-like systems) you'll also find ones that get a fit if the input is not strictly ASCII.

Postel's principle: "Be conservative in what you send, be liberal in what you accept". Here you are the sender.

6
  • 1
    I don't understand the downvote. As long as you only need to support current desktop/server environments, you'll probably be ok with UTF-8. But if you ever need to run your bash script on some older system, or a "minimal" system that has only the LANG=C locale definitions installed, or on a system that needs the locale set to some iso8859 or asian non-utf8 character set, even a cat on your script might mess up the terminal and will show wrong characters. If there's a chance you ever need to work with one of those, using UTF-8 instead of ASCII just isn't worth the headache. Commented Jun 20 at 18:34
  • @GuntramBlohm: I did not downvote, but IMHO this is overly conservative. Ken Thompson knew perfectly well what would and would not run on the Unixes of the day, and explicitly designed UTF-8 so that it would run on most reasonable systems that were in use at the time. Pretty much the only way to break UTF-8 is to clobber the high bit, or to deliberately configure the system with a gratuitously incompatible encoding (e.g. UTF-16, Shift JIS, EBCDIC, etc.), and if you do that, then you know perfectly well what you're getting yourself into.
    – Kevin
    Commented Jun 21 at 8:15
  • 1
    "the above answers are fine" — assumes that this answer will never get more votes than the others.. Commented Jun 22 at 1:00
  • @RayButterworth, fixed. Thanks for the encouragement!
    – vonbrand
    Commented Jul 3 at 15:31
  • @Kevin, there are extremely limited, overly restictive environments around. I had the privilege to develop on one a long time back. Latin-1 gave it fits. Hopefully extinct by now, but Murphy's law being what it is...
    – vonbrand
    Commented Jul 3 at 15:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .