If I run the following .sh file:

#!/bin/sh -a
echo "a" | sed -e 's/[\d001-\d008]//g'

The result is an error:

sed: -e expression #1, char 18: Invalid range end

But if I run the following .sh file:

#!/bin/sh
set -a
echo "a" | sed -e 's/[\d001-\d008]//g'

It runs without error. Isn't the second code supposed to be equivalent of the first one? Why the error in the first one?

  • Not all sh are the same. Nor all sed are equivalent. Which sh are you using? In which OS? and Which sed (maybe? sed --version if it doesn't fail)? – Isaac Aug 13 at 16:11
  • 1
    setting LC_COLLATE=C (or POSIX) for the call to sed works around the issue – Jeff Schaller Aug 13 at 16:13
  • 4
    One difference I found: the first script invokes sed (and presumably any other utility) with POSIXLY_CORRECT=y in the environment, the second does not have POSIXLY_CORRECT in the environment. The shell I invoke both scripts from does not have POSIXLY_CORRECT in its environment. – Mark Plotnick Aug 13 at 18:11
  • 1
    Ah, echo "a" | POSIXLY_CORRECT=y sed -e 's/[\d001-\d008]//g' reproduce your problem – Isaac Aug 13 at 18:20
  • 1
    Confirming that the above fails for me exactly as the OP has shown on CentOS 7.x - GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu) & CentOS Linux release 7.5.1804 (Core). – slm Aug 14 at 3:47
up vote 31 down vote accepted

When bash is called with the name sh, it does this:

if (shell_name[0] == 's' && shell_name[1] == 'h' && shell_name[2] == '\0')
    act_like_sh++;

and then later sets the POSIXLY_CORRECT shell variable to y:

if (act_like_sh)
  {
    bind_variable ("POSIXLY_CORRECT", "y", 0);
    sv_strict_posix ("POSIXLY_CORRECT");
  }

bind_variable calls bind_variable_internal, which, if the shell attribute a is on at the time (which it would be if you invoked the shell with -a), marks the shell variable as exported.

So in your first script:

#!/bin/sh -a
echo "a" | sed -e 's/[\d001-\d008]//g'

sed is invoked with POSIXLY_CORRECT=y in its environment, which will make it complain about [\d001-\d008]. (Same thing happens if sed is given the --posix option.)

In GNU sed, \dNNN is an escape code for the character whose numerical value in base-10 is NNN, but in POSIX mode, this is disabled inside a bracket expression, so [\d001-\d008], means literally the characters \, d, etc., with the range being from 1 to \. In order of character codes, 1 comes before \ (and the range includes all digits except zero, plus all uppercase letters, plus some special characters). In the en_US.UTF-8 locale which you were using, \ sorts before 1, however, so the range is invalid.

In your second script:

#!/bin/sh
set -a
echo "a" | sed -e 's/[\d001-\d008]//g'

even though POSIXLY_CORRECT is set in the shell, it isn't exported, so sed is invoked without POSIXLY_CORRECT in the environment, and sed runs with GNU extensions.

If you add export POSIXLY_CORRECT near the top of your second script, you'll also see sed complain.

  • 6
    To me, that's a bug. – Stéphane Chazelas Aug 13 at 19:23
  • 1
    holy shell horror, Batman! That's an interesting quirk (and a bit of a change to see an issue that comes from /bin/sh actually being Bash). The same happens if POSIXLY_CORRECT is in the environment before the sh Bash starts: it will also pass it on as POSIXLY_CORRECT=y. – ilkkachu Aug 13 at 19:30
  • 3
    @StevenPenny, but POSIXLY_CORRECT isn't in the environment when the shell starts, and the script doesn't set it. The shell does. It creates an environment variable out of nowhere, which is extra bad since it does that in a mode where it's supposed to be, and tries to be standard-compliant. – ilkkachu Aug 13 at 19:51
  • 4
    FWIW, Bash also doesn't seem to document that it would set POSIXLY_CORRECT on its own. There's no mention of it in the list of effects of POSIX mode and the variable description only says that setting it changes the shell to POSIX mode, not the other way around. – ilkkachu Aug 13 at 19:55
  • 1
    @ilkkachu. Done. I think the POSIX spec should also be updated to clarify what variables are affected by allexport. – Stéphane Chazelas Aug 14 at 15:54

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.