How do I differentiate between uppercase and lowercase characters in a case statement?

Question

I'm trying to revive my rusty shell scripting skills, and I've run into a problem with case statements. My goal in the program below is to evaluate whether a user-supplied string begins with a capital or lowercase letter:

# practicing case statements
echo "enter a string"
read yourstring
echo -e "your string is $yourstring\n"

case "$yourstring" in
    [A-Z]* )
       echo "your string begins with a Capital Letter"
       ;; 
    [a-z]* )
       echo "your string begins with a lowercase letter"
       ;; 
    *)     
       echo "your string did not begin with an English letter"
       ;;
esac

myvar=nope

case $myvar in
   N*)
     echo "begins with CAPITAL 'N'"
     ;;
   n*)
     echo "begins with lowercase 'n'"
     ;;
   *)
     echo "hahahaha"
     ;;
esac

When I enter a string beginning with a lowercase letter (e.g., "mystring" with no quotes), the case statement matches my input to the first case and informs me that the string begins with a capital letter. I wrote the second case statement to see if I was making some obvious syntax or logic error (perhaps I still am), but I don't have the same problem. The second case structure correctly tells me that the string held by $myvar begins with a lowercase letter.

I have tried using quotes to enclose $yourstring in the first line of the case statement, and I've tried it without quotes. I read about the 'shopt' options and verified that 'nocasematch' was off. (For good measure, I toggled it on and tried again, but I still didn't get the correct result from my first case statement.) I've also tried running the script with sh and bash, but the output is the same. (I call the shell explicitly with "sh ./case1.sh" and "bash ./case1.sh" because I did not set the execution bit. Duplicating the file and setting the execution bit on the new file did not change the output.)

Though I don't understand all of the output from running the shell with the '-x' debug option, the output shows the shell progressing from the first "case" line to execution of the command following the first pattern. I interpret this to mean that the first pattern was a match for the input string, but I am uncertain why.

When I switch the order of the first two patterns (and corresponding commands), the case statement succeeds for lowercase letters but incorrectly reports "MYSTRING" as beginning with lowercase letters. Since anything alphabetic is detected as matching whichever pattern appears first, I think I have a logical error...but I'm not sure what.

I found a post by "pludi" on unix.com in which it is advised that "the tests for lowercase and upper case characters were [a-z] and [A-Z]. This no longer works in certain locales and/or Linux distros." (see https://www.unix.com/shell-programming-and-scripting-128929-example-switch-case-bash.html) Sure enough, replacing the character ranges with [[:upper:]] and [[:lower:]] resolved the problem.

I'm on Fedora 31, and my locale output is as follows:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8" 
LC_NUMERIC="en_US.UTF-8" 
LC_TIME="en_US.UTF-8" 
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8" 
LC_MESSAGES="en_US.UTF-8" 
LC_PAPER="en_US.UTF-8" 
LC_NAME="en_US.UTF-8" 
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8" 
LC_IDENTIFICATION="en_US.UTF-8" 
LC_ALL= 

I'd like to know whether I'm not understanding character ranges, or not understanding how pattern matching works in case statements, or if the underlying shell capabilities changed (and why?). If anyone has the patience, I would greatly appreciate an explanation; I'm also happy to read relevant documentation. Thanks!

Answer 1

A simple answer, one which no doubt others can supersede.

The character set ordering is now different depending on which locale is in use. The concept of locale was introduced to support different nationalities and their different languages. As you can see from the output of locale there are several different areas now addressed - not just collation.

In your case it's US, and for sorting and collation purposes the alphabet is either AaBbCc...Zz or A=a, B=b, C=c, etc.(I forget which, and I'm not at a computer where I can verify one over the other). Locales are very complicated, and in certain locales there can be characters that are invisible as far as sorting and collation are concerned. The same character can sort differently depending on which locale is in use.

As you've found, the correct way to identify lowercase characters is with [[:lower:]]; this will include accented characters where necessary, and even lowercase characters in different alphabets (Greek, Cyrillic, etc.).

If you want the classic ordering you can revert per application or even per command by setting LC_ALL=C. For a contrived example,

grep some_pattern | LC_ALL=C sort | nl

Answer 2

There has been an enduring battle between dictionary order and ASCII order.
For a long time.

From the point of view of Unicode, characters should be sorted by local customs in their dictionary order, thus a A b B ... for American letters (ASCII letters). That is usually matched by the [a-zA-Z] range in the en_US.utf-8 locale. Internationalization usually agree with this.

From the point of view of programmers, due to the C language, the [a-z] should match only the ascii characters from 97 up to 122 as one byte value. Similarly for [A-Z]. That usually will match the C language definition of a character as one byte. Some script writers want to use this definition.

That battle has moved from one interpretation to the other from time to time.
Sometimes the [a-z] range becomes only abcdefghijklmnopqrstuvwxyz.
Sometimes it shift to aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYz.
Or to some other quite more complex list.

The details are complex. The history is long. The battle is still raging.

---

So, you may get (testing the string book):

• "your string begins with a Capital Letter" for bash versions 2, 3 and 4 and
• "your string begins with a lowercase letter" for bash version 5 (and 1)
• Most shells will report that as a "lowercase letter".

If you test the string úber (in the en_US.UTF-8), you will get:

• "lowercase" in ksh/ATT-sh
• "Not an English Letter" in dash, zsh, bash 5.0+ or [lm]ksh.
• "Capital Letter" in bash 2,3, and 4.

As well as the string Úber.

So, the result is varied.

You could also set LC_ALL=C to enforce the interpretation that a-z are only lowercase letters (and A-Z are only Uppercase letters). That will freeze the collation used to only the one from C. No change if the locale change. A more robust script, but a less adaptable script.

There is also the option to use [[:lower:]] but, again, that is warranted to be the ASCII range a-z only in the C locale. It may get enforced to all locales in future versions of POSIX (but not yet published 2020).

All considered, the only safe way to ensure that no external decision (from a shell developer of Unix specification) will change the range of your code will be:

# practicing case statements
echo "enter a string"
read yourstring
echo -e "your string is $yourstring\n"

low='abcdefghijklmnopqrstuvwxyz'
cap='ABCDEFGHIJKLMNOPQRSTUVWXYZ'

case "$yourstring" in
    [$cap]* ) echo "your string begins with a Capital Letter"   ;; 
    [$low]* ) echo "your string begins with a lowercase letter" ;; 
    *)      echo "your string did not begin with an English letter" ;;
esac