10
$ echo ABC | awk '$0 ~ /^[a-b]/'
ABC
$ echo ABC | awk '$0 ~ /^[a-a]/'
$ echo ABC | awk '$0 ~ /^a/'
$ 

You see. /[a-b]/ captures A, but /[a-a]/ or /a/ doesn't. Why?

3
  • 1
    See Does (should) LC_COLLATE affect character ranges? for more (unresolved) info on this topic.
    – Caleb
    Commented Aug 24, 2011 at 21:07
  • This appears to be more than just a simple(?) LC_COLLATE issue, because using some non-C values for LC_COLLATE produces different results, depending on which utility is used. eg. 'sed' and 'grep' give different results to 'awk' when using LC_COLLATE=en_AU.UTF-8 or en_US.UTF-8 ... sed and grep manage to resolve the case issue, and only lower-case is printed (using the same values as above)
    – Peter.O
    Commented Aug 25, 2011 at 9:38
  • At least in gawk (GNU Awk) this has been fixed ([a-z] matches only lowercase letters) since version 4.0: gnu.org/software/gawk/manual/html_node/Ranges-and-Locales.html Commented May 30, 2016 at 6:08

1 Answer 1

8

It is a "locale" problem, I think.

In my locale, it_IT, the following snippet

if [[ a < A ]]; then
  echo "a < A"
elif [[ a > A ]]; then
  echo "a > A"
else
  echo "a = A"
fi

if [[ b < A ]]; then
  echo "b < A"
elif [[ b > A ]]; then
  echo "b > A"
else
  echo "b = A"
fi

shows

a < A
b > A

so that A is (surprisingly) between a and b, so in the range.

Try executing

echo ABC | LC_COLLATE=C awk '$0 ~ /^[a-b]/'

Edit

the following command shows the collating order in your locale:

echo $(LC_COLLATE=C printf '%s\n' {A..z} | sort)

the output on my machine is

` ^ _ [ ] a A b B c C d D e E f F g G h H i I j J k K l L m M n N o O p P q Q r R s S t T u U v V w W x X y Y z Z

(cannot understand from bash's manual page if sequence expressions are expanded in locale collating order or not; it seems not).

11
  • +1, but you just need LC_COLLATE, not LC_ALL, for this particular case.
    – mattdm
    Commented Aug 24, 2011 at 15:40
  • @mattdm: you're right, i'm lazy
    – enzotib
    Commented Aug 24, 2011 at 15:52
  • @enzotib: I am puzzled... This idea seems to mean that every time I want to set a range /[a-x]/, I must use LC_COLLATE....What on earth has a collating sequence got to do identifying what is Upper-case vs Lower case? ... I can't see how a Collating sequence dictates what is upper case and what isn't... I keep grappling with these locale issues, and am slowly making headway, but this one has me stumped.
    – Peter.O
    Commented Aug 24, 2011 at 18:14
  • @fred: frequently, when using sort, join or the like, I start my scripts with export LC_COLLATE=C. Now I have to start this way also scripts using awk :)
    – enzotib
    Commented Aug 24, 2011 at 18:44
  • 3
    The sequence eval order doesn't matter in this case since you sort after the sequence is generated. However, your example would work more accurately with LC_COLLATE next to sort: "echo $(printf '%s\n' {A..z} | LC_COLLATE='C' sort)" ... which would contrast correctly with the default case "echo $(printf '%s\n' {A..z} | LC_COLLATE='' sort)". The original syntax above never actually applies the locally modified LC_COLLATE to the sort command [of course, all bets are off if LC_ALL was set somewhere...] Commented Jun 1, 2012 at 19:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .