Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to filter valid/invalid UTF-8 bytes, but I get strange results from the following regex (which is intended to handle a 3 byte form of UTF-8).

I have intended the pattern should not match the test bytes '\xE0\xA1\x63', but it does...
What am I missing?

showmatch() {
  echo -ne "    --> "
  echo -ne "$bytes" | 
    # strip whitespace from the pattern
    perl -l -ne '/^'${1// /}'$/x and print' |
      tr -d '\n' |
        xxd -p |
          tr -d '\n'

echo -n "before: "; echo -ne "$bytes" |xxd -p
# Note: all whitespace is stripped from each regex pattern.
#           Bytes 1 and 2 and 3
#          (---------------------------------------------------------------------------------------------------)
#              Bytes 1 and 2
#             (------------------------------------------------------------------------------)                  
#                [byt1][byt2-----]  |  [byt1][byt2-----]  |  [byte-1------------][byt2-----]      [byt3----]                                                          
#                =================     =================     ===============================      ==========                            
showmatch '(  ( ([\xE0][\xA0-\xBF]) | ([\xED][\x80-\x9F]) | ([\xE1-\xEC\xEE-\xEF][\x80-\xBF]) )  ([\80-\xBF])  )'
# witout spaces:
showmatch '((([\xE0][\xA0-\xBF])|([\xED][\x80-\x9F])|([\xE1-\xEC\xEE-\xEF][\x80-\xBF]))([\80-\xBF]))'

Here is the output

before: e0a163
    --> e0a163
    --> e0a163
share|improve this question
I reverted your subject line back to the original so we don't send people chasing a problem that is no longer written up as a problem. Also I would move your edit note from the top of the question to a comment. It's ok that it got solved by you shortly afterwards; the question can stay but leave the main question body as a question, comment on it here, and if need be provide an answer below (although it looks like that happened already) – Caleb May 4 '11 at 21:20
up vote 4 down vote accepted

Looks like you forgot an x in the last part of the regex:

[\80-\xBF]  -->  [\x80-\xBF]
share|improve this answer
:) thanks Andomar.... so simple.... As I've now mentioned in an edit in the question; actually writing and preparing the question gave me a fresh view.... I noticed a similar mistake in the question itslef.... I am soooooo relieved.... one and half days later :) – Peter.O May 4 '11 at 19:04

You've spotted the mistake, good. What would be useful now is ways to spot similar mistakes or to avoid them in the future.

You've already hit on Perl's x modifier to the regexp operators, which lets you have whitespace in the regexps. Your matching construct would be written with newlines (which would let you add comments).

/(  ( ([\xE0][\xA0-\xBF]) |
      ([\xED][\x80-\x9F]) |
      ([\xE1-\xEC\xEE-\xEF][\x80-\xBF]) )
    ([\80-\xBF])  )/x

or without the parentheses that aren't necessary here:

 [\80-\xBF] /x

I find that the missing x stands out more this way.

share|improve this answer
Ahhh..thanks Gilles, a good hint... I tried the x but I hadn't quite worked it out. I used a shell line ending, but that didn't work (maybe it was a trailing space..), so I used a quick 'sed' (I just forgot to remove the x), but come to think of it, it's pretty much the same as a quoted string, or braces spanning multiple lines. The x just tidies up – Peter.O May 5 '11 at 4:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.