1

How would one use sed or another tool to remove all characters except a specific multibyte range?

For instance, the range \xd8\xa0 - \xd9\x8a covers Arabic characters encoded in UTF-8. This was my naive hopeful attempt at using sed:

$ echo "Peace be upon you. السلام عليكم. שלום עליכם. Paz sobre vosotros." | sed -n -e 's|[^\xd8\xa0-\xd9\x8a]||p'
sed: -e expression #1, char 25: Invalid range end

Had sed understood multibyte characters, that should have left us with the string:

السلام عليكم

I have considered using iconv to convert to some Arabic-specific encoding, use sed, then convert back. However the work that I'm doing is non-trival, such as I might want to preserve both Arabic and Hebrew, so I'd like to keep everything in UTF-8 for the entire time if possible.

If sed cannot work with multibyte characters, what should I use? I've searched for both sed and perl solutions with no luck.

Note that I've tried to add a UTF-8 tag to the question, but unix.SE changes that to a Unicode tag. There is no Unicode here, these are encoded strings stored on disk. I think that there should be a separate UTF-8 tag.

2 Answers 2

Reset to default
1

You can use Unicode properties in Perl:

echo "Peace be upon you. السلام عليكم. שלום עליכם. Paz sobre vosotros." \
| perl -CIO -pe 's/\P{Arabic}//g'
  • -CIO tells Perl that input and output are UTF-8 encoded.
  • \P{...} is a negative property match, \p{...} would be a positive one.
5
  • 2
    Thank you! Why is perl so hard to search for?!? Even with your answer right in front of me I cannot fine the C flag documented or how I would have discovered \p{Foo}.
    – dotancohen
    Jun 18, 2022 at 20:10
  • 2
    -C
    – choroba
    Jun 18, 2022 at 20:12
  • 1
    @dotancohen See the man pages for perlre, perlunicode, perluniintro, perlunitut, utf8, and perluniprops. The man pages for Encode::Unicode and Encode may also be useful. Also, google "perl unicode". (I've linked to perlunicode, the others are linked at the bottom of that page). There's a LOT of info and documentation, and some of it will be obsolete (so check the dates on any results) - unicode support in perl is a complex topic, and has evolved over the last few decades.
    – cas
    Jun 19, 2022 at 4:40
  • @cas Thank you! man perl is near useless, but I just installed perldoc to get access to the actual docs.
    – dotancohen
    Jun 19, 2022 at 5:39
  • 1
    man perl isn't useless, it's an index page guiding you to specific topics of interest, such as overviews, tutorials, reference manuals, faqs, and more. perl is too big to document in one man page.
    – cas
    Jun 19, 2022 at 6:02
1

Using Raku (formerly known as Perl_6)

Sample Input (OP's text saved as a one-line file):

~$ cat peace.txt
Peace be upon you. السلام عليكم. שלום עליכם. Paz sobre vosotros.

Extract Arabic script:

~$ raku -ne 'put m:g/ <:Script<Arabic>>+ /;' peace.txt
السلام عليكم

Extract Hebrew script:

~$ raku -ne 'put m:g/ <:Script<Hebrew>>+ /;' peace.txt
שלום עליכם

Extract Latin script:

~$ raku -ne 'put m:g/ <:Script<Latin>>+ /;' peace.txt
Peace be upon you Paz sobre vosotros

Raku provides high-level support for Unicode, and it's Regex engine has grown to be a capable "grammar parser". See the links below. Note, the code above will return blank lines if no match. If you only want matching elements/lines you can simply add a conditional call to the $/ (or $<>) match variable:

~$ cat peace2.txt #(remove terminal period from each phrase)
Peace be upon you
السلام عليكم
שלום עליכם
Paz sobre vosotros

~$ raku -ne 'put m:g/ <:Script<Latin>>+ /;' peace2.txt
Peace be upon you


Paz sobre vosotros

~$ raku -ne 'put $/ if m:g/ <:Script<Latin>>+ /;' peace2.txt
Peace be upon you
Paz sobre vosotros

Finally, you can search for the negation of a Unicode property by interposing a ! character. You can also add (or even subtract) from the desired character class(es) with + and/or -. [Note additional regex atom :Z (whitespace) below. In the interest of completeness the code below shows both + and ! parameters--although a - might work in practice]:

~$ raku -ne 'put $/ if m:g/ <+:!Script<Latin> + :!Z >+ /' peace2.txt
السلام عليكم
שלום עליכם

https://docs.raku.org/language/unicode
https://docs.raku.org/language/unicode#index-entry-UTF-8_Clean-8
https://stackoverflow.com/a/66540269/7270649

3
  • 1
    Terrific answer, thank you. This concisely covers much material that I will likely need sometime.
    – dotancohen
    Jun 19, 2022 at 5:50
  • Thank you, @dotancohen! I also wanted to mention in addition to the Unicode <:Script< … >> property, Raku also lets you define regex characters via the Unicode block property, e.g. <:Block("Basic Latin")> or <:Block(q{Basic Latin})>, albeit with a slightly-different calling syntax (most likely to escape the enclosed text properly).
    – jubilatious1
    Jun 19, 2022 at 7:14
  • 2
    I foresee much sed and awk being replaced with Raku in my future. I'd until now avoided perl unless absolutely necessary due to the "12 ways to do anything" nature of the language. But I really like the way perl replaces both sed and awk, I should invest in learning that List Comprehension-type syntax.
    – dotancohen
    Jun 19, 2022 at 10:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .