How would one use
sed or another tool to remove all characters except a specific multibyte range?
For instance, the range
\xd8\xa0 - \xd9\x8a covers Arabic characters encoded in UTF-8. This was my naive hopeful attempt at using sed:
$ echo "Peace be upon you. السلام عليكم. שלום עליכם. Paz sobre vosotros." | sed -n -e 's|[^\xd8\xa0-\xd9\x8a]||p' sed: -e expression #1, char 25: Invalid range end
sed understood multibyte characters, that should have left us with the string:
I have considered using
iconv to convert to some Arabic-specific encoding, use
sed, then convert back. However the work that I'm doing is non-trival, such as I might want to preserve both Arabic and Hebrew, so I'd like to keep everything in UTF-8 for the entire time if possible.
sed cannot work with multibyte characters, what should I use? I've searched for both
perl solutions with no luck.
Note that I've tried to add a
UTF-8 tag to the question, but unix.SE changes that to a
Unicode tag. There is no Unicode here, these are encoded strings stored on disk. I think that there should be a separate