For example:

sed 's/\u0091//g' file1

Right now, I have to do hexdump to get hex number and put into sed as follows:

$ echo -ne '\u9991' | hexdump -C
00000000  e9 a6 91                                          |...|

And then:

$ sed 's/\xe9\xa6\x91//g' file1

Just use that syntax:

sed 's/馑//g' file1

Or in the escaped form:

sed "s/$(echo -ne '\u9991')//g" file1

(Note that older versions of Bash and some shells do not understand echo -e '\u9991', so check first.)

  • 1
    Does sed count 馑 as one character or 3? That is, does echo 馑 | sed s/...// print anything? – immibis Apr 17 '15 at 11:22
  • @immibis Since sed has the g modifier it replaces all occurence also when they follow each other. Also sed should count it as one character, see: echo -ne "馑" | wc -m gives 1. If you count the bytes (wc -c) it would return 3. Did I understand your question correctly? – chaos Apr 17 '15 at 11:28
  • I meant: does . mean "one character" or "one byte"? – immibis Apr 17 '15 at 11:30
  • @immibis I matches one character hence echo 馑 | sed s/...// gives me (nothing is replaced) – chaos Apr 17 '15 at 11:33
  • 4
    @chaos: It works under en_US.UTF-8, but doesn't under C. – choroba Apr 17 '15 at 12:28

Perl can do that:

echo 汉典“馑”字的基本解释 | perl -CS -pe 's/\N{U+9991}/Jin/g'

-CS turns on UTF-8 for standard input, output and error.

  • 7
    Perl can do almost anything..... – wobbily_col Apr 17 '15 at 10:49

A number of versions of sed support Unicode:

  • Heirloom sed, which is based on "original Unix material".
  • GNU sed, which is its own codebase.
  • Plan 9 sed, which has been ported to Unix-like operating systems.

I couldn't find information on BSD sed, which I thought was strange, but I think the odds are good that it supports Unicode too. Unfortunately, there is no standard way to tell sed which encoding to use, so each one does this in its own ways.

  • Do they support UTF-16 with and without BOM ? – Bon Ami Apr 17 '15 at 17:12
  • 10
    UTF-16 is pretty unusable in Unix-based OSes. It's also an abomination that should have never seen the light of day. – Brian Bi Apr 17 '15 at 19:11
  • Whether or not they support UTF-16 depends on the implementation, and I'm afraid I don't have that data. I doubt that Plan 9 sed does (the original OS is UTF-8 everywhere), but I can't be sure, and even if it doesn't, the others might. – The Spooniest Apr 17 '15 at 19:30

This works for me:

$ vim -nEs +'%s/\%u9991//g' +wq file1

It’s a drop more verbose than I’d like; here’s a full explanation:

  • -n disable vim swap file
  • -E Ex improved mode
  • -s silent mode
  • +'%s/\%u9991//g' execute the substitution command
  • +wq save and exit
  • I suppose this modifies file1 in-place, is that correct? – gerrit Jan 10 at 10:32
  • @gerrit that’s correct, and thanks for pointing it out. – Aryeh Leib Taurog Jan 10 at 19:21

Works for me with GNU sed (version 4.2.1):

$ echo -ne $'\u9991' | sed 's/\xe9\xa6\x91//g' | hexdump -C
$ echo -ne $'\u9991' | hexdump -C
00000000  e9 a6 91

(As another replacement for sed you could also use GNU awk; but it don't seem necessary.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.