Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to replace this with "D7 A0 D7 A8 D7 9B D7 A9". I would need to replace every "C3 B9" with "D7 A0" throughout the file. There will be about 30 different "C3 XX" values to be replaced by "D7 YY" values. I would need a table to loop through for each replacement.

Ideally this would replace the original file but it can also create a new output file.

Is there a utility that runs under Ubuntu that will do this job?

share|improve this question
I can't help directly, but people report using python which handles Unicode for this kind of thing. – EightBitTony Oct 23 '12 at 13:12
Is that Unicode encoded as UTF-8 (so e.g. C3B9 is U+00F9 LATIN SMALL LETTER U WITH GRAVE (ù))? Or as UCS-2 (so e.g. C3B9 is HANGUL SYLLABLE SSYEOT)? – Gilles Oct 23 '12 at 22:18

That's possible with perl,

A sample file:

echo -e "\xe8\x90" > a

And now if you do:

perl -pne 's/\xe8\x90/\x41\x42/g' < a > output

You get 0x41 0x42, which is AB

share|improve this answer

An alternative to the perl call is to use sed

sed -i.bckp -e "s/\xc3\xb9/\xd7\xa0/" file

Using the stream editor (sed) I have it doing inplace execution (meaning it will modify the file outright) and I supplied .bckp as the argument for -i so that the old file would be backed up. Then I run a swap expression which looks for your hex bytes with regex and swaps it out with the bytes you want to be using. And the call to file is the large file you wish to do this with.

If you want inplace replacement without backing it up (e.g. the file is in a repo), then you can just remove the .bckp after the -i and it will not create any new files.

Hope this helps.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.