I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to replace this with "D7 A0 D7 A8 D7 9B D7 A9". I would need to replace every "C3 B9" with "D7 A0" throughout the file. There will be about 30 different "C3 XX" values to be replaced by "D7 YY" values. I would need a table to loop through for each replacement.

Ideally this would replace the original file but it can also create a new output file.

Is there a utility that runs under Ubuntu that will do this job?

  • I can't help directly, but people report using python which handles Unicode for this kind of thing. – EightBitTony Oct 23 '12 at 13:12
  • 1
    Is that Unicode encoded as UTF-8 (so e.g. C3B9 is U+00F9 LATIN SMALL LETTER U WITH GRAVE (ù))? Or as UCS-2 (so e.g. C3B9 is HANGUL SYLLABLE SSYEOT)? – Gilles Oct 23 '12 at 22:18

That's possible with perl,

A sample file:

echo -e "\xe8\x90" > a

And now if you do:

perl -pne 's/\xe8\x90/\x41\x42/g' < a > output

You get 0x41 0x42, which is AB


An alternative to the perl call is to use sed

sed -i.bckp -e "s/\xc3\xb9/\xd7\xa0/" file

Using the stream editor (sed) I have it doing inplace execution (meaning it will modify the file outright) and I supplied .bckp as the argument for -i so that the old file would be backed up. Then I run a swap expression which looks for your hex bytes with regex and swaps it out with the bytes you want to be using. And the call to file is the large file you wish to do this with.

If you want inplace replacement without backing it up (e.g. the file is in a repo), then you can just remove the .bckp after the -i and it will not create any new files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.