I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to replace this with "D7 A0 D7 A8 D7 9B D7 A9". I would need to replace every "C3 B9" with "D7 A0" throughout the file. There will be about 30 different "C3 XX" values to be replaced by "D7 YY" values. I would need a table to loop through for each replacement.

Ideally this would replace the original file but it can also create a new output file.

Is there a utility that runs under Ubuntu that will do this job?

  • I can't help directly, but people report using python which handles Unicode for this kind of thing. – EightBitTony Oct 23 '12 at 13:12
  • 1
    Is that Unicode encoded as UTF-8 (so e.g. C3B9 is U+00F9 LATIN SMALL LETTER U WITH GRAVE (ù))? Or as UCS-2 (so e.g. C3B9 is HANGUL SYLLABLE SSYEOT)? – Gilles 'SO- stop being evil' Oct 23 '12 at 22:18

That's possible with perl,

A sample file:

echo -e "\xe8\x90" > a

And now if you do:

perl -pne 's/\xe8\x90/\x41\x42/g' < a > output

You get 0x41 0x42, which is AB

| improve this answer | |
  • How do I make it work across lines? I'm trying to enforce no more than one newline after a {, so I tried perl -pne "s/\x7b\x0a\x0a/\x7b\x0a/g" < a > output and it doesn't work. I've tested that perl -pne "s/\x7b\x0a/\x7b/g" < a > output does remove the \x0a newline character, but it does so where it shouldn't, and I only want it to remove the trailing \x0a character when there is a sequence of \x7b\x0a\x0a – Aaron Franke May 3 at 2:22

An alternative to the perl call is to use sed

sed -i.bckp -e "s/\xc3\xb9/\xd7\xa0/" file

Using the stream editor (sed) I have it doing inplace execution (meaning it will modify the file outright) and I supplied .bckp as the argument for -i so that the old file would be backed up. Then I run a swap expression which looks for your hex bytes with regex and swaps it out with the bytes you want to be using. And the call to file is the large file you wish to do this with.

If you want inplace replacement without backing it up (e.g. the file is in a repo), then you can just remove the .bckp after the -i and it will not create any new files.

| improve this answer | |
  • How do I make sed modify newline characters (\x0a)? The command sed -i -e "s/\x0a//g" file does nothing. If I use literally any character other than \x0a it works fine. – Aaron Franke May 3 at 2:27
  • try using "s/\n//g" instead (see Reg Expression section in Manpage). if that doesn't work, confirm the hex value (depending on OS, it could be a carriage return instead of a newline) that is in the file. – sparticvs May 6 at 2:03
  • I figured out the solution is to use sed -z, since this causes sed to delimit lines with null characters instead of newline characters, so the whole file is treated as one massive line and therefore I can make sed edit newline characters. – Aaron Franke May 7 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.