Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to replace this with "D7 A0 D7 A8 D7 9B D7 A9". I would need to replace every "C3 B9" with "D7 A0" throughout the file. There will be about 30 different "C3 XX" values to be replaced by "D7 YY" values. I would need a table to loop through for each replacement.

Ideally this would replace the original file but it can also create a new output file.

Is there a utility that runs under Ubuntu that will do this job?

share|improve this question
    
I can't help directly, but people report using python which handles Unicode for this kind of thing. –  EightBitTony Oct 23 '12 at 13:12
1  
Is that Unicode encoded as UTF-8 (so e.g. C3B9 is U+00F9 LATIN SMALL LETTER U WITH GRAVE (ù))? Or as UCS-2 (so e.g. C3B9 is HANGUL SYLLABLE SSYEOT)? –  Gilles Oct 23 '12 at 22:18
add comment

2 Answers

That's possible with perl,

A sample file:

echo -e "\xe8\x90" > a

And now if you do:

perl -pne 's/\xe8\x90/\x41\x42/g' < a > output

You get 0x41 0x42, which is AB

share|improve this answer
add comment

An alternative to the perl call is to use sed

sed -i.bckp -e "s/\xc3\xb9/\xd7\xa0/" file

Using the stream editor (sed) I have it doing inplace execution (meaning it will modify the file outright) and I supplied .bckp as the argument for -i so that the old file would be backed up. Then I run a swap expression which looks for your hex bytes with regex and swaps it out with the bytes you want to be using. And the call to file is the large file you wish to do this with.

If you want inplace replacement without backing it up (e.g. the file is in a repo), then you can just remove the .bckp after the -i and it will not create any new files.

Hope this helps.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.