XML text file with `^@` characters in it?

Question

I have an XML file that I need to parse. When I open it in nano, nano give me the message (converted from Mac format). However between each character, there is a ^@ sequence, like so:

^@t^@h^@e^@ ^@q^@u^@i^@c^@k^@ ...

What format is this, and how do I get it to display properly? How do I convert it?

Answer 1

It is a double byte format, probably UTF-16. See if you can identify a BOM (Byter Order Mark) header at the beginning of the file. This will tell you the encoding if it is present, although it may not be. Note that a text editor may hide this from you, so you will probably need to look at the file with a hex dump utility such as od or something similar to see the BOM.

A few character set conversion utilities exist that may be of help. A couple of examples are iconv and recode. You may be able to use one of these to re-code the file into a format that will play nicely with your text editor.

If you need to do the translation programatically, most languages such as Python have libraries for mapping between coding schemes. www.unicodetools.com shows some of the relevant incantations for various languages. A good Unicode FAQ can be found here and the Unicode Consortium's web site has many, many useful resources on the subject.