# How do I properly convert the file to UTF-16LE encoding without strange characters appearing in the file?

I'm having some peculiarities with the dictionary file of .dsl format I'm trying to convert. It's essentially a text file with the dictionary pairs. The dictionary software I use is GoldenDict. It requires UTF-16 dictionaries so they render properly.

All the dictionaries I have are UTF-16LE format. There is one standing out however. It has iso-8859-1 encoding. An entry looks like this when I open it with vim:

abandonarse
[m2][c crimson][b]Sinónimos[/b][/c][/m]
[m2][i][c green]verbo[/c][/i][/m]
[m1][trn][b]desanimarse:[/b] <<desanimarse>>, <<abatirse>>, <<tumbarse>>, <<plegarse>>, <<entregarse>>, <<desligarse>>[/trn][/m]


I have to convert it to UTF-16LE because Goldendict renders some Cyrillic characters instead of Spanish accented characters. Then I try:

iconv -f iso-8859-1 -t utf-16le dictionary.dsl -o test.dsl


The new test.dsl dictionary is rendered correctly by Goldendict, however I can see some peculiar things I would love to get rid of. First is that the just converted file's encoding is not recognized as it usually is with the other dictionaries:

    aleksandr@desktop:~/windoc/Dic/Es extra/dictionary.dsl> file dictionary.dsl
dictionary: data


When I open the file test.dsl with vim every character inside has ^@ added to it. Here is the example of the same entry:

    ^@<^@<^@e^@n^@t^@r^@e^@g^@a^@r^@s^@e^@>^@>^@,^@ ^@<^@<^@d^@e^@s^@l^@i^@g^@a^@r^@s^@e^@>^@>^@[^@/^@t^@r^@n^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@c^@ ^@c^@r^@i^@m^@s^@o^@n^@]^@[^@b^@]^@A^@n^@t^@ó^@n^@i^@m^@o^@s^@[^@/^@b^@]^@[^@/^@c^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@i^@]^@[^@c^@ ^@g^@r^@e^@e^@n^@]^@v^@e^@r^@b^@o^@[^@/^@c^@]^@[^@/^@i^@]^@[^@/^@m^@]^@


I tried removing this characters in vim

%s/<Ctrl-V><Ctrl-J>//g


However, then I save the file, it has the encoding iso-8859-1 again. I would like to have this file to be show without ^@ characters, because I may need to edit some headings in the dictionary manually.

• The real question here is How do I get VIM to correctly recognize my text file as UTF-16LE (or ucs2-le) when the file does not have a BOM?, iconv not adding BOMs when it is explicitly told the endinanness to use. – JdeBP Sep 8 '20 at 15:54
• How do I get file to correctly recognize my text file as UTF-16LE when the file does not have a BOM? is really a separate question. – JdeBP Sep 8 '20 at 16:00

Your vim hasn't recognised the encoding, and is showing the 16-bit characters as 8-bit characters. The ^@ markers represent the higher order 8-bits, which for common Latin characters will be zero valued.

You can type this after reading in the file to force recognition of UTF-16LE

:e ++enc=utf-16le


(Credit: StackOverflow)

It seems that this would also work, but whether it's ideal is far from clear to me

vim -c 'e ++enc=utf-16le' dictionary.dsl


Finally, from your comment it seems that a BOM would be quite acceptable. You can't use iconv to add a BOM but you can add one yourself

(
printf "%s" \$'\xFF\xFE'
iconv -f iso-8859-1 -t utf-16le dictionary.dsl
) > dictionary-utf16le.dsl


Confirm with

file dictionary-utf16le.dsl
dictionary-utf16le.dsl: Little-endian UTF-16 Unicode text