1

I'm having some peculiarities with the dictionary file of .dsl format I'm trying to convert. It's essentially a text file with the dictionary pairs. The dictionary software I use is GoldenDict. It requires UTF-16 dictionaries so they render properly.

All the dictionaries I have are UTF-16LE format. There is one standing out however. It has iso-8859-1 encoding. An entry looks like this when I open it with vim:

abandonarse
    [m2][c crimson][b]Sinónimos[/b][/c][/m]
    [m2][i][c green]verbo[/c][/i][/m]
    [m1][trn][b]desanimarse:[/b] <<desanimarse>>, <<abatirse>>, <<tumbarse>>, <<plegarse>>, <<entregarse>>, <<desligarse>>[/trn][/m]

I have to convert it to UTF-16LE because Goldendict renders some Cyrillic characters instead of Spanish accented characters. Then I try:

iconv -f iso-8859-1 -t utf-16le dictionary.dsl -o test.dsl

The new test.dsl dictionary is rendered correctly by Goldendict, however I can see some peculiar things I would love to get rid of. First is that the just converted file's encoding is not recognized as it usually is with the other dictionaries:

    aleksandr@desktop:~/windoc/Dic/Es extra/dictionary.dsl> file dictionary.dsl
dictionary: data

When I open the file test.dsl with vim every character inside has ^@ added to it. Here is the example of the same entry:

    ^@<^@<^@e^@n^@t^@r^@e^@g^@a^@r^@s^@e^@>^@>^@,^@ ^@<^@<^@d^@e^@s^@l^@i^@g^@a^@r^@s^@e^@>^@>^@[^@/^@t^@r^@n^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@c^@ ^@c^@r^@i^@m^@s^@o^@n^@]^@[^@b^@]^@A^@n^@t^@ó^@n^@i^@m^@o^@s^@[^@/^@b^@]^@[^@/^@c^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@i^@]^@[^@c^@ ^@g^@r^@e^@e^@n^@]^@v^@e^@r^@b^@o^@[^@/^@c^@]^@[^@/^@i^@]^@[^@/^@m^@]^@

I tried removing this characters in vim

%s/<Ctrl-V><Ctrl-J>//g

However, then I save the file, it has the encoding iso-8859-1 again. I would like to have this file to be show without ^@ characters, because I may need to edit some headings in the dictionary manually.

2
  • 1
    The real question here is How do I get VIM to correctly recognize my text file as UTF-16LE (or ucs2-le) when the file does not have a BOM?, iconv not adding BOMs when it is explicitly told the endinanness to use. – JdeBP Sep 8 '20 at 15:54
  • How do I get file to correctly recognize my text file as UTF-16LE when the file does not have a BOM? is really a separate question. – JdeBP Sep 8 '20 at 16:00
3

Your vim hasn't recognised the encoding, and is showing the 16-bit characters as 8-bit characters. The ^@ markers represent the higher order 8-bits, which for common Latin characters will be zero valued.

You can type this after reading in the file to force recognition of UTF-16LE

:e ++enc=utf-16le

(Credit: StackOverflow)

It seems that this would also work, but whether it's ideal is far from clear to me

vim -c 'e ++enc=utf-16le' dictionary.dsl

Finally, from your comment it seems that a BOM would be quite acceptable. You can't use iconv to add a BOM but you can add one yourself

(
    printf "%s" $'\xFF\xFE'
    iconv -f iso-8859-1 -t utf-16le dictionary.dsl
) > dictionary-utf16le.dsl

Confirm with

file dictionary-utf16le.dsl
dictionary-utf16le.dsl: Little-endian UTF-16 Unicode text
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.