2

I have a requirement to convert from ASCII text format to UTF-8.

Below is what I am performing through the iconv command:

[root@main tmp]# cat File1
1
5
6
[root@main tmp]# file File1
File1: ASCII text
[root@main tmp]# iconv -f ascii -t utf-8 File1 > File2
[root@main tmp]# file File2
File2: ASCII text

(Still ASCII not utf-8)

Any suggestions on how to convert it from ascii to utf-8?

  • 5
    ASCII 127 is a subset of utf-8. see wikipedia – dor Aug 9 '13 at 7:25
  • 3
    Indeed, "cp" is an efficient ASCII to UTF-8 converter. – jlliagre Aug 9 '13 at 8:00
  • Your ASCII file is already in UTF-8. What are you trying to do? – Gilles 'SO- stop being evil' Aug 9 '13 at 22:21
5

Two things that are relevant here:

  1. the stock file utility on Solaris sucks
  2. 7-bit ASCII characters are byte compatibly included in UTF-8. That means that when your input file just contains 7-bit ASCII characters no actual conversion takes place. And even a good file utility would display ASCII.

Thus, you probably want to convert a file in some sort of 'extended' 8 byte ASCII encoding. For example latin1. Then you have to specify it with iconv, e.g.:

$ iconv -f latin1 -t utf8 file1 > file2

You then can compare the output like this:

$ cmp file1 file2
$ hexdump ...
$ $EDITOR file2

Editors like vim provide some commands to look at the byte values of certain characters, change the used encoding on the fly etc.

| improve this answer | |
1

Should you really want file to state your ASCII file is UTF-8, which it is already anyway, you can run this command which prepends an UTF-8 BOM (Byte Order Mark) to it:

(printf "\357\273\277";cat File1) > File2

Edit: a couple of issues though:

  • This will work with the find version usually provided with Linux distributions but Solaris file won't detect the file as UTF-8 but just "data"

  • The BOM might confuse tools used to process the text file

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.