2

I have a requirement to convert from ASCII text format to UTF-8.

Below is what I am performing through the iconv command:

[root@main tmp]# cat File1
1
5
6
[root@main tmp]# file File1
File1: ASCII text
[root@main tmp]# iconv -f ascii -t utf-8 File1 > File2
[root@main tmp]# file File2
File2: ASCII text

(Still ASCII not utf-8)

Any suggestions on how to convert it from ascii to utf-8?

closed as unclear what you're asking by Gilles, slm, Renan, jasonwryan, Anthon Aug 10 '13 at 4:17

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    ASCII 127 is a subset of utf-8. see wikipedia – dor Aug 9 '13 at 7:25
  • 3
    Indeed, "cp" is an efficient ASCII to UTF-8 converter. – jlliagre Aug 9 '13 at 8:00
  • Your ASCII file is already in UTF-8. What are you trying to do? – Gilles Aug 9 '13 at 22:21
5

Two things that are relevant here:

  1. the stock file utility on Solaris sucks
  2. 7-bit ASCII characters are byte compatibly included in UTF-8. That means that when your input file just contains 7-bit ASCII characters no actual conversion takes place. And even a good file utility would display ASCII.

Thus, you probably want to convert a file in some sort of 'extended' 8 byte ASCII encoding. For example latin1. Then you have to specify it with iconv, e.g.:

$ iconv -f latin1 -t utf8 file1 > file2

You then can compare the output like this:

$ cmp file1 file2
$ hexdump ...
$ $EDITOR file2

Editors like vim provide some commands to look at the byte values of certain characters, change the used encoding on the fly etc.

1

Should you really want file to state your ASCII file is UTF-8, which it is already anyway, you can run this command which prepends an UTF-8 BOM (Byte Order Mark) to it:

(printf "\357\273\277";cat File1) > File2

Edit: a couple of issues though:

  • This will work with the find version usually provided with Linux distributions but Solaris file won't detect the file as UTF-8 but just "data"

  • The BOM might confuse tools used to process the text file

Not the answer you're looking for? Browse other questions tagged or ask your own question.