4

I have a CSV file which is in binary character set but I have to convert to UTF-8 to process in HDFS (Hadoop).

I have used the below command to check characterset.

file -bi filename.csv

Output :

application/octet-stream; charset=binary

when I try to convert the file from binary to UTF-8 it throws error.

iconv -f binary -t utf-8 fiename.csv
iconv: conversion from binary' is not supported
Try iconv --help' or iconv --usage' for more information.

can anyone please help me to understand is it possible to convert or not, I can able to see the data using head command.

What does it mean , Binary means non-readable but how head command or notepad can read the data.

od -tc < filename.csv | head

0000000 357 273 277   |   |   R   e   q   u   e   s   t   _   I   D   #
0000020   D   #   T   y   p   e   #   D   #   S   u   b   m   i   t   t
0000040   e   r   #   D   #   S   h   o   r   t   _   D   e   s   c   r
0000060   i   p   t   i   o   n   #   D   #   L   o   g   _   T   e   x
0000100   t   #   D   #   S   t   a   t   u   s   #   D   #   A   s   s
0000120   i   g   n   e   d   _   T   o   #   D   #   A   s   s   i   g
0000140   n   e   e   #   D   #   C   r   e   a   t   e   _   D   a   t
0000160   e   #   D   #   F   o   r   w   T   o   E   x   t   H   D   #
0000200   D   #   L   a   s   t   _   M   o   d   i   f   i   e   d   _
0000220   B   y   #   D   #   L   o   g   _   I   D   #   D   #   L   o
  • Can you please add the output of od -tc < yourfile.csv | head to your question? – Stéphane Chazelas Aug 18 '16 at 8:58
  • file -bi shows its binary character , When I check from Vi set characterencoding shows utf-8. – William R Aug 18 '16 at 9:47
  • That od output shows a UTF-8 file with BOM. For file to report binary, there must be some non-text characters (likely at least one NUL byte followed by some non-NUL bytes) past that first section. – Stéphane Chazelas Aug 18 '16 at 10:08
  • I have removed all "NUL" after that file -bi shows as utf-8 , Hope removing "NUL" will not be a problem. – William R Aug 18 '16 at 10:13
11

"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job.

The file utility doesn't give useful information when it doesn't recognize the file format. It could be UTF-16 for example, without a byte-encoding-mark (BOM). notepad reads that. The same applies to UTF-8 (and head would display that since your terminal may be set to UTF-8 encoding, and it would not care about a BOM).

If the file is UTF-16, your terminal would display that using head because most of the characters would be ASCII (or even Latin-1), making the "other" byte of the UTF-16 characters a null.

In either case, the lack of BOM will (depending on the version of file) confuse it. But other programs may work, because these file formats can be used with Microsoft Windows as well as portable applications that may run on Windows.

To convert the file to UTF-8, you have to know which encoding it uses, and what the name for that encoding is with iconv. If it is already UTF-8, then whether you add a BOM (at the beginning) is optional. UTF-16 has two flavors, according to which byte is first. Or you could even have UTF-32. iconv -l lists these:

ISO-10646/UTF-8/
ISO-10646/UTF8/
UTF-7//
UTF-8//
UTF-16//
UTF-16BE//
UTF-16LE//
UTF-32//
UTF-32BE//
UTF-32LE//
UTF7//
UTF8//
UTF16//
UTF16BE//
UTF16LE//
UTF32//
UTF32BE//
UTF32LE//

"LE" and "BE" refer to little-end and big-end for the byte-order. Windows uses the "LE" flavors, and iconv likely assumes that for the flavors lacking "LE" or "BE".

You can see this using an octal (sic) dump:

$ od -bc big-end
0000000 000 124 000 150 000 165 000 040 000 101 000 165 000 147 000 040
         \0   T  \0   h  \0   u  \0      \0   A  \0   u  \0   g  \0    
0000020 000 061 000 070 000 040 000 060 000 065 000 072 000 060 000 061
         \0   1  \0   8  \0      \0   0  \0   5  \0   :  \0   0  \0   1
0000040 000 072 000 065 000 067 000 040 000 105 000 104 000 124 000 040
         \0   :  \0   5  \0   7  \0      \0   E  \0   D  \0   T  \0    
0000060 000 062 000 060 000 061 000 066 000 012
         \0   2  \0   0  \0   1  \0   6  \0  \n
0000072

$ od -bc little-end
0000000 124 000 150 000 165 000 040 000 101 000 165 000 147 000 040 000
          T  \0   h  \0   u  \0      \0   A  \0   u  \0   g  \0      \0
0000020 061 000 070 000 040 000 060 000 065 000 072 000 060 000 061 000
          1  \0   8  \0      \0   0  \0   5  \0   :  \0   0  \0   1  \0
0000040 072 000 065 000 067 000 040 000 105 000 104 000 124 000 040 000
          :  \0   5  \0   7  \0      \0   E  \0   D  \0   T  \0      \0
0000060 062 000 060 000 061 000 066 000 012 000
          2  \0   0  \0   1  \0   6  \0  \n  \0
0000072

Assuming UTF-16LE, you could convert using

iconv -f UTF-16LE// -t UTF-8// <input >output
  • So how do I can convert the file to utf-8 , My intension is to convert any format(comes from source) to utf-8. – William R Aug 18 '16 at 8:49
  • @StéphaneChazelas ,@Thomas how can I find my source encoding, it doesn't work for utf-32 (le/be) | 16(le/be). Is there any command to find the source file encoding? – William R Aug 18 '16 at 9:29
  • As suggested, showing an octal dump would help with advice... – Thomas Dickey Aug 18 '16 at 9:31
  • @StéphaneChazelas The below worked but what does "NUL" is it good to remove ?? cat filename.csv | tr -d '\000' > out.csv file -bi out.csv text/plain; charset=utf-8 – William R Aug 18 '16 at 9:56
  • @WilliamR, those NUL bytes in your csv file are probably an indication of a wider problem, like the file was corrupted in some way as those are not expected in text file, and that's what you should be looking at. Look at where they appear in the csv file (though should show as ^@ in vim) and see if the data around it is valid. – Stéphane Chazelas Aug 18 '16 at 10:29
0

https://pypi.python.org/pypi/chardet can be used to determine the encoding of your text, then you can convert from that to what you need.

0

strings (from binutils) succeeds to "print the strings of printable characters in files" when both iconv and recode failed as well, with file still reporting the content as binary data:

$ file -i /tmp/textFile
/tmp/textFile: application/octet-stream; charset=binary

$ chardetect /tmp/textFile
/tmp/textFile: utf-8 with confidence 0.99

$ iconv -f utf-8 -t utf-8 /tmp/textFile -o /tmp/textFile.iconv
$ file -i /tmp/textFile.iconv
/tmp/textFile.iconv: application/octet-stream; charset=binary

$ cp /tmp/textFile /tmp/textFile.recode ; recode utf-8 /tmp/textFile.recode
$ file -i /tmp/textFile.recode 
/tmp/textFile.recode: application/octet-stream; charset=binary

$ strings /tmp/textFile > /tmp/textFile.strings
$ file -i /tmp/textFile.strings
/tmp/textFile.strings: text/plain; charset=us-ascii

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.