Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I received a .csv file with the FF FE BOM:

$ head -n1 dotan.csv | hd
00000000  ff fe 41 00 64 00 20 00  67 00 72 00 6f 00 75 00  |..A.d. .g.r.o.u.|

When using awk to parse it I'm getting a bunch of null bytes, which I suspect is due to the byte order. How can I swap the byte order on this file (using the CLI) so that normal tools will work with it?

Note that I think that this file is only ASCII characters (except for the BOM), but I cannot confirm that as grep thinks that it is a binary file:

$ grep -P '^[\x00-\x7f]' dotan.csv 
Binary file dotan.csv matches

Searching for the same string in VIM shows every character matching!

Using iconv to convert to ASCII does not get rid of \x00 values, actually it makes the problem worse as now they look like null bytes instead of UTF-8!

$ iconv -f UTF-8 -t ASCII dotan.csv > fixed.txt 
iconv: illegal input sequence at position 0

$ iconv -f UTF-8 -t ASCII//IGNORE dotan.csv > fixed.txt

$ head -n1 fixed.txt | hd
00000000  41 00 64 00 20 00 67 00  72 00 6f 00 75 00 70 00  |A.d. .g.r.o.u.p.|

How can I swap the byte order on this file (using the CLI) so that normal tools will work with it?

share|improve this question
    
The CSV file you created in Windows or Mac? –  cuonglm Jun 15 at 8:25
    
Can you give a portion of file? –  cuonglm Jun 15 at 8:27
    
Here is a link to an anonymized portion of the file which preserves the unique problems with it. Thank you! –  dotancohen Jun 15 at 9:04

1 Answer 1

up vote 8 down vote accepted

From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8:

iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt
share|improve this answer
    
Perfect, thank you! I had the UTF-8 and UTF-16 BOM mixed up: I thought that FFFE and FEFF were UTF-8 and I never knew the UTF-16 BOM(s). Actually, those are UTF-16 BOMs, and I never knew the (useless) UTF-8 BOM!. –  dotancohen Jun 15 at 9:19
    
@dotancohen: I test in my Fedora and the tail solution works fine. What OS do you use? –  cuonglm Jun 15 at 9:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.