I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error:

"Your file is not "us-ascii" encoded. Please specify the correct encoding with the -e flag or with the PYTHONIOENCODING environment variable"

csvkit documentation can be found here.

I also found the file has HEX codes like 0xd1, which caused some issues. So how do I find the correct encoding of this file? Ideally I would like to convert it to utf-8 encoding. What shall do?

  • For example include a part of your txt file in your question. – Ipor Sircer Sep 10 '18 at 19:04
  • Hi @IporSircer, to prepare the example file, I took a small part of the file out and made sure it contains the row of 0xd1 which caused trouble. Then I re-ran the file -i command and this time it says the encoding is iso-8559-1. And that seems to be the correct encoding for the original file. So I guess the file -i only looks at a portion of a file and draws a conclusion? – user3768495 Sep 10 '18 at 19:32
  • Bottom line? Determining the correct encoding of a file is actually very, very difficult, unfortunately.If Python is an option, you can try chardet which gives great results in my opinion. FYI even chardet will give you results with a confidence interval (which is pretty smart!). pypi.org/project/chardet – pi0tr Sep 10 '18 at 20:45
  • Please, could you share the file? – aborruso Dec 29 '18 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.