Tag Info

New answers tagged


I find uconv (in icu-devtools package in Debian) useful to inspect UTF-8 data: $ print '\\xE9 \xe9 \u20ac \ud800\udc00 \U110000' | uconv --callback escape-c -t us \xE9 \xE9 \u20ac \xED\xA0\x80\xED\xB0\x80 \xF4\x90\x80\x80 (The \xs help spotting the invalid characters (except for the false positive voluntarily introduced with a literal \xE9 above)). ...


Python has had a built-in unicode function since version 2.0. #!/usr/bin/env python2 import sys for line in sys.stdin: try: unicode(line, 'utf-8') except UnicodeDecodeError: sys.stdout.write(line) In Python 3, unicode has been folded into str. It needs to be passed a bytes-like object, here the underlying buffer objects for the ...


If you want to use grep, you can do: grep -av '^.*$' file in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).


Detecting UTF-8 encoding: file will usually give you the encoding; to get a more processable output from file, you've got to pass some options, and cut it until you get the part that describes the encoding: file --brief --mime myfile.txt | cut -d '=' -f 2 Note that it may either be 'us-ascii' or 'utf-8', depending on whether it finds some UTF-8 ...


You should try the file command, it already does a good job of determining information about the contents, although it doesn't analyse the full data file. If you have a large file somefile where the first utf-8 character doesn't appear until the end, file somefile will not detect that as an utf-8 file.


This will work for some things: iconv -f utf-8 -t ascii//TRANSLIT echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks. iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. An inferior but ...


You can use iconv or recode to convert the file. But you will need to specify the source encoding. The information about the source encoding has to come from somewhere. A plain text file doesn't contain any information about its encoding. Some types of formatted text contain an indication (for example headers in HTML or in LaTeX), but in general, you're on ...

Top 50 recent answers are included