Hot answers tagged

92

POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) and bash mailing lists). The printf builtin of bash and most other POSIX shells honour that. zsh ignores that silly requirement (even in sh emulation) so printf ...


89

If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of sed) will remove the BOM if it exists, or make no changes if it doesn't. sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt You can also overwrite the existing file with the -i option: sed -i '1s/^\xEF\xBB\xBF//' orig.txt If you are using the BSD version ...


77

A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes. dos2unix will remove it and also take care of other idiosyncrasies of Windows text files. dos2unix test.xml


67

No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same. If you look at /usr/share/i18n/locales/iso14651_t1_common (as used as basis for most locales) on a GNU system (here with glibc 2.27), you'll see: <U0065> <e>;<BAS>;<MIN>;IGNORE # 259 e <U025B>...


55

A useful guideline for this is the "Portable Operating System Interface" (POSIX), a family of standards that is implemented by most Unix-like systems. It is usually a good idea to limit shell scripts to features mandated by POSIX to make sure they will be usable across different shells and platforms. According to the POSIX specification of function ...


39

If you want to use grep, you can do: grep -axv '.*' file in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).


38

The same exact thing happened to me. Building on what Thomas said above, I was able to fix it by uncommenting en_US.UTF-8 UTF-8 in my /etc/locale.gen file (previously none of the lines had been uncommented), then running locale-gen.


37

Using VIM Open file in VIM: vi text.xml Remove BOM encoding: :set nobomb Save and quit: :wq


33

file will tell you if there is a BOM. You can test: $ /usr/bin/printf "\ufeff...\n" | file - /dev/stdin: UTF-8 Unicode (with BOM) text Note: according to the file changelog, this feature existed already in 2007. So, this should work on any current machine.


31

Just use that syntax: sed 's/馑//g' file1 Or in the escaped form: sed "s/$(echo -ne '\u9991')//g" file1 (Note that older versions of Bash and some shells do not understand echo -e '\u9991', so check first.)


31

That's a known (1, 2, 3, 4, 5, 6) limitation of the GNU implementation of tr. It's not as much that it doesn't support foreign, non-English or non-ASCII characters, but that it doesn't support multi-byte characters. Those Cyrillic characters would be treated OK, if written in the iso8859-5 (single-byte per character) character set (and your locale was ...


29

It is possible to remove the BOM from a file with the tail command: tail -c +4 withBOM.txt > withoutBOM.txt


27

Since it's a fixed set of numbers, you can do it by hand: $ echo ۲۱ | LC_ALL=en_US.UTF-8 sed -e 'y/۰۱۲۳۴۵۶۷۸۹/0123456789/' 21 (or using tr, but not GNU tr yet) Setting your locale to en_US.utf8 (or better to the locale which characters set belongs to) is required for sed to recognize your characters set. With perl: $ echo "۲۱" | perl -CS -MUnicode::...


27

This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). That isn't a useful character for any modern system, but it's unlikely to be harmful that it's there. The original source for this was likely a byte 0x96 in ...


27

help printf defers to printf(1) for the escape sequences interpreted, and the docs for GNU printf says: printf interprets two character syntaxes introduced in ISO C 99: \u for 16-bit Unicode (ISO/IEC 10646) characters, specified as four hexadecimal digits hhhh, and \U for 32-bit Unicode characters, specified as eight hexadecimal digits hhhhhhhh. ...


26

If you look at the source code of Apple macOS sort, it's little modified from its FreeBSD origin. In particular, you find the same awkward/broken handling of the -t option as in FreeBSD. case 't': while (strlen(optarg) > 1) { if (optarg[0] != '\\') { errc(2, EINVAL, "%s", optarg); ...


23

Look at grep: Find all lines that contain Japanese kanjis. Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in UTF-8 encoding. grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]" and grep '[अ-व]' are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of Devanagari script ...


22

You can use the uconv utility from ICU. Normalization is achieved through transliteration (-x). $ uconv -x any-nfd <<<ä | hd 00000000 61 cc 88 0a |a...| 00000004 $ uconv -x any-nfc <<<ä | hd 00000000 c3 a4 0a |...| 00000003 On Debian, Ubuntu and other ...


22

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë: $ printf "\353\n" | iconv -f ISO-8859-1 ë Other candidates would be δ in code page 437, Ù in code page 850... od -x’s output is confusing in this case because of endianness; a ...


20

It's a apparently a known bug: No characters beyond the BMP are displayed, as screen apparently only has a two byte buffer for characters. (It works in tmux).


20

When using vim -b, this displays all high characters as <xx>: set encoding=latin1 set isprint= set display+=uhex Any single-byte encoding will work, vim uses ASCII for all lower chars and has them hard-coded as printable. Setting isprint to empty will mark everything else as non-printable. Setting uhex will display them as hexadecimal. Here is how ...


20

It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc). Is there another decompression utility that will decompress my files with the correct names? I'm not aware of a zip utility that supports these code pages natively. 7z has some understanding of encodings, but I believe it has to be an encoding your system ...


20

UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding. To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just: $ printf '\U1F615\n' 😕 $...


19

On Unix/Linux, a filename is a sequence of any bytes except for a slash or a NUL. A slash separates path components, and a NUL terminates a path name. So, you can use whatever encoding you want for filenames. Some applications may have trouble with some encodings if they are naïve about what characters may be in filenames - for example, poorly-written shell ...


19

The file is encoded in ISO-8859-1, not in UTF-8: $ hd 0606461.txt | grep -B1 '^0002c520' 0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin| 0002c520 e6 0d 0a 2d 2d 48 6f 72 61 63 65 2e 0d 0a 0d 0a |...--Horace.....| And the byte "e6" alone is not a valid UTF-8 sequence. So, use iconv -f latin1 -t ascii//TRANSLIT file.


17

I came across this in 2016. A single TTF/OTF font is never going to cover all utf-8 characters. There is a hard limit of 65535 glyphs in a font, and over 1 million utf-8 glphys. You will need to use a font-family for this to work. A good font-family is the noto font family: https://www.google.com/get/noto/


16

From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8: iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt


16

Perl can do that: echo 汉典“馑”字的基本解释 | perl -CS -pe 's/\N{U+9991}/Jin/g' -CS turns on UTF-8 for standard input, output and error.


15

I assume you mean UTF-8 encoded Unicode characters. That depends what you mean by invalid. invalid_byte_sequence=$'\x80\x81' That's a sequence of bytes that, by itself, isn't valid in UTF-8 encoding (the first byte in a UTF-8 encoded character always has the two highest bits set). That sequence could be seen in the middle of a character though, so it ...


15

For Python there is the unidecode library which handles such conversions in general: https://pypi.python.org/pypi/Unidecode. In Python 2: >>> from unidecode import unidecode >>> unidecode(u"۰۱۲۳۴۵۶۷۸۹") '0123456789' In Python 3: >>> from unidecode import unidecode >>> unidecode("۰۱۲۳۴۵۶۷۸۹") '0123456789' The SO thread ...


Only top voted, non community-wiki answers of a minimum length are eligible