The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

104

You can use any printable character, bash doesn't mind. You'll probably want to configure your terminal to support Unicode (in the form of UTF-8). There are a lot of characters in Unicode, so here are a few tips to help you search through the Unicode charts: You can try to draw the character on Shapecatcher. It tries to recognize a Unicode character in ...


87

POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) and bash mailing lists). The printf builtin of bash and most other POSIX shells honour that. zsh ignores that silly requirement (even in sh emulation) so ...


75

If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of sed) will remove the BOM if it exists, or make no changes if it doesn't. sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt You can also overwrite the existing file with the -i option: sed -i '1s/^\xEF\xBB\xBF//' orig.txt


68

No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same. If you look at /usr/share/i18n/locales/iso14651_t1_common (as used as basis for most locales) on a GNU system (here with glibc 2.27), you'll see: <U0065> <e>;<BAS>;<MIN>;IGNORE # 259 e <U025B>...


64

When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on. By default, the detection process is crude. Every file that you open ...


63

A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes. dos2unix will remove it and also take care of other idiosyncrasies of Windows text files. dos2unix test.xml


55

A useful guideline for this is the "Portable Operating System Interface" (POSIX), a family of standards that is implemented by most Unix-like systems. It is usually a good idea to limit shell scripts to features mandated by POSIX to make sure they will be usable across different shells and platforms. According to the POSIX specification of function ...


34

If you want to use grep, you can do: grep -axv '.*' file in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).


31

The same exact thing happened to me. Building on what Thomas said above, I was able to fix it by uncommenting en_US.UTF-8 UTF-8 in my /etc/locale.gen file (previously none of the lines had been uncommented), then running locale-gen.


29

file will tell you if there is a BOM. You can test: $ /usr/bin/printf "\ufeff...\n" | file - /dev/stdin: UTF-8 Unicode (with BOM) text Note: according to the file changelog, this feature existed already in 2007. So, this should work on any current machine.


29

That's a known (1, 2, 3, 4, 5, 6) limitation of the GNU implementation of tr. It's not as much that it doesn't support foreign, non-English or non-ASCII characters, but that it doesn't support multi-byte characters. Those Cyrillic characters would be treated OK, if written in the iso8859-5 (single-byte per character) character set (and your locale was ...


27

Just use that syntax: sed 's/馑//g' file1 Or in the escaped form: sed "s/$(echo -ne '\u9991')//g" file1 (Note that older versions of Bash and some shells do not understand echo -e '\u9991', so check first.)


27

Since it's a fixed set of numbers, you can do it by hand: $ echo ۲۱ | LC_ALL=en_US.UTF-8 sed -e 'y/۰۱۲۳۴۵۶۷۸۹/0123456789/' 21 (or using tr, but not GNU tr yet) Setting your locale to en_US.utf8 (or better to the locale which characters set belongs to) is required for sed to recognize your characters set. With perl: $ echo "۲۱" | perl -CS -MUnicode::...


27

This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). That isn't a useful character for any modern system, but it's unlikely to be harmful that it's there. The original source for this was likely a byte 0x96 in ...


27

help printf defers to printf(1) for the escape sequences interpreted, and the docs for GNU printf says: printf interprets two character syntaxes introduced in ISO C 99: \u for 16-bit Unicode (ISO/IEC 10646) characters, specified as four hexadecimal digits hhhh, and \U for 32-bit Unicode characters, specified as eight hexadecimal digits hhhhhhhh. ...


25

It is possible to remove the BOM from a file with the tail command: tail -c +4 withBOM.txt > withoutBOM.txt


22

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë: $ printf "\353\n" | iconv -f ISO-8859-1 ë Other candidates would be δ in code page 437, Ù in code page 850... od -x’s output is confusing in this case because of endianness; a ...


21

Look at this question. Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in utf-8 encoding. grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]" and grep '[अ-व]' are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of devanagari script (that is, the matching is NOT "any ...


20

You can use the uconv utility from ICU. Normalization is achieved through transliteration (-x). $ uconv -x any-nfd <<<ä | hd 00000000 61 cc 88 0a |a...| 00000004 $ uconv -x any-nfc <<<ä | hd 00000000 c3 a4 0a |...| 00000003 On Debian, Ubuntu and other ...


20

UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding. To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just: $ printf '\U1F615\n' 😕 $...


19

On Unix/Linux, a filename is a sequence of any bytes except for a slash or a NUL. A slash separates path components, and a NUL terminates a path name. So, you can use whatever encoding you want for filenames. Some applications may have trouble with some encodings if they are naïve about what characters may be in filenames - for example, poorly-written shell ...


19

Using VIM Open file in VIM: vi text.xml Remove BOM encoding: :set nobomb Save and quit: :wq


18

It's a apparently a known bug: No characters beyond the BMP are displayed, as screen apparently only has a two byte buffer for characters. (It works in tmux).


16

When using vim -b, this displays all high characters as <xx>: set encoding=latin1 set isprint= set display+=uhex Any single-byte encoding will work, vim uses ASCII for all lower chars and has them hard-coded as printable. Setting isprint to empty will mark everything else as non-printable. Setting uhex will display them as hexadecimal. Here is how ...


16

The file is encoded in ISO-8859-1, not in UTF-8: $ hd 0606461.txt | grep -B1 '^0002c520' 0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin| 0002c520 e6 0d 0a 2d 2d 48 6f 72 61 63 65 2e 0d 0a 0d 0a |...--Horace.....| And the byte "e6" alone is not a valid UTF-8 sequence. So, use iconv -f latin1 -t ascii//TRANSLIT file.


15

I came across this in 2016. A single TTF/OTF font is never going to cover all utf-8 characters. There is a hard limit of 65535 glyphs in a font, and over 1 million utf-8 glphys. You will need to use a font-family for this to work. A good font-family is the noto font family: https://www.google.com/get/noto/


15

Perl can do that: echo 汉典“馑”字的基本解释 | perl -CS -pe 's/\N{U+9991}/Jin/g' -CS turns on UTF-8 for standard input, output and error.


15

It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc). Is there another decompression utility that will decompress my files with the correct names? I'm not aware of a zip utility that supports these code pages natively. 7z has some understanding of encodings, but I believe it has to be an encoding your system ...


15

For Python there is the unidecode library which handles such conversions in general: https://pypi.python.org/pypi/Unidecode. In Python 2: >>> from unidecode import unidecode >>> unidecode(u"۰۱۲۳۴۵۶۷۸۹") '0123456789' In Python 3: >>> from unidecode import unidecode >>> unidecode("۰۱۲۳۴۵۶۷۸۹") '0123456789' The SO thread ...


15

man sort: *** WARNING *** The locale specified by the environment affects sort order. Set LC_ALL=C to get the traditional sort order that uses native byte values. So, try: LC_ALL=C sort file.txt


Only top voted, non community-wiki answers of a minimum length are eligible