Skip to main content
161 votes
Accepted

How can I remove the BOM from a UTF-8 file?

If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of sed) will remove the BOM if it exists, or make no changes if it doesn't. sed '1s/^\xEF\xBB\xBF//' &...
CSM's user avatar
  • 2,220
129 votes

How can I remove the BOM from a UTF-8 file?

A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes. dos2unix will remove it and also take care of other idiosyncrasies of Windows text files. ...
Stéphane Chazelas's user avatar
97 votes

How can I remove the BOM from a UTF-8 file?

Using VIM Open file in VIM: vi text.xml Remove BOM encoding: :set nobomb Save the file and quit: :x For a non-interactive solution, try the following command line: vi -c ":set nobomb" ...
Joshua Pinter's user avatar
95 votes
Accepted

Why is printf "shrinking" umlaut?

POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) ...
Stéphane Chazelas's user avatar
69 votes
Accepted

Why does sort say that ɛ = e?

No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same. If you look at /usr/share/i18n/locales/iso14651_t1_common (as ...
Stéphane Chazelas's user avatar
56 votes
Accepted

Should we use UTF-8 characters like ⏰ in bash/shell script?

A useful guideline for this is the "Portable Operating System Interface" (POSIX), a family of standards that is implemented by most Unix-like systems. It is usually a good idea to limit shell scripts ...
n.st's user avatar
  • 8,168
43 votes

Find the best font for rendering a codepoint

Using fontconfig, > fc-list ':charset=<hex_code1> <hex_code2>' e.g. > fc-list ':charset=2713 2717' will display any font filenames containing ✓ and ✗. To get the codepoint ...
David Baynard's user avatar
42 votes

gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)

As suggested in https://bugs.launchpad.net/ubuntu/+source/git/+bug/1852985/comments/11, I am able to stop gitk from crashing by installing unifont: sudo apt install unifont Like in the comment, the ...
gsmendoza's user avatar
  • 521
36 votes

How can I remove the BOM from a UTF-8 file?

It is possible to remove the BOM from a file with the tail command: tail -c +4 withBOM.txt > withoutBOM.txt Be aware that this chops the first 3 bytes (-c +N makes the output start at byte nr. N, ...
m13r's user avatar
  • 2,795
34 votes

gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)

After digging with XFT_DEBUG flag I found something weird. I run the command with the flag and navigate to the problematic commit: ❯ XFT_DEBUG=1 gitk --all XFT_DEBUG=1 XftFontInfoFill: /usr/share/...
Édouard Lopez's user avatar
31 votes
Accepted

How to grep characters with their unicode value?

You can use ANSI-C quoting provided by your shell, to replace backslash-escaped characters as specified by the ANSI C standard. This should work for any command, not just grep, in shells like Bash and ...
pLumo's user avatar
  • 22.7k
30 votes

How can I convert Persian numerals in UTF-8 to European numerals in ASCII?

Since it's a fixed set of numbers, you can do it by hand: $ echo ۲۱ | LC_ALL=en_US.UTF-8 sed -e 'y/۰۱۲۳۴۵۶۷۸۹/0123456789/' 21 (or using tr, but not GNU tr yet) Setting your locale to en_US.utf8 (or ...
cuonglm's user avatar
  • 155k
28 votes

How can I enable UTF-8 support in the Linux console?

Check that you have the locales package installed dpkg -l locales If not, install it apt-get install locales As root, type dpkg-reconfigure locales you can navigate that list with the up/down ...
GAD3R's user avatar
  • 67.5k
28 votes
Accepted

Why won't some unicode characters print to my terminal?

help printf defers to printf(1) for the escape sequences interpreted, and the docs for GNU printf says: printf interprets two character syntaxes introduced in ISO C 99: \u for 16-bit Unicode (ISO/...
muru's user avatar
  • 73.9k
27 votes
Accepted

Strange character in a file

This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). ...
Michael Homer's user avatar
27 votes
Accepted

Unix sort does not accept the unconventional delimiter '±' as a field separator

If you look at the source code of Apple macOS sort, it's little modified from its FreeBSD origin. In particular, you find the same awkward/broken handling of the -t option as in FreeBSD. ...
Stéphane Chazelas's user avatar
24 votes
Accepted

UTF-8 characters in POSIX shell script *comments* - anything against it?

POSIX specifies how tokens should be recognised, including comments: If the current character is a '#', it and all subsequent characters up to, but excluding, the next <newline> shall be ...
Stephen Kitt's user avatar
22 votes
Accepted

How can I identify a strange character?

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë: $ printf "\353\n" | ...
Stephen Kitt's user avatar
18 votes
Accepted

Print check/cross mark in shell script

As revealed by the OP in the comments, they were calling script with sh file.sh. Depending on the default shell to which /bin/sh is symlinked it might not support unicode characters. For instance, ...
Sergiy Kolodyazhnyy's user avatar
17 votes

How can I correctly decompress a ZIP archive of files with Hebrew names?

I had success with the command 7z x <source.zip>. Version: p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,[...]) Potentially relevant environment: LANG=en_US.UTF-8 LC_ALL=...
vsz's user avatar
  • 171
16 votes

How can I convert Persian numerals in UTF-8 to European numerals in ASCII?

For Python there is the unidecode library which handles such conversions in general: https://pypi.python.org/pypi/Unidecode. In Python 2: >>> from unidecode import unidecode >>> ...
phk's user avatar
  • 5,983
16 votes

Box character doesn't display properly in Linux terminal

The problem is that you're using literal characters ┌, ─, ┐, └, ─, ┘ instead of using the ncurses-independent symbols. You should use the ACS_??CORNER (for Upper/Lower and Left/Right) and ACS_HLINE ...
Chris Davies's user avatar
16 votes

How should I interpret the fact that a Unicode code point is shown in two completely different ways in two different terminal emulators?

U+E000 to U+F8FF is a private-use area. It's reserved to allow systems to store and display characters that are not present in Unicode. So you can't expect it to have a consistent appearance (or ...
Gilles 'SO- stop being evil''s user avatar
15 votes
Accepted

How can I decode a file name using command line?

The standard (POSIX/Unix) command to get the byte values as hex numbers is od. file=foo.mp3 printf %s "$file" | od -An -vtx1 Which gives an output similar to: 66 6f 6f 2e 6d 70 33 $file above ...
Stéphane Chazelas's user avatar
15 votes
Accepted

tr analog for unicode characters?

GNU sed does work with multi-byte characters. So: $ echo é½Æ | sed 'y/é½Æ/ABŒ/' ABŒ It's not so much that GNU tr hasn't been internationalised but that it doesn't support multi-byte characters (like ...
Stéphane Chazelas's user avatar
15 votes

Why does sort say that ɛ = e?

man sort: *** WARNING *** The locale specified by the environment affects sort order. Set LC_ALL=C to get the traditional sort order that uses native byte values. So, try: LC_ALL=C sort ...
Ipor Sircer's user avatar
  • 14.7k
13 votes

Can not use `cut -c` (`--characters`) with UTF-8?

colrm (part of util-linux, should be already installed on most distributions) seems to handle internationalization much better : $ echo 'αβγ' | colrm 3 αβ $ echo 'αβγ' | colrm 2 α Beware of the ...
Skippy le Grand Gourou's user avatar
13 votes

Why is printf "shrinking" umlaut?

${#var} characters count is correct since bash3.0+. Try (with any version of bash): bash -c "a="$'aáíóuúüoözu\u308\u1100'';printf "%s\n" "${a} ${#a}"' That will give the ...
Léa Gris's user avatar
  • 487
13 votes
Accepted

Sort LC_ALL=C vs LC_ALL=C.utf8

LC_ALL=C sort sorts by byte value. It will sort any input written in any charset by byte value, not only ASCII¹. The UTF-8 encoding has that property that sorting by byte value is the same as sorting ...
Stéphane Chazelas's user avatar
13 votes

gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)

From my research, this appears to be a bug in xorg library "libXft" (see issue libxft#6 and pending merge request libxft!1). Related discussions: Bug report on noto-emoji (noto-emoji#183). ...
David P's user avatar
  • 231

Only top scored, non community-wiki answers of a minimum length are eligible