151
votes
Accepted
How can I remove the BOM from a UTF-8 file?
If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of sed) will remove the BOM if it exists, or make no changes if it doesn't.
sed '1s/^\xEF\xBB\xBF//' &...
- 2,110
126
votes
How can I remove the BOM from a UTF-8 file?
A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes.
dos2unix will remove it and also take care of other idiosyncrasies of Windows text files.
...
- 534k
95
votes
Accepted
Why is printf "shrinking" umlaut?
POSIX requires printf's %-20s to count those 20 in terms of bytes not characters even though that makes little sense as printf is to print text, formatted (see discussion at the Austin Group (POSIX) ...
- 534k
91
votes
How can I remove the BOM from a UTF-8 file?
Using VIM
Open file in VIM:
vi text.xml
Remove BOM encoding:
:set nobomb
Save the file and quit:
:x
For a non-interactive solution, try the following command line:
vi -c ":set nobomb" ...
- 1,160
68
votes
Accepted
Why does sort say that ɛ = e?
No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same.
If you look at /usr/share/i18n/locales/iso14651_t1_common (as ...
- 534k
56
votes
Accepted
Should we use UTF-8 characters like ⏰ in bash/shell script?
A useful guideline for this is the "Portable Operating System Interface" (POSIX), a family of standards that is implemented by most Unix-like systems. It is usually a good idea to limit shell scripts ...
- 7,988
50
votes
Accepted
Updated my arch linux server and now I get tmux: need UTF-8 locale (LC_CTYPE) but have ANSI_X3.4-1968
The same exact thing happened to me. Building on what Thomas said above, I was able to fix it by uncommenting en_US.UTF-8 UTF-8 in my /etc/locale.gen file (previously none of the lines had been ...
41
votes
gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)
As suggested in https://bugs.launchpad.net/ubuntu/+source/git/+bug/1852985/comments/11, I am able to stop gitk from crashing by installing unifont:
sudo apt install unifont
Like in the comment, the ...
- 511
40
votes
Find the best font for rendering a codepoint
Using fontconfig,
> fc-list ':charset=<hex_code1> <hex_code2>'
e.g.
> fc-list ':charset=2713 2717'
will display any font filenames containing ✓ and ✗.
To get the codepoint ...
- 501
34
votes
How can I remove the BOM from a UTF-8 file?
It is possible to remove the BOM from a file with the tail command:
tail -c +4 withBOM.txt > withoutBOM.txt
Be aware that this chops the first 3 bytes (-c +N makes the output start at byte nr. N, ...
- 2,675
34
votes
gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)
After digging with XFT_DEBUG flag I found something weird. I run the command with the flag and navigate to the problematic commit:
❯ XFT_DEBUG=1 gitk --all
XFT_DEBUG=1
XftFontInfoFill: /usr/share/...
- 1,332
30
votes
How can I convert Persian numerals in UTF-8 to European numerals in ASCII?
Since it's a fixed set of numbers, you can do it by hand:
$ echo ۲۱ | LC_ALL=en_US.UTF-8 sed -e 'y/۰۱۲۳۴۵۶۷۸۹/0123456789/'
21
(or using tr, but not GNU tr yet)
Setting your locale to en_US.utf8 (or ...
- 152k
28
votes
How can I enable UTF-8 support in the Linux console?
Check that you have the locales package installed
dpkg -l locales
If not, install it
apt-get install locales
As root, type
dpkg-reconfigure locales
you can navigate that list with the up/down ...
- 64.4k
28
votes
Accepted
How to grep characters with their unicode value?
You can use ANSI-C quoting provided by your shell, to replace backslash-escaped characters as specified by the ANSI C standard. This should work for any command, not just grep, in shells like Bash and ...
- 22.4k
28
votes
Accepted
Why won't some unicode characters print to my terminal?
help printf defers to printf(1) for the escape sequences interpreted, and the docs for GNU printf says:
printf interprets two character syntaxes introduced in ISO C 99:
\u for 16-bit Unicode (ISO/...
- 71.3k
27
votes
Accepted
How can I correctly decompress a ZIP archive of files with Hebrew names?
It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc).
Is there another decompression utility that will decompress my files with the correct names? I'm ...
- 75.7k
27
votes
Accepted
Strange character in a file
This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). ...
- 75.7k
27
votes
Accepted
Unix sort does not accept the unconventional delimiter '±' as a field separator
If you look at the source code of Apple macOS sort, it's little modified from its FreeBSD origin.
In particular, you find the same awkward/broken handling of the -t option as in FreeBSD.
...
- 534k
23
votes
How to convert an emoticon specified by a U+xxxxx code to utf-8?
UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding.
To get ...
- 534k
21
votes
Accepted
How to create string with invalid unicode characters, in Zsh?
I assume you mean UTF-8 encoded Unicode characters.
That depends what you mean by invalid.
invalid_byte_sequence=$'\x80\x81'
That's a sequence of bytes that, by itself, isn't valid in UTF-8 ...
- 534k
21
votes
Accepted
How can I identify a strange character?
Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:
$ printf "\353\n" | ...
- 423k
20
votes
What fonts are good for unicode glyphs
I came across this in 2016.
A single TTF/OTF font is never going to cover all utf-8 characters. There is a hard limit of 65535 glyphs in a font, and over 1 million utf-8 glphys. You will need to use ...
- 301
17
votes
Accepted
Specify encoding with libreoffice --convert-to csv
Apparently LibreOffice tries to use ISO-8859-1 by default, which is causing the problem.
In response to this bug report, a new parameter --infilter has been added. The following command produces U+...
- 591
16
votes
Find the best font for rendering a codepoint
This is not necessarily the best method, and it sure isn't user-friendly, but it's easy to get working: here's a Python script to do it.
Install the Python-fontconfig library. Either get it from your ...
16
votes
How can I correctly decompress a ZIP archive of files with Hebrew names?
I had success with the command 7z x <source.zip>.
Version:
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,[...])
Potentially relevant environment:
LANG=en_US.UTF-8
LC_ALL=...
- 161
16
votes
How can I convert Persian numerals in UTF-8 to European numerals in ASCII?
For Python there is the unidecode library which handles such conversions in general: https://pypi.python.org/pypi/Unidecode.
In Python 2:
>>> from unidecode import unidecode
>>> ...
- 5,913
16
votes
Accepted
Print check/cross mark in shell script
As revealed by the OP in the comments, they were calling script with sh file.sh. Depending on the default shell to which /bin/sh is symlinked it might not support unicode characters.
For instance, ...
- 16.3k
16
votes
Box character doesn't display properly in Linux terminal
The problem is that you're using literal characters ┌, ─, ┐, └, ─, ┘ instead of using the ncurses-independent symbols. You should use the ACS_??CORNER (for Upper/Lower and Left/Right) and ACS_HLINE ...
- 111k
16
votes
How should I interpret the fact that a Unicode code point is shown in two completely different ways in two different terminal emulators?
U+E000 to U+F8FF is a private-use area. It's reserved to allow systems to store and display characters that are not present in Unicode. So you can't expect it to have a consistent appearance (or ...
15
votes
Accepted
How can I decode a file name using command line?
The standard (POSIX/Unix) command to get the byte values as hex numbers is od.
file=foo.mp3
printf %s "$file" | od -An -vtx1
Which gives an output similar to:
66 6f 6f 2e 6d 70 33
$file above ...
- 534k
Only top scored, non community-wiki answers of a minimum length are eligible