Questions tagged [unicode]

Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems.

1
vote
0answers
19 views

Using iconv to convert utf-8 characters to the closest ASCII character

I can use the iconv command to "translit" a utf-8 string to an ASCII-only string with characters being replaced with their closest ASCII character. However, my problem is that I need the resulting ...
2
votes
1answer
30 views

Why does awk's printf interpret character values greater than 127 as multi-byte characters?

The ASCII character range is from 0 to 127, and within that range, awk's printf with the %c format specifier outputs one byte of data: $ awk 'BEGIN{printf "%c", 97}' a $ awk 'BEGIN{printf "%c", 127}'...
0
votes
1answer
25 views

uxterm not displaying arabic

I need to be able to deal with Arabic file names and directories from terminal and I'd like to achieve that with xterm but instead of displaying Arabic characters it shows dotted rectangles (whether ...
1
vote
1answer
38 views

How to grep characters with their unicode value?

I have the Unicode character ᚠ, represented by its Unicode code point 16A0, in a text file (the text file is encoded(?) as utf-8). When I do grep '\u16A0' test.txt I get no result. How do I grep ...
2
votes
1answer
36 views

`cat -v` for non-printing non-ascii UTF characters

cat has a -v option which converts non-printing characters to their caret notation (which is useful if we don't want the terminal to interpret the control characters literally in cat output). But ...
1
vote
0answers
26 views

Output non-ascii characters through wall command

Using Linux Slackware 14.2, KDE. I created a script in Linux with Russian text that I want to output through wall: #!/bin/bash wall <<< 'Сообщение' Instead of text I see the characters ...
3
votes
0answers
27 views

How can I debug or trace sorting for “LC_COLLATE=en_US.UTF-8 ls -l”?

(Unexpected sort order in en_US.UTF-8 locale explains the basics of UTF sorting) Consider this test scenarion: # touch abc{.{g,t,t.t},@.s,-{s-w,t.c}} # LC_COLLATE="en_US.UTF-8" ls -l test total 0 -...
0
votes
0answers
25 views

Lacking unicode fonts : a Firefox on my french Debian 9.8 displays squares instead of khmer characters. Is there an easy way to install many fonts?

When I open the Ratanakiri province wiki page (Cambodia) with Firefox on my french installation of Debian 9.8, it replaces all khmer characters by typical squares. If it happens, some others languages ...
0
votes
1answer
43 views

How do I increase the amount of supported Unicode characters on my Debian desktop?

Here is a list of characters I see squares with instead of the proper characters. The list is extracted from here: CJK Unified Ideographs Extension B 𠀀 𠀁 𠀂 𠀃 𠀄 𠀅 𠀆 𠀇 𠀈 𠀉 𠀊 𠀋 𠀌 𠀍 𠀎 𠀏 �...
1
vote
0answers
25 views

Which tools can find unicode (UTF-8) text in files containing any form of normalization?

I like to find international text, like "müller", or even chinese text, in files, no matter which unicode representation and case is used. I.e, it should not matter whether the "ü" is stored in NFC ...
3
votes
1answer
80 views

Neomutt - Can't display Swedish characters

I have an issue regarding neomutt not displaying the Swedish characters å, ä and ö. When typing locale in the terminal I get: LANG=sv_SE.UTF-8 LC_CTYPE=sv_SE.UTF-8 LC_NUMERIC=sv_SE.UTF-8 LC_TIME=...
0
votes
0answers
12 views

Is it possible to make bash support UTF-8 without any locale files installed?

I have installed the bash binary into an Android device (with sh as the filename), using an app that I don't fully trust. It interprets non-ASCII Unicode characters as random control codes, or has no ...
2
votes
1answer
74 views

Grep search for words containing capital letters in Unicode

After some searching, I found a grep command that will find words containing capital letters, and list them without duplicates: grep -hoP "\w*[[:upper:]]+\w*" * | sort -u However, when it comes to ...
0
votes
1answer
84 views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
1
vote
1answer
111 views

Recursively converting Windows files to Unix files

I have a PHP application with is located on Linux with multiple directories (and sub-directories) and many PHP, JS, HTML, CSS, etc files. Many of the files have Windows EOL control characters and I ...
0
votes
1answer
44 views

Patch a glyph into a font from another font

I have a monaco font patched for powerline that I downloaded from this repository. Now for licensing issue there are no free version of this font. Anyway, I use tmux which have U+2502 as a separator ...
0
votes
0answers
17 views

New entry in xmodmap creates new group instead of layer

Here is my dotfile: ! Key ! Shift+Key ! Mode_switch+Key ! Mode_switch+Shift+Key ! ISO_Level3_Shift+Key ! ISO_Level3_Shift+Shift+Key keycode 43 = h H BackSpace NoSymbol U20BD NoSymbol Cyrillic_er ...
0
votes
0answers
10 views

Some symbols are not rendered in Linux/Plasma5

I have Opensuse Tumbleweed with all the latest updates installed on my laptop and my PC, on my laptop with same font CURLY BRACKET SECTION symbols are rendered, but on my PC they are not, I see only ...
0
votes
1answer
35 views

UTF8String example for seagull

I am trying to test a diameter application using seagull. In the captured PCAP file I have 3GPP-SGSN-MCC-MNC, which is defined as UTF8String. In seagull, I defined UTF8String in base_cc.xml as: &...
1
vote
1answer
47 views

(Possible) inconsistent behavior of grep and less

I have a utf-8 file containing some Turkish text inside. (My system is MacOSX) $ file -I foo.merge $foo.merge: text/plain; charset=utf-8 When I try to see some Turkish specific characters by using ...
0
votes
1answer
96 views

Hebrew (Unicode) file-name support on CentOs 7

I have a VPS with CentOS 7 installed. I am trying to upload to the server a file that has a Hebrew name. I tried to upload it via FTP (Pure-ftpd) and I get: Failed to convert command to 8 bit charset. ...
0
votes
0answers
36 views

Fonts needed for browsers

I'm having this issue where many unicode symbols aren't displayed(just boxes instead) on websites. But aren't they supposed to be loaded(by the browser) from external sources? If not, what fonts do I ...
2
votes
1answer
77 views

Unicode in Gnuplot terminal

E.g., on the bash command line I can type a δ character using the Compose key and an ad hoc ~/.XCompose file, but if I try to do the same in Gnuplot what I get is 13:48 boffi@debian:~ $ δ bash: δ: ...
1
vote
1answer
44 views

Is there a fast tool to show all the unique unicode characters in a file and their count?

Does anybody know a tool to tabulate all the unique unicode characters and their counts in a file?
7
votes
3answers
2k views

Strange character in a file

I have an UTF-8 file that contains a strange character -- visible to me just as <96> This is how it appears on vi and how it appears on gedit and how it appears under LibreOffice and that ...
2
votes
1answer
51 views

UTF8 Character Makes File Inaccessible

If I run: scp me@example.com:/home/me/cömmön_file.jpg /home/me/ from my remote server I get: scp: /home/me/cömmön_file.jpg: No such file or directory If I swap out the utf8 characters though ...
2
votes
1answer
73 views

How to write a file list with foreign names to a csv file in the terminal?

I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below. When I do ls -t | tail -n +2, I see the real ...
0
votes
1answer
26 views

Remove non-alphas while ignoring certain unicode chars

I have a text like the following: []përgjithshme [" është "] 28.748 [< kilometra katrorë.>] This text contains cyrillic characters in the range U400 - U4FF. I want to strip out the non-...
1
vote
0answers
124 views

Unicode and Telegram Desktop app

Is there any way to enable unicode input in Telegram Desktop App for unix/linux systems? Addendm: The problem seems to be due to some qt-ibus configuration problem according to this unfortunately ...
4
votes
1answer
126 views

Is linux kernel implemented tty always render data using UTF-8?

It seems so. I read IUTF8, but the document only said it allows you to handle input as UTF-8 when doing line-editing. But what about output? If you use a GUI Terminal, it usually allows you to ...
7
votes
1answer
195 views

How to correctly display Polish diacritic symbols in groff?

I'm playing around with groff and I wanted to generate a pdf from the following test.ms .TL Tytuł .AU Imię Nazwisko .NH Wstęp .PP Pierwszy paragraf. Jakieś informacje, żeby były polskie znaki. .PP ...
1
vote
1answer
124 views

DWM Unicode Support

I'm in the process of creating a nice looking statusbar with DWM and I want to implement unicode symbols with font awesome. Supposedly I can use the Pango patch but its out of date (for version 6.1) ...
3
votes
1answer
353 views

SSH to FreeBSD in UTF-8

I'm trying to ssh to FreeBSD-11.2-Release from my Windows machine using Mintty, but when I set LC_ALL to en_US.UTF-8 and try to enter/paste a non-ASCII character, connection got closed. I'm using ...
36
votes
1answer
4k views

Should we use UTF-8 characters like ⏰ in bash/shell script?

The simple code here is working as expected on my machine if launched with bash : function ⏰(){ date } ⏰ Could there be a problem for other people using this, or is it universal ? I'm wondering ...
3
votes
1answer
94 views

Why is bash removing other digits?

On this (It is not intended to be a range, but an explicit list): $ a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९' $ echo "${a//[0123456789]}" ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९ Bash is ...
3
votes
1answer
147 views

Bash prompt wrap broken when path has unicode (Greek) characters

When a path is too long to fit on the terminal and it wraps, the following prompt is broken and you can't see properly what is typed. See screenshot below: These are the contents of .bashrc that set ...
25
votes
3answers
4k views

Why does sort say that ɛ = e?

ɛ ("Latin epsilon") is a letter used in certain African languages, usually to represent the vowel sound in English "bed". In Unicode it's encoded as U+025B, very distinct from everyday e. However, if ...
0
votes
1answer
233 views

Terminal shows non-ascii? characters in UTF-16 hex codes

todoroki@todoroki-VJZ13B ~>printf "ä\n" echo "ä" ä ä ä \udcc3\udca4: \u30b3\u30de\u30f3\u30c9\u304c\u898b\u3064\u304b\u308a\u307e\u305b\u3093 ...
3
votes
3answers
301 views

Using uniq on unicode text

I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical. $ cat file.txt ܐܒܘܢ ܢܗܘܐ ܐܒܘܢ When I use sort and uniq, the result ...
0
votes
0answers
59 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
2
votes
2answers
65 views

Finding Greek from the command-line

How can I use perl on the command-line to print only lines containing Greek unicode characters? For example, only the second of the following lines hullo, world χαῖρε, ὦ κόσμε It should be ...
2
votes
1answer
261 views

Bash behaves weird when PS1 contains wide unicode characters

I'm trying to customize the bash prompt and here is my PROMPT_COMMAND prompt_command() { local time="${BLUE}\t${RESET}" local user="${GREEN}\u${RESET}" local hostname="${GREEN}\H${RESET}" ...
0
votes
0answers
54 views

How to check if `libc`, `font` and `terminal` all agree on Unicode version?

In my case, when I open certain files in tmux such as this file, and have multiple panes, the characters overflow the pane's boundaries. Nicholas Marriott suggested that the libc, font and terminal ...
1
vote
0answers
253 views

Chromium in raspbian is not loading fonts

I have flash project which has verdana/lohit font embedded into it but they do not load and shows boxes in Chromium (Chromium 65.0.3325.181) in raspbian. However same project gets loaded when ran in ...
0
votes
1answer
172 views

Filename displays with ? char when `ls` while with junk character when `ls | grep fran`?

What is the configuration that defines how file name with unrecognizable character be replaced with some other character. For example in my Linux system same file is displayed differently in UTF8 ...
1
vote
0answers
132 views

Onboard 1.4 crashes X.org on Arch Linux

Recently, a problem arose, which I suspect of being a bug, but I am not sure. I use Onboard virtual keyboard on my tablet PC powered by Arch Linux. It works nicely, but one thing is not working well. ...
0
votes
1answer
318 views

Wrong font for some unicode characters [closed]

I have been experiencing some issues with how my terminal shows some of the Unicode characters form my language. The picture shows the issue in the editor in the left terminal. It seems like some of ...
1
vote
0answers
166 views

refuse to cp/rsync/etc some files with strange characters

We have a Windows server with filenames containing unusual characters. For exemple picture.jpg (the unusual character is between the e and the .jpg). We mount it with mount -t cifs -o 'ro,...
1
vote
1answer
103 views

Can I write a console program that works with multiple character encodings? [duplicate]

I am writing a console program in C. I expect the Terminal that my program is running in to have its character encoding set to UTF-8. This means that I am sending UTF-8 encoded strings to the ...
0
votes
1answer
240 views

Bash (Yad) using floating point numbers instead of whole numbers [closed]

I wrote this bash script that runs multiple timers. The problem is my array stores whole numbers: |1|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|1 But the other user's LC_NUMERIC is setup as it_IT.UTF-8 and their ...