Questions tagged [unicode]

Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems.

Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

Unable to encode the unicode character during writing to a file in python

Here i am extracting the bible verses from locally stored html files using python actually this python is not written by me i got from somewhere, when i run the python code to extract the bible verses ...
1
vote
1answer
25 views

Extra Junk at the end of [99synaptic] file

When I try to launch synaptic I get this error: E: Syntax error /etc/apt/apt.conf.d/99synaptic:1: Extra junk at end of file" I found out that semicolons could be missed in that file, but when try to ...
2
votes
0answers
43 views

Can Unicode characters be programmatically defined in python 3.7? [migrated]

I'm trying to write a python 3.7 function that prints the Unicode character for a playing card (using wikipedia as a reference). Each of the Unicode character is a string that can be represented as '\...
15
votes
2answers
1k views

Why won't some unicode characters print to my terminal?

I'm running Arch Linux with simple terminal using the Adobe Source Code Pro font. My locale is correctly set to LANG=en_US.UTF-8. I want to print Unicode characters representing playing cards to my ...
1
vote
2answers
40 views

Cannot display 8-bit characters in xterm in ubuntu 18.04

I have tried every method I could find in the forums, and cannot display any 8-bit character in an xterm window, under kubuntu 18.04 (nor in earlier versions). All characters in range 0x20-0x7e ...
0
votes
1answer
30 views

Bash: can I decode unicode char codes? [duplicate]

I've a bash script that scans php files to find rows like the followin @include "\057var/\167ww/n\143v/ad\155inis\164rato\162/com\160onen\164s/co\155_spp\141gebu\151lder\057.9e7\064214f\056ico"; I ...
0
votes
0answers
21 views

Unable to get webpage contents with unicode character using python 3

I am trying to read a webpage of particular tag using python3 but due to unable to handle unicode character its getting error as UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in ...
1
vote
0answers
45 views

Using iconv to convert utf-8 characters to the closest ASCII character

I can use the iconv command to "translit" a utf-8 string to an ASCII-only string with characters being replaced with their closest ASCII character. However, my problem is that I need the resulting ...
2
votes
1answer
34 views

Why does awk's printf interpret character values greater than 127 as multi-byte characters?

The ASCII character range is from 0 to 127, and within that range, awk's printf with the %c format specifier outputs one byte of data: $ awk 'BEGIN{printf "%c", 97}' a $ awk 'BEGIN{printf "%c", 127}'...
1
vote
1answer
31 views

uxterm not displaying arabic

I need to be able to deal with Arabic file names and directories from terminal and I'd like to achieve that with xterm but instead of displaying Arabic characters it shows dotted rectangles (whether ...
1
vote
1answer
51 views

How to grep characters with their unicode value?

I have the Unicode character ᚠ, represented by its Unicode code point 16A0, in a text file (the text file is encoded(?) as utf-8). When I do grep '\u16A0' test.txt I get no result. How do I grep ...
2
votes
1answer
48 views

`cat -v` for non-printing non-ascii UTF characters

cat has a -v option which converts non-printing characters to their caret notation (which is useful if we don't want the terminal to interpret the control characters literally in cat output). But ...
1
vote
0answers
31 views

Output non-ascii characters through wall command

Using Linux Slackware 14.2, KDE. I created a script in Linux with Russian text that I want to output through wall: #!/bin/bash wall <<< 'Сообщение' Instead of text I see the characters ...
3
votes
0answers
31 views

How can I debug or trace sorting for “LC_COLLATE=en_US.UTF-8 ls -l”?

(Unexpected sort order in en_US.UTF-8 locale explains the basics of UTF sorting) Consider this test scenarion: # touch abc{.{g,t,t.t},@.s,-{s-w,t.c}} # LC_COLLATE="en_US.UTF-8" ls -l test total 0 -...
0
votes
0answers
35 views

Lacking unicode fonts : a Firefox on my french Debian 9.8 displays squares instead of khmer characters. Is there an easy way to install many fonts?

When I open the Ratanakiri province wiki page (Cambodia) with Firefox on my french installation of Debian 9.8, it replaces all khmer characters by typical squares. If it happens, some others languages ...
0
votes
1answer
47 views

How do I increase the amount of supported Unicode characters on my Debian desktop?

Here is a list of characters I see squares with instead of the proper characters. The list is extracted from here: CJK Unified Ideographs Extension B 𠀀 𠀁 𠀂 𠀃 𠀄 𠀅 𠀆 𠀇 𠀈 𠀉 𠀊 𠀋 𠀌 𠀍 𠀎 𠀏 �...
1
vote
0answers
34 views

Which tools can find unicode (UTF-8) text in files containing any form of normalization?

I like to find international text, like "müller", or even chinese text, in files, no matter which unicode representation and case is used. I.e, it should not matter whether the "ü" is stored in NFC ...
3
votes
1answer
90 views

Neomutt - Can't display Swedish characters

I have an issue regarding neomutt not displaying the Swedish characters å, ä and ö. When typing locale in the terminal I get: LANG=sv_SE.UTF-8 LC_CTYPE=sv_SE.UTF-8 LC_NUMERIC=sv_SE.UTF-8 LC_TIME=...
0
votes
0answers
15 views

Is it possible to make bash support UTF-8 without any locale files installed?

I have installed the bash binary into an Android device (with sh as the filename), using an app that I don't fully trust. It interprets non-ASCII Unicode characters as random control codes, or has no ...
2
votes
1answer
80 views

Grep search for words containing capital letters in Unicode

After some searching, I found a grep command that will find words containing capital letters, and list them without duplicates: grep -hoP "\w*[[:upper:]]+\w*" * | sort -u However, when it comes to ...
0
votes
1answer
110 views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
1
vote
1answer
149 views

Recursively converting Windows files to Unix files

I have a PHP application with is located on Linux with multiple directories (and sub-directories) and many PHP, JS, HTML, CSS, etc files. Many of the files have Windows EOL control characters and I ...
0
votes
1answer
49 views

Patch a glyph into a font from another font

I have a monaco font patched for powerline that I downloaded from this repository. Now for licensing issue there are no free version of this font. Anyway, I use tmux which have U+2502 as a separator ...
0
votes
0answers
17 views

New entry in xmodmap creates new group instead of layer

Here is my dotfile: ! Key ! Shift+Key ! Mode_switch+Key ! Mode_switch+Shift+Key ! ISO_Level3_Shift+Key ! ISO_Level3_Shift+Shift+Key keycode 43 = h H BackSpace NoSymbol U20BD NoSymbol Cyrillic_er ...
0
votes
0answers
11 views

Some symbols are not rendered in Linux/Plasma5

I have Opensuse Tumbleweed with all the latest updates installed on my laptop and my PC, on my laptop with same font CURLY BRACKET SECTION symbols are rendered, but on my PC they are not, I see only ...
0
votes
1answer
40 views

UTF8String example for seagull

I am trying to test a diameter application using seagull. In the captured PCAP file I have 3GPP-SGSN-MCC-MNC, which is defined as UTF8String. In seagull, I defined UTF8String in base_cc.xml as: &...
1
vote
1answer
48 views

(Possible) inconsistent behavior of grep and less

I have a utf-8 file containing some Turkish text inside. (My system is MacOSX) $ file -I foo.merge $foo.merge: text/plain; charset=utf-8 When I try to see some Turkish specific characters by using ...
0
votes
1answer
125 views

Hebrew (Unicode) file-name support on CentOs 7

I have a VPS with CentOS 7 installed. I am trying to upload to the server a file that has a Hebrew name. I tried to upload it via FTP (Pure-ftpd) and I get: Failed to convert command to 8 bit charset. ...
0
votes
0answers
39 views

Fonts needed for browsers

I'm having this issue where many unicode symbols aren't displayed(just boxes instead) on websites. But aren't they supposed to be loaded(by the browser) from external sources? If not, what fonts do I ...
2
votes
1answer
90 views

Unicode in Gnuplot terminal

E.g., on the bash command line I can type a δ character using the Compose key and an ad hoc ~/.XCompose file, but if I try to do the same in Gnuplot what I get is 13:48 boffi@debian:~ $ δ bash: δ: ...
1
vote
1answer
46 views

Is there a fast tool to show all the unique unicode characters in a file and their count?

Does anybody know a tool to tabulate all the unique unicode characters and their counts in a file?
7
votes
3answers
2k views

Strange character in a file

I have an UTF-8 file that contains a strange character -- visible to me just as <96> This is how it appears on vi and how it appears on gedit and how it appears under LibreOffice and that ...
2
votes
1answer
56 views

UTF8 Character Makes File Inaccessible

If I run: scp me@example.com:/home/me/cömmön_file.jpg /home/me/ from my remote server I get: scp: /home/me/cömmön_file.jpg: No such file or directory If I swap out the utf8 characters though ...
2
votes
1answer
74 views

How to write a file list with foreign names to a csv file in the terminal?

I have some files with names in foreign languages in my terminal folder. On the terminal when I do 'ls' i see the list on the right in the picture below. When I do ls -t | tail -n +2, I see the real ...
0
votes
1answer
26 views

Remove non-alphas while ignoring certain unicode chars

I have a text like the following: []përgjithshme [" është "] 28.748 [< kilometra katrorë.>] This text contains cyrillic characters in the range U400 - U4FF. I want to strip out the non-...
1
vote
0answers
133 views

Unicode and Telegram Desktop app

Is there any way to enable unicode input in Telegram Desktop App for unix/linux systems? Addendm: The problem seems to be due to some qt-ibus configuration problem according to this unfortunately ...
4
votes
1answer
130 views

Is linux kernel implemented tty always render data using UTF-8?

It seems so. I read IUTF8, but the document only said it allows you to handle input as UTF-8 when doing line-editing. But what about output? If you use a GUI Terminal, it usually allows you to ...
7
votes
1answer
225 views

How to correctly display Polish diacritic symbols in groff?

I'm playing around with groff and I wanted to generate a pdf from the following test.ms .TL Tytuł .AU Imię Nazwisko .NH Wstęp .PP Pierwszy paragraf. Jakieś informacje, żeby były polskie znaki. .PP ...
1
vote
1answer
138 views

DWM Unicode Support

I'm in the process of creating a nice looking statusbar with DWM and I want to implement unicode symbols with font awesome. Supposedly I can use the Pango patch but its out of date (for version 6.1) ...
3
votes
1answer
402 views

SSH to FreeBSD in UTF-8

I'm trying to ssh to FreeBSD-11.2-Release from my Windows machine using Mintty, but when I set LC_ALL to en_US.UTF-8 and try to enter/paste a non-ASCII character, connection got closed. I'm using ...
36
votes
1answer
4k views

Should we use UTF-8 characters like ⏰ in bash/shell script?

The simple code here is working as expected on my machine if launched with bash : function ⏰(){ date } ⏰ Could there be a problem for other people using this, or is it universal ? I'm wondering ...
3
votes
1answer
96 views

Why is bash removing other digits?

On this (It is not intended to be a range, but an explicit list): $ a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९' $ echo "${a//[0123456789]}" ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९ Bash is ...
3
votes
1answer
154 views

Bash prompt wrap broken when path has unicode (Greek) characters

When a path is too long to fit on the terminal and it wraps, the following prompt is broken and you can't see properly what is typed. See screenshot below: These are the contents of .bashrc that set ...
25
votes
3answers
4k views

Why does sort say that ɛ = e?

ɛ ("Latin epsilon") is a letter used in certain African languages, usually to represent the vowel sound in English "bed". In Unicode it's encoded as U+025B, very distinct from everyday e. However, if ...
0
votes
1answer
268 views

Terminal shows non-ascii? characters in UTF-16 hex codes

todoroki@todoroki-VJZ13B ~>printf "ä\n" echo "ä" ä ä ä \udcc3\udca4: \u30b3\u30de\u30f3\u30c9\u304c\u898b\u3064\u304b\u308a\u307e\u305b\u3093 ...
3
votes
3answers
337 views

Using uniq on unicode text

I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical. $ cat file.txt ܐܒܘܢ ܢܗܘܐ ܐܒܘܢ When I use sort and uniq, the result ...
0
votes
0answers
66 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
2
votes
2answers
68 views

Finding Greek from the command-line

How can I use perl on the command-line to print only lines containing Greek unicode characters? For example, only the second of the following lines hullo, world χαῖρε, ὦ κόσμε It should be ...
2
votes
1answer
288 views

Bash behaves weird when PS1 contains wide unicode characters

I'm trying to customize the bash prompt and here is my PROMPT_COMMAND prompt_command() { local time="${BLUE}\t${RESET}" local user="${GREEN}\u${RESET}" local hostname="${GREEN}\H${RESET}" ...
0
votes
0answers
57 views

How to check if `libc`, `font` and `terminal` all agree on Unicode version?

In my case, when I open certain files in tmux such as this file, and have multiple panes, the characters overflow the pane's boundaries. Nicholas Marriott suggested that the libc, font and terminal ...