2022 Developer Survey is open! Take survey.

Questions tagged [unicode]

Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems.

Filter by
Sorted by
Tagged with
2 votes
2 answers
100 views

replacement for tr with utf-8 capabilities

in order to isolate the last word in any line of a poem (to have a list of all rhymes), I put together several snippets of code obtaining this awk '{print $NF}' input.txt | tr 'A-Z' 'a-z' | tr -sc 'a-...
user avatar
  • 297
0 votes
0 answers
13 views

Using UIM to insert Unicode input from keyboard

According to "Transparent" Unicode input from keyboard UIM supports entering Unicode characters with Ctrl+Shift+u like IBus, but without showing an interactive prompt. I haven't been able ...
user avatar
  • 1
2 votes
1 answer
46 views

VIM shows ^@ every other character and ^M^@ at the end of line

Sorry if it's answered somewhere, I have no idea how to look for it. I received a series of reports from a bank that I'm supposed to process and they seem to be... badly encoded? First two lines in ...
user avatar
  • 975
0 votes
0 answers
15 views

How write a plugin or script for Kate that takes a unicode code point and inserts the corresponding unicode character at cursor?

Entering arbitrary Unicode characters according to their code point is simple in e.g. Gedit (press ctrl+shift+u, enter the (hex) code point, press enter). One can even use the terminal. For example, ...
user avatar
0 votes
1 answer
48 views

Unable to change file encoding to utf-8

I modified a UTF-8 encoded xml file using vi editor and saved it. I'm on Redhat Linux 7.9 I checked the file encoding after the changes and found it to be us-ascii file --mime-encoding tmpfiles/08/...
user avatar
  • 297
1 vote
2 answers
132 views

Showing 𝐒, 𝐽, and 𝑁 in the terminal

How can I make 𝐒(U+1d412), 𝐽(U+1d43d), and 𝑁(U+1d441), which I can display clearly in my browser, show in the terminal? I use Urxvt on Archlinux. The ~/.Xdefaults file has this content: URxvt*...
user avatar
  • 1,012
0 votes
1 answer
33 views

Disparity in rendering Malayalam glyph

Migrating from https://github.com/rsms/inter/issues/436 This is the correct rendering of the word prominence in Malayalam. പ്രാധാന്യം. And this is how it renders on: Arch Linux Where Screenshot ...
user avatar
0 votes
0 answers
78 views

Suckless's st messes up UTF-8 sometimes

I wrote an echo program in Rust (which can deal with UTF-8) and occasionally st seems to mess up the UTF-8 input. I've tried different terminal emulators, like alacritty and gnome-terminal and they ...
user avatar
0 votes
2 answers
99 views

How to remove all soft hyphens (U+00AD) from a file

There is a thing called Soft hyphen. Its unicode value is U+00AD and it cannot be seen in my text editor. Apparently, a few of my files are riddled with it: $ perl -C7 -ne 'for(split(//)){print ...
user avatar
  • 1,201
0 votes
0 answers
37 views

How do I set the point type to UTF8 characters in Gnuplot

I am attempting to plot data points using UTF8 characters. I am using Gnuplot V5.2 running in Debian Buster. The plot is saved to .eps for later inclusion in a Latex document. If I plot using pt &...
user avatar
  • 1
2 votes
1 answer
85 views

echo checkmark in shellscript (zsh) how-to

Why is this checkmark not printed correctly when executing my shellscript, even though echo alone outputs corrctly? script: #!/bin/sh YELLOW='\033[1;33m' NC='\033[0m' # no color echo "enter your ...
user avatar
2 votes
1 answer
137 views

What is "length" of a string in Bourne shell compatibles' `${#string}`?

Arising from this discussion: When I have (zsh 5.8, bash 5.1.0) var="ASCII" echo "${var} has the length ${#var}, and is $(printf "%s" "$var"| wc -c) bytes long" ...
user avatar
0 votes
0 answers
47 views

UTF-8 Decoding And Fonts

I created an application that users write fancy texts. The only problem is that I don't know how to make command-line render this utf-8 charterer. It works well on every other charterers but it fails ...
user avatar
0 votes
0 answers
24 views

UTF file names listing in xterm

I noticed that file names containing UTF characters have different listing results in my xterm & UXTerm. The first and the worst of all is they are listed/output as 8-bit enabled ASCII characters ...
user avatar
  • 485
0 votes
0 answers
16 views

Uxterm showing arab characters in a german locale system

I have xterm installed with german language but my xterm shows the menu in arab: How can make the menu the same language as my system? I'm using Manjaro Linux. My .Xresources config: !background ...
user avatar
0 votes
0 answers
81 views

How to render unicode like Terminator in Kitty?

I am configuring my Arch Linux installation (currently on a Virtual Machine with Garuda as host system) and I've noticed that on Terminator or Gnome Terminal unicode symbols display pretty cool, for ...
user avatar
0 votes
0 answers
39 views

Editing single characters in a PDF file (generated from QPDF)

Friends, I would love your help navigating a PDF editing situation. I've extracted the PDF streams using QPDF (per the instructions here). I now have an editable PDF. My task is to replace certain ...
user avatar
0 votes
1 answer
45 views

How do you type a number with a bar symbol over it?

100 divided by 3 equals 33.3 bar. How can I put a bar symbol over the three like this: You'd think there would be 10 unicode characters dedicated to this, but I couldn't find them.
user avatar
  • 4,313
1 vote
1 answer
163 views

How can I enable Unicode support in buildroot?

I want to build a 32 bit x86 ISO image from buildroot that can display Unicode characters. How can I do that? In make menuconfig, I've set BR2_ENABLE_LOCALE_WHITELIST="C en_US en_US.UTF-8" ...
user avatar
0 votes
0 answers
71 views

Copying windows csv file data to Unix server file

I have this Unix server at work where resides a script which picks up a file on that same server and process each row. I believe we receive those files from data providers, which SFTP the files to us. ...
user avatar
0 votes
0 answers
33 views

File with Multiple ISO encodings

I have an Arabic file with multiple ISO encodings; encoded in ISO-8859-6 and a few characters encoded in ISO-8859-1 (Because they don't have an equivalent in ISO-8859-6). I want to convert this whole ...
user avatar
  • 167
0 votes
1 answer
264 views

How does Linux display Unicode characters?

I have a simple Python PIL script that's render some list of unicode symbols with FreeSerif font, which is have the most full unicode support (about 65k glyph) but in reality Arch Linux with last ...
user avatar
  • 13
0 votes
0 answers
30 views

pcre2grep cannot work for UTF-16LE

How come pcre2grep cannot work for UTF-16LE data format in file $ cat standard.bjt |pcre2grep model $ Please guide it to the working way/solution
user avatar
0 votes
1 answer
83 views

No batch processing when: busybox find . -exec

I have access to only busybox. I have a folder, like this: prueba/$: ls ./uno ./dos ./tres Each two subdirectories (uno, dos) have many files (with spaces in their names and non-ASCII characters) and ...
user avatar
  • 143
0 votes
0 answers
25 views

Problem with utf-8 encode

locale-command return the following: LANG=ru_RU.UTF-8 LC_CTYPE="ru_RU.UTF-8" LC_NUMERIC="ru_RU.UTF-8" LC_TIME="ru_RU.UTF-8" LC_COLLATE="ru_RU.UTF-8" LC_MONETARY=...
user avatar
  • 123
0 votes
2 answers
101 views

Can a file path be invalid UTF-8?

Title says it all: if I have the bytes of a file path and I try to decode them as UTF-8 is it possible that the operation will fail (i.e. the bytes of a valid unix file path can be invalid UTF-8)?
user avatar
0 votes
1 answer
224 views

How do I make .bdf Format Bitmap Font work

I'm using an Arch-based Linux distro. Recently I started a New Ricing for my BSPWM. This time I want to use a bitmap font. Well I would like a font with glyphs, so I chose Siji. Previously I used some ...
user avatar
  • 1
0 votes
1 answer
22 views

Accessing the UTF-8 Cuneiform Numbers and Punctuation Symbols

Does anybody know how to access the UTF-8 Cuneiform Numbers and Punctuation Symbols? twash="\u12400" # two ash thash="\u12401" # three ash frash="\u12402&...
user avatar
  • 39
0 votes
1 answer
49 views

Using variables defined in other bash scripts [duplicate]

I have a bash script where I define UTF-8 Greek Symbols in a file named greek-utfb.sh. I want to run tests that display the variables for printing the greek letters in the file greek-utfb-scout. --- ...
user avatar
  • 39
0 votes
2 answers
1k views

How to print UTF-8 symbols

How can I print UTF-8 symbols on a terminal using bash commands. This works echo -e '\U2586' But the following is failing printf '%s\n' $(tput setaf 118) "\\u2586" $(tput sgr0)
user avatar
  • 39
0 votes
1 answer
123 views

UTF-8 characters are not printed correctly on the console

I've seen some questions similar to this one but I'm not being able to use the solutions of those to help me. So, what is my problem?, I have a file with the characters ç, ã, à, etc... but they don't ...
user avatar
  • 21
0 votes
0 answers
74 views

Command substitution with local characters

It's kinda basic but surprised me a lot. Im using on WSL (Unubtu 18.04 LTS) poppler-utils for managing PDFs, usually it's as simple as cd <DIR of bunch of PDFs> pdfunite `ls` out.pdf as ...
user avatar
  • 101
2 votes
1 answer
77 views

How to set `export LC_PAPER=a4` to not break apps?

I had to remove export LC_PAPER=a4 from my ~/.bashrc as it was breaking characters display in screen and some other apps : symbols display wrong in `screen` - ascii or utf . Therefore, question, how ...
user avatar
0 votes
2 answers
224 views

Listing PCI Devices By Reading From File (Instead of lspci Command)

I need to read PCI device information from files. But it gives unusable output when I use command like that: cat /proc/bus/pci/05/00.0 Output: �h�� How could I fix this? OS: Debian-like Linux x64, ...
user avatar
  • 15
9 votes
3 answers
693 views

Are ∈ and ℝ symbols available in eqn/roff?

A set of commonly used symbols to represent that a variable belongs to a given real coordinate space are ∈ ("ELEMENT OF", Unicode U+2208) and ℝ ("DOUBLE-STRUCK CAPITAL R", Unicode ...
user avatar
  • 569
1 vote
1 answer
293 views

How to replace Unicode code numbers by UTF-8 characters in a number of files?

I have 500 files that contain escaped Unicode characters like so: Albert Vel\u00e1zquez V\u00e9ronique Ekin etc. The data was queried from a server by a script through XMLHttpRequests and I assume ...
user avatar
  • 7,487
25 votes
8 answers
3k views

What's the best way to actually "type" special UTF-8 chars?

Everything on my system (that needs it) supports UTF-8 just fine. That's all nice when you want output... But what if you want easy input ? At the moment the only non-ASCII chars I can easily type are ...
user avatar
  • 1,844
0 votes
1 answer
816 views

How to map compose key to insert via setxkbmap?

I'm using US (non-international) layout. One way of typing üäöß is setting a compose key via: setxkbmap -layout us -option compose:ralt Unfortunately, this removes the normal "alt" ...
user avatar
1 vote
1 answer
586 views

How to force unzip listing to use unicode UTF-8

ZIP zip -r --symlinks ../a.zip a a.txt adding: a/šáčžřŠÁČÝŘŽÚ§/ (stored 0%) Filename is stored properly in unicode. LIST unzip -l ../a.zip Length Date Time Name --------- ...
user avatar
1 vote
2 answers
130 views

Making ls do a reasonable sort order, but not print Greek characters as "??"

Suppose I have the following files in a directory _y a b c x z β The following command lists them in the above order, as God intended: perl -e '@files = <*>; print join(" ",sort @...
user avatar
0 votes
1 answer
20 views

Samba and utf8: my avi became strange title

I want to share my movie folder from one server. So I use utf8 This is smb.conf [global] workgroup = BLU server string = nas netbios name = NAS server role = standalone server hosts allow = 192.168.0. ...
user avatar
  • 10.9k
-1 votes
1 answer
158 views

"Transparent" Unicode input from keyboard

How can I enter Unicode characters from my keyboard? I'm aware of the Ctrl+Shift+u, hex, Return method of entering an arbitrary Unicode character provided by (I think) IBus, but this causes a prompt (...
user avatar
  • 117
0 votes
1 answer
137 views

Map from XKB key names to LDML ISO key names?

XKB names keyboard keys with names such as "TLDE" and "AB01" https://www.charvolant.org/doug/xkb/html/node5.html#Fig:KeyNames seen in /usr/share/X11/xkb/keycodes/evdev , whereas ...
user avatar
1 vote
1 answer
164 views

XTerm only display part of some unicodes and Urxvt displays nothing

In my case, (U)XTerm does display some unicodes, but only partially. For exmaple: for letters are: 𝒫φϕ𝒟. Here is a screenshot of UXTerm and Urxvt respectively It seems that the problem is related ...
user avatar
32 votes
4 answers
2k views

gitk crashes when viewing commit containing emoji: X Error of failed request: BadLength (poly request too large or internal Xlib length error)

I'm able to open gitk but it crashes as soon as I open a commit whom changes contains an emoji (not the commit message). Error ❯ gitk --all X Error of failed request: BadLength (poly request too ...
user avatar
3 votes
1 answer
394 views

Formatting from ISO-8859-1 to Windows-1251

I want to fix the encoding in a string "Ïåðåïðîøèòü Ñûðîåæêèíà" and get the correct UTF-8 string "Перепрошить Сыроежкина". This website told me that I need to convert from ISO-8859-...
user avatar
2 votes
2 answers
251 views

Is there a way to extend iconv to support more encodings?

The iconv command is used to convert from one character encoding to another. You can see the list of all the encodings it supports with iconv -l. Is there a way to extend iconv by adding more ...
user avatar
  • 1,205
0 votes
1 answer
113 views

Converting UTF8 characters to slash escaped numeric values

My system language is not Japanese, but my user language/encoding is. Using shell things in terminal emulators 'directly' works okay, but it seems that 'behind the scene' stuff does not. 保存 = Save for ...
user avatar
7 votes
1 answer
545 views

Why do some Linux utilities output Unicode (when it is not expected)?

While building a tool that reads command output I realized that some Linux utilities output Unicode characters when I didn't really expect them to do so. For instance, find uses ‘ (as opposed to the ...
user avatar
0 votes
1 answer
68 views

Can't understand how xargs is treating unicode literals

I'm trying to convert unicode literals into readable format. I'm using the -t flag to see what command xargs is going to run before it runs it. I see the following weirdness: $ echo -n "\\\t\\\...
user avatar

1
2 3 4 5
9