Skip to main content

Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
2 votes
1 answer
98 views

removing hidden control characters in filenames

I have a huge number of files spread across a large directory structure that have hidden control characters in their names. ls lists them as, e.g.: '614.7-4-F1-00-090-007-RozvadØ'$'\302\237'' RP1-...
atapaka's user avatar
  • 591
1 vote
1 answer
38 views

How can I set the character to Latn-1 or MCS when using serial-getty?

I'd like to use my old VT420 terminal as system console. Adding RS232 ports and setting up serial-getty are not a problem, but: For years, almost all Linux distros have been using UTF-8 as the ...
Neppomuk's user avatar
  • 346
11 votes
3 answers
2k views

UTF-8 characters in POSIX shell script *comments* - anything against it?

I would like to include a couple of non-ASCII characters in my POSIX shell script comments. Note this is in no way a duplicate of e.g. "Which character encodings are supported by posix?" as ...
Vlastimil Burián's user avatar
0 votes
0 answers
16 views

Displaying European accents without Xorg [duplicate]

I'm setting up Arch Linux on a new computer, and I intend to use bspwm as a window manager. I know bspwm runs on Xorg, and there is plenty of information online on how to display region-specific ...
Giorgio's user avatar
  • 101
0 votes
1 answer
79 views

regex: how come the trademark symbol matches to a-z?

Sorry if this is a repeat or basic question but it is hard to search for a ™. I'm writing a script to remove weird characters from file names. How come the trade mark symbol ™ matches [^a-z] ??? $ ...
codywohlers's user avatar
0 votes
1 answer
46 views

nmtui is not rendering correctly

When using nmtui, ┌ and │ are added in places where they definitely should not be (see attached screenshot): How can I solve this?
user20695956's user avatar
3 votes
1 answer
649 views

How can I convert full-width characters to half-width characters (and vice versa)?

Here is my simple problem, how can I convert half-width to full-width from the command line. I thought this would be built-in my iconv command line, but I did not find anything here: $ iconv -l | ...
malat's user avatar
  • 3,112
0 votes
1 answer
453 views

If I have a json string how do I calculate the number of bytes needed when stored?

I have a json string formatted displayed in a web page. What I am trying to understand is what is the size in terms of bytes that this json string requires. If I copy and pipe to wc -c I get 1000 ...
Jim's user avatar
  • 1,421
0 votes
0 answers
246 views

Strange/Buggy Characters in the Terminal

I use Debian SID and the Terminator is my terminal emulator. After updating the system the last time (yesterday 2023/11/22) and rebooting, some characters in my terminal in certain commands are ...
rhuanpk's user avatar
  • 415
2 votes
1 answer
779 views

How to change ISO-8859 text with CRLF line terminators

I am trying to change the encoding of a text file to UTF-8. I am new to Unix so any help would be great. I am not sure how to code it properly with recode function.
Warrenkevin Henderson's user avatar
0 votes
1 answer
114 views

How to preserve non-ASCII characters?

We have default POSIX locale in our server but when non-ASCII character like רקטות לגוש דן וירושלים(hebrew) uploaded in server its getting changes to רק××ת ×××ש ×× ××ר×ש×××, How can preserve it ...
Amrita's user avatar
  • 1
2 votes
1 answer
398 views

Can awk be told to count the character string length rather than byte string length for '%10s' printf formats?

Try this for an output of |Ü| X|: echo 'Ü X' | awk '{printf("|% 2s|% 2s|\n", $1, $2)}' Obviously awk counts the byte length, not the character length of the Ü, so the count is 2 and no left ...
Harald's user avatar
  • 928
6 votes
1 answer
362 views

Converting from ISO-IR-87 to UTF-8 encoding

I am working on Debian and derivatives system. I'd like to convert from an original input ISO-IR-87 to UTF-8. Is there an easy way to do it ? For reference: % iconv -l | grep "IR-8" ISO-IR-8-...
malat's user avatar
  • 3,112
0 votes
1 answer
75 views

Why files are created with different encodings using vi

When I work with Bash scripts, I always create files using vi. I am debugging a script that in theory is written correcly and I decided to check the files I have created using the file command. This ...
Duck's user avatar
  • 4,714
-1 votes
1 answer
142 views

Incorrect shell output encoding

I'm not an expert in Linux, but I am following the development of a software that runs on Linux Buildroot. The device can only use the program for the graphical interface, access the shell, or connect ...
porrokynoa's user avatar
1 vote
0 answers
114 views

bash cursor moves command

In bash, when entering a diacritic, moving around the cursor messes up the command, making it hard to edit. I've noticed this on some servers, so I'm accessing them over ssh, but not directly on my ...
Herbert's user avatar
  • 241
3 votes
0 answers
277 views

MacOS files to NAS with rsync --iconv repeats the sync over and over

According to this hint and similar advice I am using the --iconv option in rsync (version 3.2.7) to sync file with umlauts (ä ü ö ...) to my Synology NAS. However the --iconv option does not work as ...
Haegar's user avatar
  • 31
0 votes
1 answer
1k views

Specifying Character Sets With The Curl Command

I am attempting to extract a list of Chinese characters from https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO to make a bash script. However, when I ran curl -o list.txt ...
James Norman 's user avatar
4 votes
4 answers
357 views

Collect chars from strings and print their unicode

Context (skip, if you don't care; read, if you suspect I'm totally on the wrong track) For an embedded system with small memory, I want to generate fonts which contain only those glyphs actually ...
Philippos's user avatar
  • 13.5k
0 votes
1 answer
1k views

How to enable all locales (character encodings) on Debian 11 (over SSH, in Bash)?

I am unsure if I should not better ask: How to display Windows UTF-16 encoding in Linux? Since I just found out that Windows uses UTF-16LE to be exact: https://stackoverflow.com/q/66072117/1997354 I ...
Vlastimil Burián's user avatar
1 vote
1 answer
64 views

Command similar to ascii for ascii extended and/or for unicode?

ascii command in Linux is fast and great. It allows us to search for a character or for a code point and returns all relevant results for a given search. Is there something similar for ASCII extended (...
demacj's user avatar
  • 13
0 votes
1 answer
62 views

Do shells have a command equivalent of the "|" pipe symbol? I need something like that in Emacs org-mode tables

I like to construct commands which use the | pipe symbol in org tables and | symbol messes them up. Do bash et al have a text command that is the equivalent of the symbol? Are there alternative ways ...
vfclists's user avatar
  • 7,619
3 votes
1 answer
274 views

Different encoding/Unicode interpretation using terminal vs using shell script

I was working on a keymap script (map keys from one language keyboard layout to another). And after a lot of hard time trying to get everything working I found out that different characters are ...
Andrew15_5's user avatar
3 votes
0 answers
167 views

Unnormalized UTF-8 directory names

I noticed something interesting in one of my directories: $ ls -li total 36 2625309 drwxrwxr-x 2 dotancohen dotancohen 4096 Jul 4 2022 Español 2625385 drwxrwxr-x 2 dotancohen dotancohen 4096 Jul ...
dotancohen's user avatar
  • 16.1k
0 votes
1 answer
74 views

testdisk utility reports nonexistent files from a exFAT drive used with Windows - why?

I tried to recover lost files from an exFAT thumb drive with the testdisk package on linux. It was very good at finding deleted files. However as I went through the entries, I saw weird entries. The ...
ero47543's user avatar
0 votes
0 answers
76 views

Unable to switch encoding in GNOME terminal

In GNOME terminal version: 3.40.3 fro GNOME 40,installed with RHEL 9.1 I Set two kinds of character encodinng and background color in different Profile but only the background changed after switch ...
yeangigi's user avatar
0 votes
0 answers
100 views

Repairing mixed encoding

I got some files containing Finnish text with mixed encoding, something one would get by (echo Mäntysalo ; echo Mäntysalo | recode utf-8..iso-8859-1) > problem.txt. Is there a "right" way ...
Jori Mäntysalo's user avatar
1 vote
2 answers
1k views

Print ASCII extended table with C++ (won't print cyrillics - ukrainian letters)

So, I am to print ASCII extended table in C++, it must include Ukrainian letters (cyrilics). I am on KDE neon 5.26 linux distribution, it is based on Ubuntu. Terminal is KDE's Console obiviously. I ...
Vladyslav Rehan's user avatar
0 votes
1 answer
388 views

How can i give a program raw binary byte input which is produced by a different file?

I have 2 programs: x - prompts user for input from stdin. binary - prints something to stdout, the stuff it prints is made up of various raw binary bytes which are not fully supported by my terminals ...
sirix's user avatar
  • 1
7 votes
2 answers
776 views

What is "modifier" in locale name?

The arch wiki page says: Locale names are typically of the form language[_territory][.codeset][@modifier] Unfortunately it does not says what is "modifier". What is that "modifier&...
illiterate's user avatar
2 votes
1 answer
110 views

Is my text mangled beyond repair?

My mangled Czech text: NOTE ON CZECH BIRTH NUMBER VALIDATION IN CZECH LANGUAGE; in Czechia birth number = personal identification number ======================================================== Do ...
Vlastimil Burián's user avatar
2 votes
1 answer
983 views

How to set character encoding for OpenBSD tty session?

On OpenBSD, in order to appropriately see language specific characters in X sessions all I need to do is set LC_CTYPE to one of the values listed using locale -a. I do this by adding export LC_CTYPE=...
zajer's user avatar
  • 213
0 votes
1 answer
28 views

when I use tab on rsync and remote pc magical characters happear in display

Simple. I have the file "longname.server" on remote pc, I want to copy on my pc, but..I don't remind the name because is long and I use tab completion. \rsync -avP remote:^[\\\[0\\\;...
elbarna's user avatar
  • 12.8k
1 vote
1 answer
69 views

head and grep display text different than gedit

I try to get some information from log file, but I get numbers using grep and I want get text like I see in gedit. Result of file command: ASCII text, with CR, LF line terminators Also I checked that ...
Jarek's user avatar
  • 308
1 vote
1 answer
47 views

Chaotic Command-line Interface Layout

When I type a long command on a command-line interface. Something strange may happen in the layout. The characters I typed don't show in lines correctly. Instead, they merge into 1 line or overwrite ...
user avatar
0 votes
2 answers
141 views

get only the visible output of a terminal-interpreted string

I am developping a script that will allow me to find easily which columns are "only spaces", by creating a mask showing any column with anything other than a space. To do this: I print each ...
Olivier Dulac's user avatar
1 vote
0 answers
73 views

GNU Recode - Properly decode mixed HTML Character/Numeric encoded text?

I recently found GNU recode as something that can be used to decode HTML entities, however when looking at a piece of malware I noticed that it appears to be mixed HTML character/entity encoding, such ...
4oo4's user avatar
  • 153
2 votes
2 answers
720 views

replacement for tr with utf-8 capabilities

in order to isolate the last word in any line of a poem (to have a list of all rhymes), I put together several snippets of code obtaining this awk '{print $NF}' input.txt | tr 'A-Z' 'a-z' | tr -sc 'a-...
Dingo's user avatar
  • 329
1 vote
1 answer
2k views

Characters do not display properly in terminal (st)

When typing German umlauts (ä,ö,ü) into the terminal (I am using st on Arch Linux, $XTERM is st-256color), it displays only <ffffffff>. Locale seems to be set properly. Output of locale is ...
l2poca's user avatar
  • 13
2 votes
1 answer
825 views

VIM shows ^@ every other character and ^M^@ at the end of line

Sorry if it's answered somewhere, I have no idea how to look for it. I received a series of reports from a bank that I'm supposed to process and they seem to be... badly encoded? First two lines in ...
cprn's user avatar
  • 1,025
1 vote
2 answers
3k views

Fix character encoding chaos

I‘m building a script to extract some data from a website with broken character encoding: The html header claims it‘s iso-8859-1, but it‘s not wgetting the file shows me it‘s actually utf-8, but with ...
Philippos's user avatar
  • 13.5k
1 vote
1 answer
2k views

Ubuntu and Debian have different character encoding for filenames

I am mounting a NFS filesystem to a Debian, an Amazon Linux and an Ubuntu file system and they are showing different character encoding for file name. An example file is displayed as follows in the ...
madu's user avatar
  • 135
0 votes
1 answer
1k views

Unable to change file encoding to utf-8

I modified a UTF-8 encoded xml file using vi editor and saved it. I'm on Redhat Linux 7.9 I checked the file encoding after the changes and found it to be us-ascii file --mime-encoding tmpfiles/08/...
Ashar's user avatar
  • 511
0 votes
4 answers
780 views

>= unary operator expected or -eq integer expression expected

monterey2="12.0" macOS=$(sw_vers -productVersion) if [ "$macOS" >= "$monterey2" ] ; then echo "Monterey" else echo "not Monterey" fi exit 0 ...
Lucy's user avatar
  • 1
2 votes
1 answer
590 views

How to find a locale name for a given character encoding?

The man page for setlocale seems to say that the language code and the character encoding name are sufficient to set the appropriate locale: A locale name is typically of the form language[_territory]...
Dmitry Grigoryev's user avatar
3 votes
2 answers
4k views

How to substitute non printable characters with space character in a file

I have read nearly every answer about this topic on this website or Stackoverflow but didn't manage to solve the issue below. When I copy the text from a PDF file and paste it into a text file file....
Amazigh_05's user avatar
2 votes
1 answer
4k views

How to fix the UTF-8 character encoded filenames which don't look good in sub directories

I have filenames like Käyttöohje.pdf. This should be Käyttöohje.pdf. I can do the conversion of all files in sub directories with the command: convmv -f utf8 -t iso-8859-1 -r --notest * This ...
George's user avatar
  • 23
5 votes
1 answer
1k views

View file names in hex?

I have some files with special characters like accented letters. They are valid names, but for some reason when they are copied across the network to a drive, maybe in another format, the name still ...
DAB's user avatar
  • 475
0 votes
0 answers
27 views

Weird characters in output file [duplicate]

I opened the output file created from a script in my terminal and the output file is fine, shows the output i want. cat 2_val.txt 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 ...
Basil's user avatar
  • 63
1 vote
2 answers
2k views

How do you execute ls -la when encoded as hex?

There used to be a really bad joke from the internet where you would copy something starting with \x0B\x2E... (just an example) and when you execute this command the terminal would interpret it as ...
Sir Muffington's user avatar

1
2 3 4 5
9