Questions tagged [character-encoding]
Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.
398
questions
-1
votes
1
answer
73
views
Incorrect shell output encoding
I'm not an expert in Linux, but I am following the development of a software that runs on Linux Buildroot. The device can only use the program for the graphical interface, access the shell, or connect ...
1
vote
0
answers
60
views
bash cursor moves command
In bash, when entering a diacritic, moving around the cursor messes up the command, making it hard to edit.
I've noticed this on some servers, so I'm accessing them over ssh, but not directly on my ...
- 241
2
votes
0
answers
67
views
MacOS files to NAS with rsync --iconv repeats the sync over and over
According to this hint and similar advice I am using the --iconv option in rsync (version 3.2.7) to sync file with umlauts (ä ü ö ...) to my Synology NAS. However the --iconv option does not work as ...
- 21
0
votes
1
answer
407
views
Specifying Character Sets With The Curl Command
I am attempting to extract a list of Chinese characters from https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO to make a bash script. However, when I ran
curl -o list.txt ...
4
votes
4
answers
262
views
Collect chars from strings and print their unicode
Context (skip, if you don't care; read, if you suspect I'm totally on the wrong track)
For an embedded system with small memory, I want to generate fonts which contain only those glyphs actually ...
- 13.3k
0
votes
1
answer
390
views
How to enable all locales (character encodings) on Debian 11 (over SSH, in Bash)?
I am unsure if I should not better ask:
How to display Windows UTF-16 encoding in Linux?
Since I just found out that Windows uses UTF-16LE to be exact:
https://stackoverflow.com/q/66072117/1997354
I ...
- 27.8k
0
votes
1
answer
37
views
Command similar to ascii for ascii extended and/or for unicode?
ascii command in Linux is fast and great. It allows us to search for a character or for a code point and returns all relevant results for a given search. Is there something similar for ASCII extended (...
- 3
0
votes
1
answer
49
views
Do shells have a command equivalent of the "|" pipe symbol? I need something like that in Emacs org-mode tables
I like to construct commands which use the | pipe symbol in org tables and | symbol messes them up. Do bash et al have a text command that is the equivalent of the symbol?
Are there alternative ways ...
- 7,235
3
votes
1
answer
187
views
Different encoding/Unicode interpretation using terminal vs using shell script
I was working on a keymap script (map keys from one language keyboard layout to another). And after a lot of hard time trying to get everything working I found out that different characters are ...
- 111
3
votes
0
answers
107
views
Unnormalized UTF-8 directory names
I noticed something interesting in one of my directories:
$ ls -li
total 36
2625309 drwxrwxr-x 2 dotancohen dotancohen 4096 Jul 4 2022 Español
2625385 drwxrwxr-x 2 dotancohen dotancohen 4096 Jul ...
- 15.6k
0
votes
1
answer
42
views
testdisk utility reports nonexistent files from a exFAT drive used with Windows - why?
I tried to recover lost files from an exFAT thumb drive with the testdisk package on linux. It was very good at finding deleted files. However as I went through the entries, I saw weird entries. The ...
- 1
0
votes
0
answers
31
views
Unable to switch encoding in GNOME terminal
In GNOME terminal version: 3.40.3 fro GNOME 40,installed with RHEL 9.1
I Set two kinds of character encodinng and background color in different Profile but only the background changed after switch ...
- 1
0
votes
0
answers
54
views
Repairing mixed encoding
I got some files containing Finnish text with mixed encoding, something one would get by (echo Mäntysalo ; echo Mäntysalo | recode utf-8..iso-8859-1) > problem.txt. Is there a "right" way ...
- 153
0
votes
2
answers
732
views
Print ASCII extended table with C++ (won't print cyrillics - ukrainian letters)
So, I am to print ASCII extended table in C++, it must include Ukrainian letters (cyrilics).
I am on KDE neon 5.26 linux distribution, it is based on Ubuntu. Terminal is KDE's Console obiviously.
I ...
- 101
0
votes
1
answer
161
views
How can i give a program raw binary byte input which is produced by a different file?
I have 2 programs:
x - prompts user for input from stdin.
binary - prints something to stdout, the stuff it prints is made up of various raw binary bytes which are not fully supported by my terminals ...
- 1
2
votes
1
answer
100
views
Is my text mangled beyond repair?
My mangled Czech text:
NOTE ON CZECH BIRTH NUMBER VALIDATION IN CZECH LANGUAGE;
in Czechia birth number = personal identification number
========================================================
Do ...
- 27.8k
2
votes
1
answer
435
views
How to set character encoding for OpenBSD tty session?
On OpenBSD, in order to appropriately see language specific characters in X sessions all I need to do is set LC_CTYPE to one of the values listed using locale -a. I do this by adding export LC_CTYPE=...
- 213
0
votes
1
answer
24
views
when I use tab on rsync and remote pc magical characters happear in display
Simple.
I have the file "longname.server" on remote pc, I want to copy on my pc, but..I don't remind the name because is long and I use tab completion.
\rsync -avP remote:^[\\\[0\\\;...
- 12.1k
1
vote
1
answer
53
views
head and grep display text different than gedit
I try to get some information from log file, but I get numbers using grep and I want get text like I see in gedit.
Result of file command:
ASCII text, with CR, LF line terminators
Also I checked that ...
- 198
1
vote
1
answer
43
views
Chaotic Command-line Interface Layout
When I type a long command on a command-line interface. Something strange may happen in the layout. The characters I typed don't show in lines correctly. Instead, they merge into 1 line or overwrite ...
user526448
0
votes
2
answers
83
views
get only the visible output of a terminal-interpreted string
I am developping a script that will allow me to find easily which columns are "only spaces", by creating a mask showing any column with anything other than a space.
To do this: I print each ...
- 6,025
1
vote
0
answers
54
views
GNU Recode - Properly decode mixed HTML Character/Numeric encoded text?
I recently found GNU recode as something that can be used to decode HTML entities, however when looking at a piece of malware I noticed that it appears to be mixed HTML character/entity encoding, such ...
- 153
2
votes
2
answers
545
views
replacement for tr with utf-8 capabilities
in order to isolate the last word in any line of a poem (to have a list of all rhymes), I put together several snippets of code obtaining this
awk '{print $NF}' input.txt | tr 'A-Z' 'a-z' | tr -sc 'a-...
- 329
1
vote
1
answer
1k
views
Characters do not display properly in terminal (st)
When typing German umlauts (ä,ö,ü) into the terminal (I am using st on Arch Linux, $XTERM is st-256color), it displays only <ffffffff>. Locale seems to be set properly.
Output of locale is
...
- 13
2
votes
1
answer
587
views
VIM shows ^@ every other character and ^M^@ at the end of line
Sorry if it's answered somewhere, I have no idea how to look for it. I received a series of reports from a bank that I'm supposed to process and they seem to be... badly encoded?
First two lines in ...
- 1,015
1
vote
2
answers
2k
views
Fix character encoding chaos
I‘m building a script to extract some data from a website with broken character encoding:
The html header claims it‘s iso-8859-1, but it‘s not
wgetting the file shows me it‘s actually utf-8, but with ...
- 13.3k
0
votes
1
answer
1k
views
Ubuntu and Debian have different character encoding for filenames
I am mounting a NFS filesystem to a Debian, an Amazon Linux and an Ubuntu file system and they are showing different character encoding for file name.
An example file is displayed as follows in the ...
- 125
0
votes
1
answer
873
views
Unable to change file encoding to utf-8
I modified a UTF-8 encoded xml file using vi editor and saved it.
I'm on Redhat Linux 7.9
I checked the file encoding after the changes and found it to be us-ascii
file --mime-encoding tmpfiles/08/...
- 459
0
votes
4
answers
573
views
>= unary operator expected or -eq integer expression expected
monterey2="12.0"
macOS=$(sw_vers -productVersion)
if [ "$macOS" >= "$monterey2" ] ; then
echo "Monterey"
else
echo "not Monterey"
fi
exit 0
...
- 1
2
votes
1
answer
456
views
How to find a locale name for a given character encoding?
The man page for setlocale seems to say that the language code and the character encoding name are sufficient to set the appropriate locale:
A locale name is typically of the form language[_territory]...
- 7,135
3
votes
2
answers
3k
views
How to substitute non printable characters with space character in a file
I have read nearly every answer about this topic on this website or Stackoverflow but didn't manage to solve the issue below.
When I copy the text from a PDF file and paste it into a text file file....
- 405
2
votes
1
answer
3k
views
How to fix the UTF-8 character encoded filenames which don't look good in sub directories
I have filenames like Käyttöohje.pdf. This should be Käyttöohje.pdf.
I can do the conversion of all files in sub directories with the command:
convmv -f utf8 -t iso-8859-1 -r --notest *
This ...
- 23
5
votes
1
answer
1k
views
View file names in hex?
I have some files with special characters like accented letters.
They are valid names, but for some reason when they are copied across the network to a drive, maybe in another format, the name still ...
- 465
0
votes
0
answers
27
views
Weird characters in output file [duplicate]
I opened the output file created from a script in my terminal and the output file is fine, shows the output i want.
cat 2_val.txt
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
...
- 63
1
vote
2
answers
1k
views
How do you execute ls -la when encoded as hex?
There used to be a really bad joke from the internet where you would copy something starting with \x0B\x2E... (just an example) and when you execute this command the terminal would interpret it as ...
- 739
3
votes
1
answer
675
views
ASCII to ANSI convertion does not work
Trying to convert an ASCII file to ANSI on Linux:
$ file -bi relocate.xml
text/xml; charset=us-ascii
$ iconv -f us-ascii -t ms-ansi < relocate.xml | file -bi -
text/xml; charset=us-ascii
$ cat ...
- 1,991
1
vote
1
answer
262
views
Why is the special character escaped in the command line
I have an AlmaLinux server, in which I struggle with some encoding issue.
For example, if I type this in the CLI:
mkdir asdfü
ls ./
I get
'asdf'$'\303\274'
How can I fix this?
- 473
0
votes
0
answers
80
views
File with Multiple ISO encodings
I have an Arabic file with multiple ISO encodings; encoded in ISO-8859-6 and a few characters encoded in ISO-8859-1 (Because they don't have an equivalent in ISO-8859-6). I want to convert this whole ...
- 233
8
votes
2
answers
3k
views
Problems with UTF-8 when attaching to a tmux session over ssh
Scenario:
I have a server, on which a tmux session is running. Let's call that session tmux_session in the rest of this post.
Attaching to that session can be done in two, slightly different ways:
By ...
- 11.2k
2
votes
1
answer
196
views
Can I determine at runtime the available NLS character sets?
I can find the list of the kernel's supported filesystem types in /proc/filesystems.
Is there a way to determine which character sets I'm able to use with the mount options iocharset or nls, other ...
- 8,462
0
votes
0
answers
273
views
Unicode Characters Not Displaying after scp to Server
I want to upload some html files to my webserver (running Debian 10 buster) from my laptop (running Ubuntu 20.04). I did so by using scp to transfer the files as a tarball to the remote server.
For ...
- 101
0
votes
1
answer
682
views
cryptsetup with veracrypt container: set encoding?
I'm trying to unlock (and later mount) a veracrypt container like this:
sudo cryptsetup --type tcrypt --veracrypt open /home/me/my_container.vc my_container
sudo mount /dev/mapper/my_container /media/...
- 103
1
vote
0
answers
463
views
Strace displaying results in ASCII only at process ending and not runtime
I'm searching a way for strace to print the content of the write(...) syscall to ASCII and not useless bytes !
The Strace command I use :
sudo strace -e write=1 -e trace=write -s9999 -p 551 2>&...
- 133
2
votes
0
answers
370
views
extracting filenames in ubuntu with non-latin encoding which were archived in windows
This issue has been asked quite a lot (e.g q1, q2), but is still far from solved , at least in Ubuntu 20.04 with the default zip tools (zip, file-roller).
The zip files are created in Windows, and in ...
- 21
1
vote
1
answer
3k
views
Finding text file's encoding
The iconv command may change file encodings. But is there a command to find the file encoding of a certain text file. Also, I am trying to find what file encoding standards each country use. So that I ...
- 113
1
vote
1
answer
804
views
symbols display wrong in `screen` - ascii or utf
I don't know since when and why :
$ mkdir -p a/b/c x/y/{1,2,3}
$ tree
.
├── a
│ └── b
│ └── c
└── x
└── y
├── 1
├── 2
└── 3
8 directories, 0 files
when I run ...
- 13.9k
2
votes
1
answer
1k
views
How to replace Unicode code numbers by UTF-8 characters in a number of files?
I have 500 files that contain escaped Unicode characters like so:
Albert Vel\u00e1zquez
V\u00e9ronique Ekin
etc. The data was queried from a server by a script through XMLHttpRequests and I assume ...
- 8,536
2
votes
1
answer
226
views
How are characters encoded in stty's output?
I ran stty --all on a terminal which had been reconfigured by a badly behaved process which exited before putting my terminal back to its original settings. Part of the output reads: eol = M-^?;. What ...
- 125
0
votes
1
answer
595
views
File is having lots of special characters. How to replace?
I have a file with record like:
CENTURY®KETTLEBELLS
COLEMANTT40°BAG
AlphaPro2¾TDLE
Lot™Stretch
M6×30
It shows normally in notepad files but in terminal when i cat it, it shows:
CENTURY▒KETTLEBELL
...
- 13
17
votes
1
answer
2k
views
Why did {} start appearing as äå in Terminal.app?
I'm running CentOS 7.9 and today my terminal is showing weird characters. Some letters look fine, but some symbols appear as some non-English characters. For example I made a text file with the ...
- 533