Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

2
votes
1answer
34 views

Disabling utf-8 in all terminals?

This question pertains to my Ubuntu 18 system, as follows ... % uname -a Linux myhost.name.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I don't ...
0
votes
0answers
46 views

Set default file encoding to ASCII?

Years ago, most text editing applications had the option to set the default file encoding to US-ASCII. Now I can't find any recent editor on Linux that allows you to choose ASCII. If the purpose is to ...
1
vote
2answers
48 views

ls won't display Cyrillic characters properly

I've got a problem with Cyrillic alphabet. My terminal will display Cyrillic normally as well most of my programs the only exception I've found so far is ls. Instead of normal Cyrillic it displays ????...
-1
votes
1answer
45 views

Can't see the contents of a data file in terminal

I have downloaded a number of input files from the NASDAQ website (for example ftp://emi.nasdaq.com/ITCH/01302018.NASDAQ_ITCH50.gz) After decompressing (I double click on a Mac) I can't see the file ...
0
votes
0answers
42 views

How can I import csv file with special characters?

I am trying to import CSV sheet to Postgres database using PG-ADMIN. I am able to import sheet but after importing sheet due to special characters values are changed. For example: Before Importing: ...
1
vote
1answer
39 views

(Possible) inconsistent behavior of grep and less

I have a utf-8 file containing some Turkish text inside. (My system is MacOSX) $ file -I foo.merge $foo.merge: text/plain; charset=utf-8 When I try to see some Turkish specific characters by using ...
0
votes
0answers
63 views

How to open windows txt with cyrillic text?

How to open Windows text file that have Cyrillic text ? If I open with mousepad there is understandable text. If I type file -bi test.txt text/plain; charset=iso-8859-1
7
votes
3answers
1k views

Strange character in a file

I have an UTF-8 file that contains a strange character -- visible to me just as <96> This is how it appears on vi and how it appears on gedit and how it appears under LibreOffice and that ...
2
votes
1answer
44 views

UTF8 Character Makes File Inaccessible

If I run: scp me@example.com:/home/me/cömmön_file.jpg /home/me/ from my remote server I get: scp: /home/me/cömmön_file.jpg: No such file or directory If I swap out the utf8 characters though ...
0
votes
3answers
68 views

Demonstrate file encoding

I am developing some training and would like to demonstrate file encoding. What I am trying to accomplish is create a text file with one type of encoding that show up as nonsense when read by Linux. ...
0
votes
1answer
186 views

Convert text file to non-iso extended ascii file

I have a file which is Non-ISO Extended ASCII type. I can't see its content just by opening it normally via cat or vim. However, I'm able to see its content in readable form with this command... od -...
0
votes
1answer
92 views

Choosing mousepad encoding

I have some text file with windows-1251 encoding (rus), but when I open it in mousepad there is no such encoding in the drop-down list, and there is no utf-8 also. The question is: how to add ...
0
votes
0answers
49 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
2
votes
1answer
166 views

Can't get iso-8859-1 to work for 'umlauts' (Debian / Plesk)

I think I have configured ISO-8859-1 at all necessary locations, but still, my pages are displaying black '?' instead of German umlauts! My server runs Plesk 17.8.11 / Debian 8 / Apache 2 / PHP 7.1 (...
3
votes
1answer
86 views

CONVMV and cyrillic filenames

I am trying to convert filenames in russian zipfile to UTF-8 using convmv. Original filename: "æ óá¡¿½∞¡δ¼ áα«¼áΓ«¼.jpg" (æ\ óá¡¿½∞¡δ¼\ áα«¼áΓ«¼.jpg with slashes) This analyzer (https://2cyr.com/...
0
votes
1answer
43 views

Is a POSIX-conformant implementation required to support other than default locales?

I would like to implement a minimal POSIX-conformant make utility. However, the specification mentions that the implementation shall be affected by the following environment variables: LANG, LC_ALL, ...
2
votes
2answers
95 views

How to generate a ascii char in a string with Bash / ZSH correctly?

I'm trying to generate a string of 10 followed by a NULL (\0) followed by a 10. However echo "10\010" does not seem to work (I'm guessing it generates two chars - a 10 followed by \010. I'm not sure ...
0
votes
1answer
50 views

AIX x Linux line endings

Using Meld to compare two identical files in terms of contents, it give a warning like this: ... There was a problem opening the file "...\CHXXXX_...\psart20p.pc". There was an encoding ...
0
votes
1answer
137 views

Filename displays with ? char when `ls` while with junk character when `ls | grep fran`?

What is the configuration that defines how file name with unrecognizable character be replaced with some other character. For example in my Linux system same file is displayed differently in UTF8 ...
2
votes
1answer
49 views

Encoding problems with German man pages

On the machine I am working on I have English man pages under /usr/share/man/and German ones under /usr/share/man/de. The former are encoded with us-ascii, the latter ones with utf-8. Unfortunately ...
1
vote
1answer
193 views

Strange Konsole character encoding behavior

I set the character encoding of Konsole in Lubuntu to UTF-16 (not sure if Konsole uses UTF-16 little endian or UTF-16 big endian). Then I run the following command in the Run window: Then I pressed ...
1
vote
1answer
103 views

Can I write a console program that works with multiple character encodings? [duplicate]

I am writing a console program in C. I expect the Terminal that my program is running in to have its character encoding set to UTF-8. This means that I am sending UTF-8 encoded strings to the ...
0
votes
3answers
2k views

What's the right way to base64 encode a binary file on CentOS 7?

I'm using CentOS 7 with bash shell. I thought base64-encoding a binary file would be as simple as [rails@server lib]$ cat mybinary.file | base64 > /tmp/output.base64 However, I notice when I ...
1
vote
0answers
33 views

Can the shell know what is the character encoding of the terminal? [duplicate]

Some terminals allow you to set their character encoding, but can the shell (or whatever program running on the other end) know what is the character encoding of the terminal?
0
votes
1answer
77 views

Mutt and character sets issue

I updated to mutt 1.10.0 and now my character sets are all messed up. They worked fine with 1.5.21 -- system version and now old. Following the FAQ, I have these set up: ; echo $TERM ...
2
votes
1answer
146 views

Why is sed failing to follow collate order?

On this command: $ echo "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" | > sed 's/[a-z]//g' ABCDEFGHIJKLMNOPQRSTUVWXYZ It seems correct, doesn't it? However, the locale in effect is ...
0
votes
2answers
45 views

How to only keep BMP in the utf-8 text file? [duplicate]

This is the content of a text file, and I want to only keep BMP. 这是无效的字符 𡛂 这是正常的字符 哈 In the How to remove non UTF-8 characters from text file , I find there is a solution, but it doesn't work ...
1
vote
1answer
277 views

Tmux is displaying strange characters

I'm having an issue with tmux where it's displaying these strange characters I've found a few posts the seemed to be talking about the same issue and they suggested that I need to change my character ...
0
votes
1answer
60 views

Different output when executing script manually and via cron/jenkins job

I have a shell script which opens sqlplus and select some data from db containing chinese characters. When I execute it manually, the output contains those characters. When I do it via cron or jenkins ...
0
votes
0answers
36 views

iconv surprising syntax [duplicate]

I used the iconv command recently, and I noticed that if I do iconv -f WINDOWS-1252 -t UTF-8 my_file.txt > my_file.txt , then the contents of my_file.txt are erased. If, however, I do it in two ...
0
votes
0answers
321 views

Text file (incorrectly) identified as binary with file -i

My problem is similar to this one but the solutions did not work for me. I have a csv-file which is identified as application/octet-stream; charset=binary via file -i. I used several commands to ...
1
vote
1answer
347 views

Unexpected non-null encoding of /proc/<pid>/cmdline

I am parsing the /proc/pid/cmdline value for a number of processes on my Linux system (Ubuntu 16.04) and have found that while most of the entries are null-encoded, as expected, at least one uses ...
4
votes
1answer
704 views

Making grep understand byte escapes

I'm trying to match against some UTF-8 characters. The problem is grep doesn't translate \x byte escapes, so this fails: echo -e '\xd8\xaa' | grep -P '\xd8\xaa' while this succeeds: echo -e '\xd8\...
2
votes
1answer
647 views

How to translate Unicode characters? [duplicate]

I'm trying to convert some characters to the fullwidth form like this tr 'abcdefghijklmnopqrstuvwxyz' 'abcdefghijklmnopqrstuvwxyz' However, it doesn't work. I did a search and it turns out tr doesn'...
0
votes
1answer
225 views

MySQL: Incorrect string value

I have been having some problems with some characters in webpages, that we have in UTF8, often in a while. I am running MySQL 5.7 with Debian 9. My databases are using the utf8charset. Today, when ...
1
vote
3answers
532 views

Delete folder with invalid character (-encoding)

On my Linux Mint 18.3 installation I had troubles with my locale setting which caused the generation of a folder that has invalid encoding in its name. I would now like to delete said folder but I ...
3
votes
1answer
1k views

Problems with umlauts (can't type in terminal + can't open files)

On my laptop I have Linux Mint 18.3 installed and right now I am experiencing the following problem: As a German I happen to have some files (or directories) that contain umlauts in their names (which ...
1
vote
1answer
221 views

How to make `less` understand codepage?

LC_ALL=ru_RU.CP1251 less ~/t/show_discounts.js shows nonsense characters instead of showing this CP1251 file as it should be viewed. How to make less work well with specified encoding? Debian Linux,...
1
vote
2answers
266 views

Generate the collating order of a string

It is known that the collating order in the C locale is the ASCII character numbering order (characters from 33 to 126): `^~<=>|_-,;:!?/.'"()[]{}@$*\&#%+...
4
votes
1answer
453 views

Which Chinese locales are needed to avoid missing or mismatched characters?

Arch Linux lists the following different Chinese locales in /etc/locale.gen: #zh_CN.GB18030 GB18030 #zh_CN.GBK GBK #zh_CN.UTF-8 UTF-8 #zh_CN GB2312 #zh_HK.UTF-8 UTF-8 #zh_HK BIG5-HKSCS #zh_SG.UTF-8 ...
4
votes
3answers
2k views

How to print a variable that contains unprintable characters?

I want to display the value of the $IFS variable, which could contain unprintable characters (for example: newline). I used the following command to do so: echo -n "$IFS" | hexdump -C Which worked ...
0
votes
2answers
204 views

How to change encoding for an unnamed buffer in VIM?

Is it possible? I only know how it can be done for a named buffer (file). It should be reopened with ++enc=<encoding> option. But an unnamed buffer cannot be reopened. I executed :set ...
2
votes
2answers
339 views

sed remove spaces (multiple encoding) from end of line

I like remove spaces from the beginning and the end of line. result="$(mysql -r --column-names=0 -e "SELECT String FROM Table")" echo $result | cat -E +stringA +StringB $ echo $result | sed 's/^\(...
0
votes
0answers
347 views

How can I insert the acute accent in Manjaro Linux?

I have two keyboard layouts installed: an English (US) and a Greek (extended). I can switch between the two languages and I am able to type without any problems. But, I am unable to insert accented ...
1
vote
1answer
46 views

Different cat Output for Hard Linked File over SSH? [closed]

I have a single initial file created using a C program on OpenSUSE Leap VirtualBox VM being accessed via ssh from OS X. The file is created with mkstemp() that a hard link is created for using link(). ...
6
votes
4answers
927 views

printf: multibyte characters

When trying to format printf output involving strings containing multi-byte characters, it became clear that printf does not count literal characters but the number of bytes, which makes formatting ...
3
votes
3answers
2k views

Bash convert \xC3\x89 to É?

I'm running iwlist wlo1 scan | grep ESSID inside a script. It displays French characters in the following format \xC3\x89 for É and \xC3\xA9 for é. I'm not sure what this format is called. I tried ...
0
votes
2answers
370 views

Bash - convert, encode integer to shortened alphanumeric string

Is there a way in BASH to convert an integer to a shortened alphanumeric string? If I start, let's say with var=20171019194210. I would like to get something like the following. echo "$var" | ...
2
votes
1answer
966 views

Netcat Bad Request [closed]

I'm using Ubuntu and I switched to netcat-traditional version but when I try to make simple HTTP requests it always failing, any cases, any servers giving 400 Bad request: (Apache Ubuntu is running ...
0
votes
1answer
343 views

Why is an explicit LANG=C required when searching for hex representations of characters in grep?

When I want to recursively search TeX files for characters unsupported by my font, I typically start with a search for non-breakable spaces and zero-width spaces. These are difficult to produce on the ...