The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

How can i find out which characters that are transliterated with iconv translit

Once a day my server needs to process a huge csv text file. It has to be converted from UTF8 to ISO8859-1 using iconv. Sometimes there has been special characters like ń in the file and this is ...
0
votes
0answers
25 views

I want to convert encoding using iconv

I want to use the iconv command (Linux) to switch to a multi-type encoding file. (Encoding type that wants to be changed) UTF-16 UTF-16BE UTF-16LE CP949 EUC-KR EUC-JP US-ASCII Big5 ISO-2022-JP ISO-...
0
votes
3answers
74 views

Creating/using my own 'custom' character

Is it possible to create your own ascii character that you can use? If so, how would it be done? Now my use case would be something like this: Let's say I created the character #12345, and I am ...
4
votes
3answers
80 views

How to find character locations of a string in a file?

I need to search for a string (a sequence of characters) in a file with a certain encoding, typically utf8, but return the character offsets (not byte offsets) of the results. So this is a search ...
2
votes
1answer
146 views

SFTP remote readdir Failure - national characters in file name

I have created a directory with national characters in it's name. I can see and browse this directory via ssh # cd /root # ls СтранноеРусскоеИмя but can't do it via SFTP: sftp> cd /root sftp>...
-1
votes
1answer
29 views

How do we get the readable string from big endian hex string

How do we get the readable string from hex string 0049006e00740072006f of BE which means as normal word, ie. we dont have to swap the byte pair per word tried with xxd in vain, cause I dont know ...
0
votes
1answer
404 views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
2
votes
1answer
225 views

Disabling utf-8 in all terminals?

This question pertains to my Ubuntu 18 system, as follows ... % uname -a Linux myhost.name.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I don't ...
0
votes
0answers
77 views

Set default file encoding to ASCII?

Years ago, most text editing applications had the option to set the default file encoding to US-ASCII. Now I can't find any recent editor on Linux that allows you to choose ASCII. If the purpose is to ...
1
vote
2answers
299 views

ls won't display Cyrillic characters properly

I've got a problem with Cyrillic alphabet. My terminal will display Cyrillic normally as well most of my programs the only exception I've found so far is ls. Instead of normal Cyrillic it displays ????...
-1
votes
1answer
116 views

Can't see the contents of a data file in terminal

I have downloaded a number of input files from the NASDAQ website (for example ftp://emi.nasdaq.com/ITCH/01302018.NASDAQ_ITCH50.gz) After decompressing (I double click on a Mac) I can't see the file ...
0
votes
0answers
249 views

How can I import csv file with special characters?

I am trying to import CSV sheet to Postgres database using PG-ADMIN. I am able to import sheet but after importing sheet due to special characters values are changed. For example: Before Importing: ...
1
vote
1answer
51 views

(Possible) inconsistent behavior of grep and less

I have a utf-8 file containing some Turkish text inside. (My system is MacOSX) $ file -I foo.merge $foo.merge: text/plain; charset=utf-8 When I try to see some Turkish specific characters by using ...
0
votes
0answers
217 views

How to open windows txt with cyrillic text?

How to open Windows text file that have Cyrillic text ? If I open with mousepad there is understandable text. If I type file -bi test.txt text/plain; charset=iso-8859-1
7
votes
3answers
3k views

Strange character in a file

I have an UTF-8 file that contains a strange character -- visible to me just as <96> This is how it appears on vi and how it appears on gedit and how it appears under LibreOffice and that ...
2
votes
1answer
60 views

UTF8 Character Makes File Inaccessible

If I run: scp me@example.com:/home/me/cömmön_file.jpg /home/me/ from my remote server I get: scp: /home/me/cömmön_file.jpg: No such file or directory If I swap out the utf8 characters though ...
0
votes
3answers
95 views

Demonstrate file encoding

I am developing some training and would like to demonstrate file encoding. What I am trying to accomplish is create a text file with one type of encoding that show up as nonsense when read by Linux. ...
0
votes
1answer
425 views

Convert text file to non-iso extended ascii file

I have a file which is Non-ISO Extended ASCII type. I can't see its content just by opening it normally via cat or vim. However, I'm able to see its content in readable form with this command... od -...
0
votes
1answer
260 views

Choosing mousepad encoding

I have some text file with windows-1251 encoding (rus), but when I open it in mousepad there is no such encoding in the drop-down list, and there is no utf-8 also. The question is: how to add ...
0
votes
0answers
101 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
2
votes
1answer
268 views

Can't get iso-8859-1 to work for 'umlauts' (Debian / Plesk)

I think I have configured ISO-8859-1 at all necessary locations, but still, my pages are displaying black '?' instead of German umlauts! My server runs Plesk 17.8.11 / Debian 8 / Apache 2 / PHP 7.1 (...
3
votes
1answer
189 views

CONVMV and cyrillic filenames

I am trying to convert filenames in russian zipfile to UTF-8 using convmv. Original filename: "æ óá¡¿½∞¡δ¼ áα«¼áΓ«¼.jpg" (æ\ óá¡¿½∞¡δ¼\ áα«¼áΓ«¼.jpg with slashes) This analyzer (https://2cyr.com/...
0
votes
1answer
62 views

Is a POSIX-conformant implementation required to support other than default locales?

I would like to implement a minimal POSIX-conformant make utility. However, the specification mentions that the implementation shall be affected by the following environment variables: LANG, LC_ALL, ...
2
votes
2answers
141 views

How to generate a ascii char in a string with Bash / ZSH correctly?

I'm trying to generate a string of 10 followed by a NULL (\0) followed by a 10. However echo "10\010" does not seem to work (I'm guessing it generates two chars - a 10 followed by \010. I'm not sure ...
0
votes
1answer
84 views

AIX x Linux line endings

Using Meld to compare two identical files in terms of contents, it give a warning like this: ... There was a problem opening the file "...\CHXXXX_...\psart20p.pc". There was an encoding ...
0
votes
1answer
208 views

Filename displays with ? char when `ls` while with junk character when `ls | grep fran`?

What is the configuration that defines how file name with unrecognizable character be replaced with some other character. For example in my Linux system same file is displayed differently in UTF8 ...
2
votes
1answer
94 views

Encoding problems with German man pages

On the machine I am working on I have English man pages under /usr/share/man/and German ones under /usr/share/man/de. The former are encoded with us-ascii, the latter ones with utf-8. Unfortunately ...
1
vote
1answer
225 views

Strange Konsole character encoding behavior

I set the character encoding of Konsole in Lubuntu to UTF-16 (not sure if Konsole uses UTF-16 little endian or UTF-16 big endian). Then I run the following command in the Run window: Then I pressed ...
1
vote
1answer
104 views

Can I write a console program that works with multiple character encodings? [duplicate]

I am writing a console program in C. I expect the Terminal that my program is running in to have its character encoding set to UTF-8. This means that I am sending UTF-8 encoded strings to the ...
1
vote
3answers
6k views

What's the right way to base64 encode a binary file on CentOS 7?

I'm using CentOS 7 with bash shell. I thought base64-encoding a binary file would be as simple as [rails@server lib]$ cat mybinary.file | base64 > /tmp/output.base64 However, I notice when I ...
1
vote
0answers
33 views

Can the shell know what is the character encoding of the terminal? [duplicate]

Some terminals allow you to set their character encoding, but can the shell (or whatever program running on the other end) know what is the character encoding of the terminal?
0
votes
1answer
129 views

Mutt and character sets issue

I updated to mutt 1.10.0 and now my character sets are all messed up. They worked fine with 1.5.21 -- system version and now old. Following the FAQ, I have these set up: ; echo $TERM ...
2
votes
1answer
180 views

Why is sed failing to follow collate order?

On this command: $ echo "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" | > sed 's/[a-z]//g' ABCDEFGHIJKLMNOPQRSTUVWXYZ It seems correct, doesn't it? However, the locale in effect is ...
0
votes
2answers
68 views

How to only keep BMP in the utf-8 text file? [duplicate]

This is the content of a text file, and I want to only keep BMP. 这是无效的字符 𡛂 这是正常的字符 哈 In the How to remove non UTF-8 characters from text file , I find there is a solution, but it doesn't work ...
1
vote
1answer
534 views

Tmux is displaying strange characters

I'm having an issue with tmux where it's displaying these strange characters I've found a few posts the seemed to be talking about the same issue and they suggested that I need to change my character ...
0
votes
1answer
95 views

Different output when executing script manually and via cron/jenkins job

I have a shell script which opens sqlplus and select some data from db containing chinese characters. When I execute it manually, the output contains those characters. When I do it via cron or jenkins ...
2
votes
1answer
506 views

Unexpected non-null encoding of /proc/<pid>/cmdline

I am parsing the /proc/pid/cmdline value for a number of processes on my Linux system (Ubuntu 16.04) and have found that while most of the entries are null-encoded, as expected, at least one uses ...
4
votes
1answer
1k views

Making grep understand byte escapes

I'm trying to match against some UTF-8 characters. The problem is grep doesn't translate \x byte escapes, so this fails: echo -e '\xd8\xaa' | grep -P '\xd8\xaa' while this succeeds: echo -e '\xd8\...
2
votes
1answer
754 views

How to translate Unicode characters? [duplicate]

I'm trying to convert some characters to the fullwidth form like this tr 'abcdefghijklmnopqrstuvwxyz' 'abcdefghijklmnopqrstuvwxyz' However, it doesn't work. I did a search and it turns out tr doesn'...
0
votes
1answer
365 views

MySQL: Incorrect string value

I have been having some problems with some characters in webpages, that we have in UTF8, often in a while. I am running MySQL 5.7 with Debian 9. My databases are using the utf8charset. Today, when ...
2
votes
3answers
842 views

Delete folder with invalid character (-encoding)

On my Linux Mint 18.3 installation I had troubles with my locale setting which caused the generation of a folder that has invalid encoding in its name. I would now like to delete said folder but I ...
4
votes
1answer
2k views

Problems with umlauts (can't type in terminal + can't open files)

On my laptop I have Linux Mint 18.3 installed and right now I am experiencing the following problem: As a German I happen to have some files (or directories) that contain umlauts in their names (which ...
1
vote
1answer
408 views

How to make `less` understand codepage?

LC_ALL=ru_RU.CP1251 less ~/t/show_discounts.js shows nonsense characters instead of showing this CP1251 file as it should be viewed. How to make less work well with specified encoding? Debian Linux,...
3
votes
2answers
453 views

Generate collating order of a list of individual characters

In some cases, the collating order of each individual character needs to be known (to be used). It is commonly expressed in the character class of a regex like [b-d]. That character class will match ...
5
votes
1answer
638 views

Which Chinese locales are needed to avoid missing or mismatched characters?

Arch Linux lists the following different Chinese locales in /etc/locale.gen: #zh_CN.GB18030 GB18030 #zh_CN.GBK GBK #zh_CN.UTF-8 UTF-8 #zh_CN GB2312 #zh_HK.UTF-8 UTF-8 #zh_HK BIG5-HKSCS #zh_SG.UTF-8 ...
5
votes
3answers
5k views

How to print a variable that contains unprintable characters?

I want to display the value of the $IFS variable, which could contain unprintable characters (for example: newline). I used the following command to do so: echo -n "$IFS" | hexdump -C Which worked ...
0
votes
2answers
296 views

How to change encoding for an unnamed buffer in VIM?

Is it possible? I only know how it can be done for a named buffer (file). It should be reopened with ++enc=<encoding> option. But an unnamed buffer cannot be reopened. I executed :set ...
2
votes
2answers
503 views

sed remove spaces (multiple encoding) from end of line

I like remove spaces from the beginning and the end of line. result="$(mysql -r --column-names=0 -e "SELECT String FROM Table")" echo $result | cat -E +stringA +StringB $ echo $result | sed 's/^\(...
1
vote
1answer
60 views

Different cat Output for Hard Linked File over SSH? [closed]

I have a single initial file created using a C program on OpenSUSE Leap VirtualBox VM being accessed via ssh from OS X. The file is created with mkstemp() that a hard link is created for using link(). ...
5
votes
4answers
1k views

printf: multibyte characters

When trying to format printf output involving strings containing multi-byte characters, it became clear that printf does not count literal characters but the number of bytes, which makes formatting ...