Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
1
vote
2answers
2k views

Keyboard input encoding

I am trying to learn the basics of Unicode and UTF-8, and it so going very smoothly so far. I know that it is possible to chose the encoding of a file when opening it. When I type text using the ...
1
vote
1answer
60 views

Cannot convert from utf16be to utf8

I'm working on a script for aix 7.2 and I'm getting an error when I try to convert from utf16be to utf8. I'm using iconv to convert. Using the follow the script: for file in *.OPEN do echo "...
2
votes
1answer
189 views

How to write a luit command that outputs a file

I need to format a luit command so I can write a file that I'm trying to fix the encoding for. What I have right now is luit -encoding gbk cat santi.txt, but I would like this to have the output ...
3
votes
2answers
541 views

Generate collating order of a list of individual characters

In some cases, the collating order of each individual character needs to be known (to be used). It is commonly expressed in the character class of a regex like [b-d]. That character class will match ...
23
votes
9answers
36k views

How to print all printable ASCII chars in CLI?

How can I list all the printable ASCII characters in the terminal?
16
votes
3answers
5k views

How to convert an emoticon specified by a U+xxxxx code to utf-8?

Emoticons seem to be specified using a format of U+xxxxx wherein each x is a hexadecimal digit. For example, U+1F615 is the official Unicode Consortium code for the "confused face" 😕 As I am ...
4
votes
3answers
2k views

MP3 tags Cyrillic chars

I have some mp3 files with Cyrillic chars on file names and also on their tags. I'm using audacious to play them. See the file info below: Is it possible to change some encoding to show the contents ...
1
vote
0answers
36 views

Unknown character encoding for zip file in archive manager

Forenote: I am quite new to Linux, and am really just a beginner. I have downloaded a zip file from a respectable-looking site with firefox; it should contain stuff to install an application. The zip ...
3
votes
3answers
26k views

AddDefaultCharset ISO-8859-1 not work

I have upgrade Apache from 2.2 to 2.4 and I am having problems with character encoding. In my pages I have è=è ò=ò ì=ì etc... In my httpd.conf I have commented the default value and added ISO-...
2
votes
1answer
77 views

MySQL Database Charset

I have just migrated a MySQL database to a new server using mysqldump. The charset of database is normally UTF-8 and everything was working fine on the old server but when we switched over to new one, ...
0
votes
0answers
46 views

Converting text file encodings

I have been having a horrible time with text files that mostly have cp1252 encoding, but some don't. When I try to iterate over them in Python, I'll hit a file with some other weird encoding and the ...
0
votes
1answer
44 views

What should ASCII character 26 do?

I have code where I display ANSI art, I've found that last 128 bytes hold SAUCE meta data. The problem I have is just before the SAUCE data there is one character that looks out of place, that is ...
0
votes
0answers
38 views

How to find which encoding algorithm is used in the file

I have a file with encoded content. I am trying to figure out what encoding algorithm is being used to encode the file contents. Is there a tool or command that can help me find which encoding ...
1
vote
0answers
30 views

If I know the corresponding encoded bytes to some characters, how can I determine matching charsets?

I have a few files where the filenames obviously use the wrong charset. As soon as I know which charset is being used, I can use convmv to convert it to UTF-8. From the rest of the filename, I can ...
10
votes
3answers
9k views

Process a file that starts with a BOM (FF FE)

I received a .csv file with the FF FE BOM: $ head -n1 dotan.csv | hd 00000000 ff fe 41 00 64 00 20 00 67 00 72 00 6f 00 75 00 |..A.d. .g.r.o.u.| When using awk to parse it I'm getting a bunch of ...
5
votes
3answers
1k views

Is it possible to convert linux salted sha512 password hash to LDAP format?

We have an LDAP server which stores passwords and other user data. The server is not used for authentication of client machines though but only for authentication of client apps. So users change their ...
2
votes
1answer
906 views

Rename files with non-utf8 characters using detox

I am trying to use detox in the following way detox -s utf_8-only -r -v -n to detect invalid filenames on my system. The command performs mostly ok. Unfortunately it falsely detects also German ...
25
votes
4answers
21k views

identify files with non-ASCII or non-printable characters in file name

In a directory size 80GB with approximately 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there: An ...
4
votes
1answer
2k views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
8
votes
3answers
24k views

How to display Chinese characters correctly on remote Red-Hat machine?

I am using Ubuntu14.04 to connect to a remote host. Which its version is: Linux version 2.6.32-431.11.5.el6.yyyzzz.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jul 3 09:42:...
7
votes
3answers
22k views

How to convert unknown-8bit file to utf8

I have a .srt file that displays as gibberish when I open it in gEdit in ubuntu. So I want to convert it to utf8 to be able to read it. When I try to figure out what the encoding it give: file -i x....
0
votes
0answers
22 views

How can i find out which characters that are transliterated with iconv translit

Once a day my server needs to process a huge csv text file. It has to be converted from UTF8 to ISO8859-1 using iconv. Sometimes there has been special characters like ń in the file and this is ...
24
votes
5answers
77k views

Converting a UTF-8 file to ASCII (best-effort)

I have a file in UTF-8 that contains texts in multiple languages. A lot of it are people's names. I need to convert it to ASCII and I need the result to look as decent as possible. There are many ...
10
votes
1answer
787 views

Some apps doesn't accept some characters from the «Compose» key

The problem is that the compose key works fine, but some application doesn't accept some characters from it. E.g. I can type the character ∞ in Emacs (Compose+8+8), but this won't work in FireFox, ...
8
votes
2answers
2k views

Encoding of cyrillic filenames in zip files

There are a few questions here about non-ASCII letters in the names of files stored as streams inside zip files (Hebrew, Chinese, Japanese or Korean). However none of the solutions provided helped me ...
0
votes
0answers
77 views

I want to convert encoding using iconv

I want to use the iconv command (Linux) to switch to a multi-type encoding file. (Encoding type that wants to be changed) UTF-16 UTF-16BE UTF-16LE CP949 EUC-KR EUC-JP US-ASCII Big5 ISO-2022-JP ISO-...
0
votes
3answers
117 views

Creating/using my own 'custom' character

Is it possible to create your own ascii character that you can use? If so, how would it be done? Now my use case would be something like this: Let's say I created the character #12345, and I am ...
4
votes
3answers
81 views

How to find character locations of a string in a file?

I need to search for a string (a sequence of characters) in a file with a certain encoding, typically utf8, but return the character offsets (not byte offsets) of the results. So this is a search ...
49
votes
3answers
103k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same files in Jedit, it warns me of invalid UTF-8 encoding... The HTML meta tag states "charset=ISO-8859-...
12
votes
5answers
4k views

How to set fallback encoding to UTF-8 in Firefox?

I've written a Norwegian markdown document: $ file brukerveiledning.md brukerveiledning.md: UTF-8 Unicode text I've converted it to HTML using the markdown command: $ markdown > brukerveiledning....
12
votes
2answers
6k views

Several questions about file-system character encoding on linux

Due to a lot of file exchange works between Windows (GBK encoding) and Linux (UTF-8 encoding), it will encounter character encoding issues easily, such as: zip/tar files whose name contains chinese ...
2
votes
1answer
338 views

SFTP remote readdir Failure - national characters in file name

I have created a directory with national characters in it's name. I can see and browse this directory via ssh # cd /root # ls СтранноеРусскоеИмя but can't do it via SFTP: sftp> cd /root sftp>...
4
votes
3answers
25k views

Convert binary encoding that head and Notepad can read to UTF-8

I have a CSV file which is in binary character set but I have to convert to UTF-8 to process in HDFS (Hadoop). I have used the below command to check characterset. file -bi filename.csv Output : ...
1
vote
1answer
11k views

Script failing with “command not found: ^M”

When I try to run the following script in zsh, via the command /bin/zsh ~/.set_color_scheme.sh I get the following error: command not found: ^M The script has u+x permissions and it used to work on ...
5
votes
2answers
2k views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called mirror.de/�%...
19
votes
6answers
48k views

How can I correctly decompress a ZIP archive of files with Hebrew names?

Someone sent me a ZIP file containing files with Hebrew names (and created on Windows, not sure with which tool). I use LXDE on Debian Stretch. The Gnome archive manager manages to unzip the file, but ...
5
votes
2answers
3k views

What does “position” mean in iconv error message?

I get error when converting a large file: $ iconv -f GB2312 -t UTF-8 2001.txt -o 2001_u.txt iconv: illegal input sequence at position 245256667 What does the position mean in this error? I tried it ...
8
votes
2answers
32k views

Bash/Korn shell script edited on Windows throws error '…^M: not found'

I wrote a Bash script in Windows using Notepad++. cxStopAllServicesOnSERVER.sh #!/usr/bin/bash cd "/some/path" echo "Hi. Logs can be found at "`pwd`"/cxStartStopLogger.log" echo "["`date`"]*** ...
20
votes
4answers
12k views

bulk rename (or correctly display) files with special characters

I have a bunch of directories and subdirectories that contain files with special characters, like this file: robbie@phil:~$ ls test�sktest.txt test?sktest.txt Find reveals an escape sequence: ...
4
votes
4answers
3k views

What are some better options for encoding email attachments than uuencode in a bash script?

I'm referencing my original post in which I was asking a question about argument placement pertaining to $2 $3 etc, and eventually ${@:2}. It was mentioned that there are better methods to encode ...
-1
votes
1answer
36 views

How do we get the readable string from big endian hex string

How do we get the readable string from hex string 0049006e00740072006f of BE which means as normal word, ie. we dont have to swap the byte pair per word tried with xxd in vain, cause I dont know ...
15
votes
3answers
4k views

Can not use `cut -c` (`--characters`) with UTF-8?

The command cut has an option -c to work on characters, instead of bytes with the option -b. But that does not seem to work, in en_US.UTF-8 locale: The second byte gives the second ASCII character (...
0
votes
1answer
687 views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
25
votes
5answers
49k views

Why do some characters show as squares in Chrome?

For example in the dev tools I get something like: Some of these squares are at the end of lines, initially I thought they were carriage returns but it turns out they aren't. Also, squares appear ...
2
votes
1answer
328 views

Disabling utf-8 in all terminals?

This question pertains to my Ubuntu 18 system, as follows ... % uname -a Linux myhost.name.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I don't ...
1
vote
1answer
1k views

Why is LXDE displaying only boxes for font - and how to fix

A picture is worth 1000 words: This is fedora 20 running LXDE using X2GOServer, with X2GO client on windows. I am on a VPS at ramnode. I assume it's an issue with encoding, but I just can't figure ...
4
votes
2answers
1k views

Replace “/U+[0-9A-Fa-f]{4}/” with proper unicode character in shell pipeline with sed eval flag

I am trying to properly visualize the existing characters that listed in the /usr/include/X11/keysymdef.h file. It has lines like: #define XK_onethird 0x0ab0 /* U+2153 VULGAR FRACTION ONE THIRD *...
11
votes
2answers
18k views

terminal: displaying special characters

In some programs, such as htop the lines and frames are not displayed correctly. Instead, they are displayed as - and /. But on another machine, they are displayed correctly as proper lines: I am ...
0
votes
0answers
86 views

Set default file encoding to ASCII?

Years ago, most text editing applications had the option to set the default file encoding to US-ASCII. Now I can't find any recent editor on Linux that allows you to choose ASCII. If the purpose is to ...
1
vote
2answers
572 views

ls won't display Cyrillic characters properly

I've got a problem with Cyrillic alphabet. My terminal will display Cyrillic normally as well most of my programs the only exception I've found so far is ls. Instead of normal Cyrillic it displays ????...