Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

learn more… | top users | synonyms

0
votes
2answers
178 views

Find out which encoding method less uses to display the content of a text file?

There is a plain text file, which Emacs can't display correctly, but less can. I wonder if less automatically detects the encoding method for each individual text file, or just use a single default ...
3
votes
1answer
617 views

How do I view cp1251 text file in UTF-8 console?

Attempt 1: $ less subs.srt "subs.srt" may be a binary file. See it anyway? <C8><F2><E0><EB><FC><FF><ED> ... Attempt 2: $ LANG=ru_RU.CP1251 less subs.srt ...
1
vote
0answers
120 views

How do I configure medit to display UTF-8?

My environment is set to use UTF-8. I produce a file in UTF-8 encoding. It works correctly on the terminal. For example: $ echo ééé > test $ cat test $ echo é | od -t x1 0000000 c3 a9 0a ...
1
vote
1answer
373 views

iconv terminated by signal 7

I was running a little bash script on some text files: find . -name "*.dat" -exec iconv --from-code='UTF-8' --to-code='ASCII//TRANSLIT' {} --output={} \; My machine is a Ubuntu 14.04 LTS. After a ...
2
votes
0answers
161 views

iconv cannot replace Ø

It appears that iconv cannot, for example, replace the letter Ø. It was also noted in the second answer to http://stackoverflow.com/questions/3371697/replacing-accented-characters-php I have two ...
3
votes
2answers
145 views

Why can't I grep this command line option?

The output of man icc contains -Ldir Tells the linker to search for libraries... Doing man icc | grep "\-L" gives back the "-Ldir" line while man icc | grep "\-Ldir" does not. What am ...
2
votes
1answer
127 views

Change Text File Encoding without knowning the source encoding

I want to change the charset encoding for a file in unix with a single command but since this will be an automated process it's impossible for me to know the source encoding. So I want a command that ...
1
vote
1answer
115 views

How to export string about France to Microsoft excel file?

One string file Str.rc about French language is here: ID_STR_BRIGHTNESS;,"Luminosité" ID_STR_CHILE_EASTER_ISLAND;,"Île de Pâques" ID_STR_CURRENT_CH;,"Saisie chaîne" ID_STR_DETAILS;,"Détails" ...... ...
5
votes
1answer
282 views

Some apps doesn't accept some characters from the «Compose» key

The problem is that the compose key works fine, but some application doesn't accept some characters from it. E.g. I can paste the character ∞ in Emacs (Compose+8+8), but this won't work in FireFox, ...
1
vote
1answer
192 views

File weird character encoding [closed]

I'm on ubuntu 13.10 and am writing a program in eclipse that's supposed to output my result in a file. I'm building an LZW compression and decompression program that uses character conversion to its ...
0
votes
0answers
24 views

Change encoding of a text file in-place: iconv alternative? [duplicate]

Apparently, iconv is unable to use the same file as input and output file. Other tools, e.g. sed, have an -i switch that allows "updating" a file in-place. (Under the hood, temporary files are ...
5
votes
1answer
734 views

Can not use `cut -c` (`--characters`) with UTF-8?

The command cut has an option -c to work on characters, instead of bytes with the option -b. But that does not seem to work, in en_US.UTF-8 locale: The second byte gives the second ASCII character ...
3
votes
1answer
480 views

How to remove Russian and Arabic strings from a text file

Problem: I have text file with approximately 1M lines, each line consists of multiple words Some lines contain Russian or Arabic words (lets call them "bad lines"), I want to remove only these bad ...
0
votes
2answers
263 views

Can CentOS be set to pure EBCDIC?

I need to directly manipulate files from a mainframe. Text would be encoded in EBCDIC. Data can contain binary and packed decimal fields. I know I can deal with the data. I am trying to avoid the ...
2
votes
1answer
1k views

How to get rid of special characters in the output of make [closed]

When I run gcc, it outputs errors but the function names are surrounded by special characters like this ‘setup_connection’: This same is repeated with variable names. I am wondering if the ...
5
votes
2answers
16k views

Bash/Korn shell script edited on Windows throws error '…^M: not found'

Some background on the issue: I am new to scripting and wrote a simple Bash script in Windows using Notepad++. cxStopAllServicesOnSERVER.sh #!/usr/bin/bash cd "/some/path" echo "Hi. Logs can be ...
1
vote
2answers
107 views

Emacs shows octal escapes for some characters in a UTF-8 file

I copied some math formulas into some files under gedit, using the default settings of gedit from websites, such as math.stackexchange.com, wikipedia, ... When I open the files now under emacs, the ...
0
votes
1answer
33 views

Mixture of encodings

For a text which is a mixture of different encodings, is there no way to find all encodings? And convert them to a single encoding such as utf8? I copied from different sources possibly with ...
0
votes
2answers
320 views

How does rsync encode filenames?

By using strace on the active rsync process, I can see output like: read(4, "9\0\0\7\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1"..., 16384) = 61 write(3, ...
2
votes
1answer
305 views

What does “position” mean in iconv error message?

I get error when converting a large file: $ iconv -f GB2312 -t UTF-8 2001.txt -o 2001_u.txt iconv: illegal input sequence at position 245256667 What does the position mean in this error? I tried it ...
11
votes
4answers
2k views

support for utf-8 encoding with lpr

While trying to send a text file to the printer via lpr from xterm, the content was corrupted beyond recognition, the cause of which was ultimately traced to the encoding of the file. If I instead ...
1
vote
1answer
3k views

Converting documents to text with UTF-8 encoding

unoconv uses LibreOffice to convert any file that LibreOffice can convert. Now I need export to text with the UTF-8 character encoding. I use the following command: unoconv -f txt -e ...
0
votes
1answer
316 views

Change default Encoding in Terminator

The Title stays pretty much for itself. Tried to figure it out myself but I can´t find a config-setting nor a option in the Preferences for the Encoding. Does anybody know how to change the default ...
5
votes
1answer
6k views

How to display Chinese characters correctly on remote Red-Hat machine?

I am using Ubuntu14.04 to connect to a remote host. Which its version is: Linux version 2.6.32-431.11.5.el6.yyyzzz.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jul 3 ...
2
votes
1answer
283 views

replace control characters in Emacs?

This is a small text: What\222s with All the Parentheses? In Emacs \222 above is shown as a control character (in red). It is supposed to be ', but I don't understand why it is \222 instead, ...
1
vote
3answers
858 views

Debian thinks my file system is encoded as ISO-8859-1

This may not be the appropriate site for this question, because it's more a complaint than a question, but here goes. Today I updated my computer from Mint 15 to Debian 7.6. And after I'd done that, ...
5
votes
2answers
12k views

iconv illegal input sequence- why?

While trying to convert a text file into its ASCII equivalent, I get error message that iconv: illegal input sequence at position. Command I use is iconv -f UTF-8 -t ascii//TRANSLIT file The ...
2
votes
1answer
51 views

Non-standard logins

As part of a sort of esoteric "hack" I'm putting together, I've modified my /etc/passwd and /etc/shadow files to result in a user called ニエット ("Niet") with a shell pointing to a custom script I have ...
14
votes
2answers
18k views

What is the ^M character called?

TexPad is creating it. I know that it is under some deadkey. I just cannot remember it is name. The blue character: I just want to mass remove them from my document. How can you type it?
2
votes
2answers
249 views

How to fix the accents in xterm top bar?

I don't have any problem with accent (I'm French) inside xterm. However, like you can see on this screenshot, the top bar of xterm does have some accents issues: However my Debian unstable system ...
2
votes
3answers
1k views

sed, grep, or tr command that returns only Latin characters from a UTF-8 file

I am working with the text of the 300 tang poems, which unfortunately is a single file containing both Chinese and English. Since I am interested in 'extracting' the English, I hope to use sed, ...
8
votes
4answers
13k views

How to print all printable ASCII chars in CLI?

How can I list all the printable ASCII characters in the terminal?
1
vote
1answer
1k views

UTF-8 works in gnome-terminal only after selecting already selected encoding

Running Centos 6.3. In gnome-terminal UTF-8 works, but only after I've chosen (the already chosen) UTF-8 encoding in gnome-terminal settings. How do I force gnome-terminal to respect its own encoding ...
6
votes
3answers
18k views

How to change encoding from Non-ISO extended-ASCII text, with CRLF line terminators to UTF-8?

I have a txt file : $ file -i x.txt x.txt: text/plain; charset=unknown-8bit $ file x.txt x.txt: Non-ISO extended-ASCII text, with CRLF line terminators And there are some characters that are ...
2
votes
2answers
118 views

Incorrect output from cp, rm, and so on

When I run, say, cp, I get output like the following: # cp -v Foo Bar âFooâ -> âBarâ What's up with the weird â characters? Why is the shell doing this? It looks like some kind of strange ...
3
votes
0answers
73 views

Character encoding of .netrc

Does anyone know which character encoding is used in .netrc? As a German user, I sometimes put umlauts in passwords. So my question is, are these treated as latin1 or may I assume that I am doing ...
4
votes
3answers
13k views

echo bytes to a file

I'm trying to connect my rasberry Pi to some display using the i2c bus. To get started I wanted to manually write stuff, bytes in particular to a file. How do you write specific bytes to a file? I ...
2
votes
0answers
298 views

Character encoding issue with my linux install?

I just installed Ubuntu 12.04.4 64 bits, all seemed to be ok, when I started to work on console I noticed: that I couldn't get some accented characters typed in to the console, for example, if I ...
1
vote
1answer
51 views

Why does SLES 11 corrupts given characters?

If I open a notepad and copy this from it: aña, to a file opened with vi on the machine, I get this: aña My question: Why? What could the problem be? Logfiles on the server has the same ...
1
vote
1answer
56 views

Why doesn't `restore` display my UTF-8 encoded filenames correctly in interactive mode?

When using the interactive mode of the restore utility to restore backups made with dump UTF-8-encoded filenames doesn't display correctly, see example below. The ??s should be ös ... $ sudo restore ...
3
votes
2answers
831 views

MP3 tags Cyrillic chars

I have some mp3 files with Cyrillic chars on file names and also on their tags. I'm using audacious to play them. See the file info below: Is it possible to change some encoding to show the ...
2
votes
3answers
162 views

Which terminal encodings are default on Linux, and which are most common?

I need to make a decision regarding whether a complicated commercial program that I work on should assume a particular terminal encoding for Linux, or instead read it from the terminal (and if so, ...
11
votes
3answers
7k views

identify files with non-ASCII or non-printable characters in file name

In a directory size 80GB with approx 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there: An easy ...
5
votes
1answer
225 views

login with xdm and umlaut in password

I want use xdm to login to my machine and start my session. My password contains an umlaut. I'd rather not change the password as it is quite long and complex and I just memorized it. Login works ...
6
votes
3answers
1k views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
1
vote
1answer
406 views

Duplicity and GPG: charset conversion

I'm using duplicity on OS X Mavericks. If I run my duplicity command, I get the following error: ===== Begin GnuPG log ===== gpg: conversion from `US-ASCII' to `utf-8' failed: Illegal byte sequence ...
7
votes
2answers
6k views

Set Gnome terminal's default charset to utf8

Currently, I opened my gnome terminal and the default charset is gb2312. How can set the default charset to utf8 in Gnome terminal? I checked the preference setting but seemed not to find it. ...
2
votes
1answer
65 views

Can't obtain the connected forms of some unicode grapheme clusters with combining diacritical marks

In Arabic and some other languages a character has two or three forms which will find its appropriate one automatically when you type a word (according to its previous and next characters). How can ...
2
votes
1answer
101 views

Mail character set for norwegian characters

When extracting a message from my inbox containing the norwegian letter "ø" I find that it is encoded with 3 bytes as "=F8" or hexadecimal "0x3d4638". In UTF-8 the letter "ø" is encoded with two bytes ...
5
votes
2answers
889 views

How to fix diacritics in PDF forms in Evince?

I happen to live in Czech Republic and we have such letters like ě, š, č, ť, etc in the alphabet and use them everywhere, including addresses. Today I was filling a PDF form in Evince and noticed that ...