Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

learn more… | top users | synonyms

2
votes
3answers
166 views

Which terminal encodings are default on Linux, and which are most common?

I need to make a decision regarding whether a complicated commercial program that I work on should assume a particular terminal encoding for Linux, or instead read it from the terminal (and if so, ...
1
vote
2answers
113 views

Emacs shows octal escapes for some characters in a UTF-8 file

I copied some math formulas into some files under gedit, using the default settings of gedit from websites, such as math.stackexchange.com, wikipedia, ... When I open the files now under emacs, the ...
5
votes
1answer
30 views

Can I set both (in and out) charsets in `less`?

I can tell less to output characters in UTF-8: export LESSCHARSET=UTF-8 But then it tries to read files as UTF-8 as well. Can I tell it to read files as ISO-8859-2 (latin2) but display them as ...
5
votes
1answer
805 views

File name encoding wrong in KDE, but correct in GTK/tty1/

I'm on a new Linux (installed yesterday). When I access files from old partitions their file names have incorrect encoding (ä instead of ä, same for other umlauts) inside KDE apps like dolphin and ...
3
votes
1answer
91 views

Is it possible to convert linux salted sha512 password hash to LDAP format?

We have an LDAP server which stores passwords and other user data. The server is not used for authentication of client machines though but only for authentication of client apps. So users change their ...
2
votes
1answer
876 views

$Unix character set conversion

I'm confused by character-sets in Unix. I have a CSV file downloaded via SFTP: $ file -ib myfile text/plain; charset=us-ascii The purpose for this character-set quest is that the data within file ...
2
votes
1answer
135 views

Change Text File Encoding without knowning the source encoding

I want to change the charset encoding for a file in unix with a single command but since this will be an automated process it's impossible for me to know the source encoding. So I want a command that ...
2
votes
1answer
52 views

Non-standard logins

As part of a sort of esoteric "hack" I'm putting together, I've modified my /etc/passwd and /etc/shadow files to result in a user called ニエット ("Niet") with a shell pointing to a custom script I have ...
2
votes
1answer
65 views

Can't obtain the connected forms of some unicode grapheme clusters with combining diacritical marks

In Arabic and some other languages a character has two or three forms which will find its appropriate one automatically when you type a word (according to its previous and next characters). How can ...
1
vote
1answer
102 views

Can not get the file names with convmv

I have just uploaded some files on a FreeBSD server. Some files have the ??? into their names. The files came from a windows machine, so I try this: $ convmv -f cp850 -t utf-8 * Starting a dry run ...
1
vote
1answer
139 views

How do I edit text in Cyrillic / Russian on OpenBSD?

I'm trying to make it possible to easily edit text in Cyrillic / Russian on a remote OpenBSD 5.2 amd64. I'm using emacs-21.4p18-no_x11, which doesn't show any Cyrillic characters whatsoever when I ...
1
vote
1answer
247 views

Rename files with non-utf8 characters using detox

I am trying to use detox in the following way detox -s utf_8-only -r -v -n to detect invalid filenames on my system. The command performs mostly ok. Unfortunately it falsely detects also German ...
1
vote
1answer
148 views

Why doesn't this sed substitution for a non-ASCII byte work?

I tried this substitution with GNU sed on OS X (4.2.2 installed through Homebrew). But it doesn't work. printf "Hello\x92 World" | gsed -r "s/[\x92]/'/g" The expected output is: Hello' World The ...
0
votes
1answer
332 views

LMDE2 / MATE locale charset inconsistency

I have a machine with Linux Mint Debian Edition 2 "Betsy" (installed as RC, with all available updates applied) and MATE desktop environment. My problem is that in some applications special characters ...
0
votes
1answer
107 views

make `tar` archive from NTFS with non-ascii file names

Loaded a live disk into a Windows based machine with NTFS files. Being a Linux user for some years, using tar came to my mind instantly and made some backup archives from partitions and stored them ...
0
votes
1answer
86 views

jumbled character set

In my Linux env, for some unknown! reason when I process a raw unformatted file which I copied from Win7prof (was encoded in utf-16), the resulting file is all jumbled. It will print OK in Linux ...
3
votes
0answers
99 views

Get consistent encoding for all files in directory

I have a directory containing lots of csv files from various vendors with two different encodings: ASCII Text / UTF-8 UCS2 / UTF-16 little endian I'd like to use grep, awk, sed and other utilities ...
3
votes
0answers
74 views

Character encoding of .netrc

Does anyone know which character encoding is used in .netrc? As a German user, I sometimes put umlauts in passwords. So my question is, are these treated as latin1 or may I assume that I am doing ...
3
votes
0answers
129 views

How to add EBCDIC support on debian/ubuntu

I have to add EBCSIC text support to deal with IBM OS text messages. I have succesfuly added EBCDIC support to a CentOS box by adding lang and encoding to /etc/i18n/sysconfig. I searched in the docs ...
2
votes
0answers
96 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent ...
2
votes
0answers
101 views

some cyrillic file names got broken after moving from server to server

I moved my website from one server to another and encounter a problem with cyrillic filenames. Some cyrillic URLs in wordpress are fine and some are now broken. It used to be OK on previous server. ...
2
votes
0answers
178 views

iconv cannot replace Ø

It appears that iconv cannot, for example, replace the letter Ø. It was also noted in the second answer to http://stackoverflow.com/questions/3371697/replacing-accented-characters-php I have two ...
2
votes
0answers
311 views

Character encoding issue with my linux install?

I just installed Ubuntu 12.04.4 64 bits, all seemed to be ok, when I started to work on console I noticed: that I couldn't get some accented characters typed in to the console, for example, if I ...
2
votes
0answers
91 views

how to guess and rename file from deleted rar has invalid encoding

i have many files and folders that extracted from many rar from jdownloader, and deleted after extract, some of it names are: �������[�ς݂��ς݂� - �Ȃ񂾂��ꂭ������ (invalid encoding) �R�e���} (invalid ...
2
votes
0answers
82 views

Search in evolution does not find umlaut in different encoding

I use Evolution 3.6.4 on Fedora 18 x86_64, my main language is German. I got a mail (on my imapx account) encoded in iso-8859-15 with the word "hügelig" in it. The mail is displayed correctly in ...
1
vote
0answers
51 views

certain characters get converted when adding to file

I have a script I am writing, that includes passing across a connection string to socat. When I paste this into the command line it is fine, but when I open a file and paste it into there (pico ...
1
vote
0answers
56 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
1
vote
0answers
31 views

Generating 'ASCII-art' banners with arrows

I recently discovered figlet which generates ASCII-art banners. Joy! ... but, alas, I want a banner with an arrow on it. Now... $ figlet unicode → arrow _ _ //\ ...
1
vote
0answers
184 views

curl --data-urlencode and underscores

I've been using curl -XPOST to post some links to a Telegram channel via a bot api/key, the urls are in form of https://site/x/pre_encoded_string, where pre_encoded_string is in form (real samples) ...
1
vote
0answers
33 views

Understanding script and font relation (character-map)

When I use charactermap, I found that if I change font, only Latin script changes. Character by all other scripts sounds independent with change in font! So, I want to know what is relationship ...
1
vote
0answers
124 views

How do I configure medit to display UTF-8?

My environment is set to use UTF-8. I produce a file in UTF-8 encoding. It works correctly on the terminal. For example: $ echo ééé > test $ cat test $ echo é | od -t x1 0000000 c3 a9 0a ...
1
vote
0answers
571 views

Convert ASCII-8BIT to UTF-8 using OSX' .bash_profile

When running certain rails scripts in OSX' Terminal, I get warnings such as the following: unable to convert "\xA9" from ASCII-8BIT to UTF-8 for ...
1
vote
0answers
329 views

Force mysqldump encoding

I have two mysql servers, viewed from the same mysql client, right_server is rendering the french characters correctly, wrong_server is not. Encoding is the same : $ mysql -h right_server -NBe "SHOW ...
0
votes
0answers
29 views

Special characters are not recognized even with UTF-8 in KDE

So, I have Manjaro KDE pre 16 and I have some character encoding issues. Characters like ã á and such are not recognised and are represented by the ? sign. And all programs seem to use this encoding ...
0
votes
0answers
63 views

Fix broken windows cyrillic filename encoding

I have the following folder: $ ls -1 --color=never ?? ??¨??_?¥¬®­áâà æ¨®­­ë© ¬ â¥à¨ « ¨ ¢ë¡®àª¨ ?१¥­â æ¨¨ ??¨??_?á­®¢­ë¥ ¨áâ®ç­¨ª¨ ?ª§ ¬¥­ ??¨??_?§ ??¨?? ­ £®á. íª§ ¬¥­ _ТВиМС_13-14.zip As you ...
0
votes
0answers
124 views

unzipping files with non-ASCII charactes in the file names

On the Linux distribution that I use, I have been frustrated with the unzip command since every time I have unzipped files where the file names contain non-ASCII characters, the command mangles the ...
0
votes
0answers
90 views

Iceweasel content encoding errors (caused by Debian 8.3 upgrade)

I recently updated my system from Debian Mate 8.2 + Iceweasel 38.5 to Debian Mate 8.3 + Iceweasel 38.6 Almost immediatly after this, I noticed that my browsing was considerably slower but I thought it ...
0
votes
0answers
28 views

info to get using last digits of ip address

What is the product/registered owners id based on the last digits of IP#: 85ff:fec9:d41b My facebook account shows a login from that IP# and I am accessing the owner info and login info for my own ...
0
votes
0answers
20 views

Character encoding keyboard suddenly changes

Now and then I lose the Swedish encoding on my Linux platform Kubuntu / just suddenly the character set on the keyboard is changed. For instance when I press the key for the Swedish letter ä, got it ...
0
votes
0answers
32 views

Why is LXDE displaying only boxes for font - and how to fix

A picture is worth 1000 words: This is fedora 20 running LXDE using X2GOServer, with X2GO client on windows. I am on a VPS at ramnode. I assume it's an issue with encoding, but I just can't ...
0
votes
0answers
18 views

iconv adding redundant Shift-in 0x0F

we are using the below command to convert the codepage from utf8 to cp935 iconv filename -f utf8 -t Cp935 > out.filename the end file will be in Cp935 encoding for Chinese. At times this ...
0
votes
0answers
33 views

Polish–Vietnamese mojibake in a .txt file, using LibreOffice, mousepad, and vim

I have a trouble with some .txt file. When I try open it with mousepad, vim or LibreOffice Writer in every case some Polish letters are changing to another, incorrectly (i.e. correct char ś and ć are ...
0
votes
0answers
228 views

Nothing helps Gnome Terminal to not default to ANSI_x3.4-1968

I would really like gnome-terminal to default to a UTF-8 encoding, but whatever I do, I can't get it to do it. I don't have sudo access, so I cannot edit /etc/environment. I'm on a university-managed ...
0
votes
0answers
44 views

Searching text data in Hebrew, Arabic or Aramaic

How can I search for a (sub-)string in a database that has several languages, including Hebrew? I am assuming that the text being right-to-left is a question of the presentation, but the logical ...