Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

learn more… | top users | synonyms

3
votes
3answers
485 views

What is the default encoding for javac in Linux?

javac says If the -encoding option is not specified, then the platform default converter is used. I guess the default encoding in Linux is UTF-8, how to get the default encoding in Linux?
8
votes
3answers
8k views

Effect of $LANG on terminal

I'm trying to learn how the $LANG variable behaves with gnome-terminal (and its character encoding preference option). I've been using iso8859-1 (latin1) as my main character-set and all my filenames ...
3
votes
2answers
3k views

Process a file that starts with a BOM (FF FE)

I received a .csv file with the FF FE BOM: $ head -n1 dotan.csv | hd 00000000 ff fe 41 00 64 00 20 00 67 00 72 00 6f 00 75 00 |..A.d. .g.r.o.u.| When using awk to parse it I'm getting a bunch of ...
2
votes
1answer
205 views

how to force shell script characters encoding from within the script

I have a few shell scripts with UTF8-encoded characters inside and i want to be sure that they are decoded correctly regardless of the machine locale settings. Is it possible to force the shell (bash ...
1
vote
1answer
42 views

How to write a luit command that outputs a file

I need help formatting a luit command so I can write a file that I'm trying to fix the encoding for. What I have write now is luit -encoding gbk cat santi.txt, but I would like this to have the output ...
6
votes
1answer
72 views

How can I make `file` output line break type and encoding for all file types?

I often find myself in the need to check the encoding and the line break type of a file. The file utility seems to be a perfect fit. Unfortunately it doesn't output the desired information for all ...
3
votes
2answers
62 views

What are these rectangular characters and where is this notation defined

Using Linux (Fedora, specifically) as my primary workstation, I noticed, at times, strange non-printable characters like in the following screenshots (bottom-left in the first image and upper-left in ...
0
votes
1answer
23 views

Keybord-key and the resulting character. Who is responsible (in the OS) [duplicate]

Pressing a key on the keybord results in different types of characters in different OS's. e.g. pressing the up arrow in windows will result in a nother sequence of characters then pressing the same ...
0
votes
0answers
23 views

Character encoding keyboard suddenly changes

Now and then I lose the Swedish encoding on my Linux platform Kubuntu / just suddenly the character set on the keyboard is changed. For instance when I press the key for the Swedish letter ä, got it ...
5
votes
1answer
449 views

Greek characters in mutt's sidebar (mailboxes names sourced via offlineimap)

Mutt's sidebar, does not read properly Greek characters (mailboxes named by using Greek characters). Anyhow, there is no such problem inside both the index and the pager, where Greek characters/words/...
0
votes
0answers
37 views

Why is LXDE displaying only boxes for font - and how to fix

A picture is worth 1000 words: This is fedora 20 running LXDE using X2GOServer, with X2GO client on windows. I am on a VPS at ramnode. I assume it's an issue with encoding, but I just can't ...
0
votes
1answer
88 views

jumbled character set

In my Linux env, for some unknown! reason when I process a raw unformatted file which I copied from Win7prof (was encoded in utf-16), the resulting file is all jumbled. It will print OK in Linux ...
2
votes
1answer
17 views

What encoding am I using? $LANG doesn't have an encoding

It seems like typically: echo $LANG results in something like this: en_US.UTF-8 What encoding is used when the result does not specify an encoding? echo $LANG en_US How do I figure out ...
0
votes
0answers
20 views

iconv adding redundant Shift-in 0x0F

we are using the below command to convert the codepage from utf8 to cp935 iconv filename -f utf8 -t Cp935 > out.filename the end file will be in Cp935 encoding for Chinese. At times this ...
2
votes
3answers
691 views

Why are file names surrounded with “â” sign?

After adding alias rm='rm -i' to my ~/.bashrc file (because, when I removed a file, it wasn't asking for confirmation), file names are surrounded with "â" signs as in the example below: rm: cannot ...
2
votes
1answer
168 views

UTF-8 not working over SSH

I have configured my terminal on OSX to show emoji properly. When I open up a terminal I can type and see emoji properly. My locale settings for OSX are shown below. When I start a tmux session it ...
3
votes
2answers
1k views

Check files for conventions

I need a script that checks and lists all text files for project conventions. With conventions I mean for example: UTF-8 encoding No trailing white spaces Newline at the end of file No not-ascii ...
1
vote
1answer
484 views

Unix command to convert character encoding in a .csv file

I need a Unix command to convert a .csv file that is in UNICODE format to ANSI format. The file is imported from the Cognos environment and I am unable to make any changes with the format in cognos ...
2
votes
1answer
69 views

Running grep for a string with non-rendering characters

So I have a log.txt files which I can search in Notepad++ on Windows, however due to the sheer number of them, I'd like to be able to run grep on them. However, the data is formatted with non-...
1
vote
1answer
258 views

Rename files with non-utf8 characters using detox

I am trying to use detox in the following way detox -s utf_8-only -r -v -n to detect invalid filenames on my system. The command performs mostly ok. Unfortunately it falsely detects also German ...
4
votes
3answers
1k views

“mv” file with garbled name by inode number?

I have several files with encoding issues in their file names (German umlauts, burned on CD with Windows, read by Windows and synced to Linux with Seafile. Something, somewhere went wrong...). Bash ...
4
votes
3answers
9k views

Converting a UTF-8 file to ASCII (best-effort)

I have a file in UTF-8 that contains texts in multiple languages. A lot of it are people's names. I need to convert it to ASCII and I need the result to look as decent as possible. There are many ...
3
votes
3answers
3k views

How to convert unknown-8bit file to utf8

I have a .srt file that displays as gibberish when I open it in gEdit in ubuntu. So I want to convert it to utf8 to be able to read it. When I try to figure out what the encoding it give: file -i x....
3
votes
1answer
201 views

How to use grep with files in arbitrary encoding, without locale installed and without sudo?

I have utf-8 locale on my computer, and i want to search in KOI8-R encoded files. But i don't have KOI8-R locale installed. And i don't have sudo rights to install this locale. Is it possible to ...
21
votes
3answers
6k views

Understanding Unix file name encoding

I have a hard time understanding how the file name encoding works. On unix.SE I find contradicting explanations. File names are stored as characters To quote another answer: Several questions about ...
6
votes
3answers
1k views

Convert an ASCII file with octal escapes for UTF-8 codes to UTF-8

I have a text file which is an ASCII file itself, but contains octal escape sequences representing codes in utf-8: \350\207\252\345\212\250\346\216 Is there some program or command that can convert ...
5
votes
1answer
677 views

Untar filenames in a character encoding different from encoding used in the filesystem

I occasionally get tarballs where the filenames are encoded in ISO-8859-1 or some other pre-Unicode scheme. My system uses UTF-8, so when I untar these archives with the usual options (tar xvf foo....
0
votes
0answers
33 views

Polish–Vietnamese mojibake in a .txt file, using LibreOffice, mousepad, and vim

I have a trouble with some .txt file. When I try open it with mousepad, vim or LibreOffice Writer in every case some Polish letters are changing to another, incorrectly (i.e. correct char ś and ć are ...
1
vote
3answers
7k views

AddDefaultCharset ISO-8859-1 not work

I have upgrade Apache from 2.2 to 2.4 and I am having problems with character encoding. In my pages I have è=è ò=ò ì=ì etc... In my httpd.conf I have commented the default value and added ISO-...
21
votes
5answers
3k views

Generating a random password; why isn't this portable?

I want to generate a random password, and am doing it like so: </dev/urandom tr -dc [:print:] | head -c 64 On my laptop, which runs Ubuntu, this produces only printable characters, as intended. ...
4
votes
4answers
660 views

Prevent tail from modifying the charset of the terminal

In a Linux terminal (CentOS) I am using the command tail --follow=name my-rolling-file.log in order to see the logs of my application. Sometimes in the log there is some binary data dumped (I dump ...
3
votes
2answers
164 views

How to search and replace dual characters by Unicode single characters in a garbled file?

I have a very long text file in french language that I need to clean up. The non ASCII characters have been replaced by combination of odd characters. As an example, the following content : passer de ...
2
votes
1answer
288 views

Encoding problem on aix system when logging using log4j2

I am using log4j2 to log message on aix system. In log when I use command less I can see: ...|LU?N ...|NOV�K ...|ZEM�NEK when I copy same file on my ubunty system and use same program I can see: ....
2
votes
2answers
949 views

view file containing DOS text (box-drawing characters, CRLF line terminators) and escape sequences

I have stumbled upon a strange text file in an archive of an old dos game downloaded from some abandonware site. The text file is named WHY_NOT.TXT. I assume the file is some sort of .NFO file from ...
3
votes
0answers
118 views

Get consistent encoding for all files in directory

I have a directory containing lots of csv files from various vendors with two different encodings: ASCII Text / UTF-8 UCS2 / UTF-16 little endian I'd like to use grep, awk, sed and other utilities ...
30
votes
3answers
21k views

What charset encoding is used for filenames and paths on Linux?

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec ...
35
votes
5answers
26k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
1
vote
1answer
336 views

Special characters in a web feed

I am running a script to check my gmail from the command line and display the first 35 characters in the title. curl -u username:password --silent "https://mail.google.com/mail/feed/atom" | \ grep -...
-1
votes
1answer
86 views

Usefulness of --from-code in command iconv

Is the --from-code option of the iconv command useful (http://man7.org/linux/man-pages/man3/iconv.3.html)? I mean, I don't see any difference if I specify this parameter (--from-code 'iso-8859-1' or ...
4
votes
1answer
139 views

Gedit displays text, but cat tells a different story

I am trying to sort a dictionary from Grady Ward's Moby and have encountered a bit of a problem. When I open it in gedit, I get: abaca×N abaciscus×N abacist×N aback×v abacli×p Abaco×N abactinal×A ...
0
votes
1answer
360 views

LMDE2 / MATE locale charset inconsistency

I have a machine with Linux Mint Debian Edition 2 "Betsy" (installed as RC, with all available updates applied) and MATE desktop environment. My problem is that in some applications special characters ...
1
vote
1answer
104 views

How to format password using SCP

I'm currently writing a Script to upload files to my server via SCP. I have the following line: sshpass -p password!% scp -P 1234 ./test.text user@mydyndns:~/test.txt But everytime I try that I ...
2
votes
0answers
38 views

Understanding script and font relation (character-map)

When I use charactermap, I found that if I change font, only Latin script changes. Character by all other scripts sounds independent with change in font! So, I want to know what is relationship ...
0
votes
0answers
47 views

Searching text data in Hebrew, Arabic or Aramaic

How can I search for a (sub-)string in a database that has several languages, including Hebrew? I am assuming that the text being right-to-left is a question of the presentation, but the logical (...
6
votes
2answers
3k views

Can vim display ASCII characters only, and treat other bytes as binary data?

I already know vim -b, however, depending on the locale used, it displays multi-byte characters (like UTF-8) as single letters. How can I ask vim to only display ASCII printable characters, and ...
1
vote
1answer
126 views

Can not get the file names with convmv

I have just uploaded some files on a FreeBSD server. Some files have the ??? into their names. The files came from a windows machine, so I try this: $ convmv -f cp850 -t utf-8 * Starting a dry run ...
2
votes
1answer
78 views

Cyrillic file names broken after upgrading Slackware

After upgrading glibc and some other libraries on Slackware instead of cyrillic names I see the garbage: | Was before | Is now | |------------+------------------| | Вождение | вожден...
4
votes
1answer
230 views

weird output when typing letter c in shell (non english input)

I had keyboard set to Georgian language input. Forgot to switch it back to English and typed letter c in shell. c = ც english c key is mapped to georgian letter ც - if this makes any difference. ...
28
votes
2answers
1k views

find(1): how is the star wildcard implemented for it to fail on some filenames?

In a file system where filenames are in UTF-8, I have a file with a faulty name; it is displayed as: D�sinstaller, actual name according to zsh: D$'\351'sinstaller, Latin1 for Désinstaller, itself a ...
2
votes
1answer
230 views

Convert TeX accents to UTF-8 in Python

edit below First of all, I am quite sure this is a duplicated question. But I have no idea how to search for it. I spent quite a long time already. I am reading with python from a .txt file the path ...