The tag has no wiki summary.

learn more… | top users | synonyms

21
votes
2answers
15k views

How can I set VIM's default encoding to UTF-8?

I'd like to contribute to an open source project providing translated strings. One of their requirements is that contributors must use UTF-8 as the encoding for the PO files. I'm using VIM 7.3 on ...
17
votes
3answers
10k views

What charset encoding is used for filenames and paths on Linux?

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec ...
16
votes
3answers
5k views

bulk rename (or correctly display) files with special characters

I have a bunch of directories and subdirectories that contain files with special characters, like this file: robbie@phil:~$ ls test�sktest.txt test?sktest.txt Find reveals an escape sequence: ...
14
votes
3answers
3k views

Understanding Unix file name encoding

I have a hard time understanding how the file name encoding works. On unix.SE I find contradicting explanations. File names are stored as characters To quote another answer: Several questions about ...
12
votes
2answers
7k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
11
votes
1answer
239 views

Command to retrieve the list of characters in a given character class in the current locale

What could be way to retrieve a list of all the characters in a given character class (like blank, alpha, digit...) in the current locale. For instance, LC_ALL=en_GB.UTF-8 that-command blank ...
9
votes
3answers
1k views

identify files with non-ASCII or non-printable characters in file name

In a directory size 80GB with approx 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there: An easy ...
8
votes
2answers
742 views

Why are wc -m and wc -c different?

As a C programmer, I was surprised to see that wc -c (which count the number of bytes), and wc -m (which counts the number of characters) output very different results for a long, text file of mine. I ...
8
votes
2answers
783 views

What is the ^M character called?

TexPad is creating it. I know that it is under some deadkey. I just cannot remember it is name. The blue character: I just want to mass remove them from my document. How can you type it?
8
votes
2answers
12k views

ssh and character encoding

When I ssh into my VPS, I have irssi running in screen. When someone sends a unicode character (such as © or €), irssi displays garbage when I use it via the screen in a ssh session. If I connect to ...
8
votes
2answers
4k views

Printing decimal to ascii character, my command does not output as intended

I wanted to output a string of all the ascii characters with the following command for i in `seq 32 127`; do printf "%c" $i; done The output of the above command is: ...
7
votes
2answers
375 views

Does OS X have its own line format?

Obviously there are at least two newline types: DOS and Unix. But does OS X have its own plaintext 'format'? I opened a text file in nano and was surprised to see: [ Read 26793 lines (Converted from ...
7
votes
4answers
2k views

How to print all printable ASCII chars in CLI?

How can I list all the printable ASCII characters in the terminal?
7
votes
2answers
11k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding... The html meta tag states ...
7
votes
3answers
8k views

How to specify characters using hexadecimal codes in `grep`?

I am using following command to grep character set range for hexadecimal code 0900 (instead of अ) to 097F (instead of व). How I can use hexadecimal code in place of अ and व? bzcat archive.bz2 | grep ...
7
votes
1answer
5k views

Foreign characters won't display in SSH

I'm having some problems with using ssh and a remote server (which I do not have admin access to) - Specifically, there are a couple of folders on there with Korean and Cyrillic text. When I display ...
7
votes
2answers
3k views

Several questions about file-system character encoding on linux

Due to a lot of file exchange works between Windows (GBK encoding) and Linux (UTF-8 encoding), it will encounter character encoding issues easily, such as: zip/tar files whose name contains chinese ...
7
votes
2answers
3k views

Effect of $LANG on terminal

I'm trying to learn how the $LANG variable behaves with gnome-terminal (and its character encoding preference option). I've been using iso8859-1 (latin1) as my main character-set and all my filenames ...
7
votes
1answer
5k views

How to fix character encoding in bash (arch linux)

I have some issues with the encoding of german umlauts in bash. It mostly is working fine, but when i am typing a invalid command, the error message generated by bash contains strange escape sequences ...
6
votes
1answer
484 views

`^M` at the end of each line of text files generated under Windows

I was wondering why if you open a textfile made in Windows notepad under unix you will find that it has ^M where there should be a new line? My understanding is that in Windows, every line is ended ...
6
votes
3answers
498 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
6
votes
2answers
149 views

What is the connection between a gedit bug and a Unix-&-Linux Q/A href?

While answering a Unix-&-Linux question, I observed that Gedit and two other editors, Leafpad and Medit (I tested 12 editors altogether) exhibit the a certain bug. As it turns out, the bug is ...
6
votes
2answers
547 views

Same file, different filename due to encoding problem?

I was about to diff a backup from it's source to manually verify that the data is correct. Some chars, like åäö, is not shown correctly on the original data but as the clients (over samba) interpret ...
6
votes
2answers
1k views

working with filenames in a different encoding over ssh

I'm ssh'ing to a remote system where a different encoding for the filenames (and for the users' locales) has been used. And this causes some problems. Problems solved by matching the locale settings ...
5
votes
3answers
3k views

Most common encoding for strings in C++ in Linux (and Unix?)

For creating a C++ program that is source code level portable between Windows and Linux and handles internationalization well, there are IMHO three main encodings to consider: The encoding of the ...
5
votes
1answer
556 views

why does my emacs whitespace mode add “” to my buffer?

I'm using Putty to ssh into a RedHat box, and I'm running GNU emacs inside the putty ssh window. When I turn on whitespace-mode using M-x whitespace mode, I notice I see a "Â" before each space or ...
5
votes
1answer
857 views

terminal: displaying special characters

In some programs, such as htop the lines and frames are not displayed correctly. Instead, they are displayed as - and /. But on another machine, they are displayed correctly as proper lines: I ...
5
votes
2answers
389 views

Lowercase all but the first (Uppercase) letter from UPPERCASE in cyrillic

To make all letters a lower case except the first letter. The first letter would look like "Uppercase" after I changed (from UPPERCASE in cyrillic). The rest (not UPPERCASED) leave unchanged. I'm ...
5
votes
3answers
4k views

How to fix Russian letters in a Wine application when adjusting LANG does not help?

An application with a Russian UI shows question marks instead the most of its text (some elements are rendered ok though). The system locale is English (en_US, en_IE), I've tried Ubuntu ant XUbuntu ...
5
votes
3answers
3k views

Why do some characters show as squares in Chrome?

For example in the dev tools I get something like: Some of these squares are at the end of lines, initially I thought they were carriage returns but it turns out they aren't. Also, squares appear ...
5
votes
1answer
2k views

How to use grep/ack with files in arbitrary encoding?

On my Linux desktop I have a UTF-8 locale. When I try to search some KOI8-R encoded files with grep (ack), it fails. If I manually encode the pattern into KOI8-R and pass that as an argument, it ...
5
votes
2answers
3k views

special character in filename (\#033OA)

I've got a slight problem with a very stubborn error during an rsync. It's caused by a file with a special character in its filename. There's been others but I could sort that out by doing some ...
5
votes
2answers
487 views

How to fix diacritics in PDF forms in Evince?

I happen to live in Czech Republic and we have such letters like ě, š, č, ť, etc in the alphabet and use them everywhere, including addresses. Today I was filling a PDF form in Evince and noticed that ...
5
votes
1answer
50 views

Greek characters in mutt's sidebar (mailboxes names sourced via offlineimap)

Mutt's sidebar, does not read properly Greek characters (mailboxes named by using Greek characters). Anyhow, there is no such problem inside both the index and the pager, where Greek ...
4
votes
2answers
913 views

Encoding changed after running cat

After running cat on some files I get the infamous 'bushes': ␌⎺└⎻⎼␊⎽⎽␊␍ ├▒⎼ °␋┌␊. B␊├├␊⎼ ┤⎽␊ ├▒⎼'⎽ ≥ which I believe are result of encoding being changed. Is there any way to recover the terminal ...
4
votes
4answers
3k views

How to convert hex chars to normal chars?

I tried, but I'm stuck at "escaping" the "sed's": sed -i 's/\\x0/NUL/g' $1 sed -i 's/\\x1/SOH/g' $1 sed -i 's/\\x2/STX/g' $1 sed -i 's/\\x3/ETX/g' $1 sed -i 's/\\x4/EOT/g' $1 sed -i 's/\\x5/ENQ/g' ...
4
votes
2answers
442 views

What would break if the C locale was UTF-8 instead of ASCII?

The C locale is defined to use the ASCII charset and POSIX does not provide a way to use a charset without changing the locale as well. What would happen if the encoding of C were switched to UTF-8 ...
4
votes
1answer
818 views

How to produce a file encoded in Mac OS Roman

Inside a bash script I have a line that creates and then inside a loop, append strings to a text file, something like echo $myTextString >> file.txt I need this final file to be formatted ...
4
votes
2answers
197 views

Strange encoding in Slackware

I am a student and learning about operating systems, I was modifying a file using vi, when I accidentally did something (I don't remember what) that change the codification/encoding and now this is ...
4
votes
1answer
867 views

Convert a .docx to a .pdf with pandoc

I am trying to convert a .docx received by mail to a correct pdf by using pandoc (I am using GNU/Linux). I have an error concerning characters encoding : $ pandoc file.docx -o file.pdf pandoc: ...
4
votes
3answers
3k views

Set Gnome terminal's default charset to utf8

Currently, I opened my gnome terminal and the default charset is gb2312. How can set the default charset to utf8 in Gnome terminal? I checked the preference setting but seemed not to find it. ...
4
votes
3answers
1k views

russian letters in gentoo console as '?'

I managed to get russian language working in my Gnome apps, but neither in the console (Alt-Shift-F2 and such) nor in the Gnome Terminal valyagentoo va1en0k # locale LANG=ru_RU.UTF-8 ...
4
votes
1answer
639 views

opendir and readdir encoding strings behind my back?

(You can skip the details to the last couple of lines if you're able to answer the question :) ) I'm on an Ubuntu 12.04. I'm trying to resolve an old issue that I've posted about in the past (if ...
4
votes
0answers
82 views

weird output when typing letter c in shell (non english input)

I had keyboard set to Georgian language input. Forgot to switch it back to English and typed letter c in shell. c = ც english c key is mapped to georgian letter ც - if this makes any difference. ...
4
votes
1answer
74 views

login with xdm and umlaut in password

I want use xdm to login to my machine and start my session. My password contains an umlaut. I'd rather not change the password as it is quite long and complex and I just memorized it. Login works ...
4
votes
1answer
514 views

File name encoding wrong in KDE, but correct in GTK/tty1/

I'm on a new Linux (installed yesterday). When I access files from old partitions their file names have incorrect encoding (ä instead of ä, same for other umlauts) inside KDE apps like dolphin and ...
3
votes
1answer
1k views

How can I change locale/encoding to avoid getting weird characters in terminal?

I have installed tree on my ubuntu box, when i connect to it from Putty and I launch tree i get this : $ tree âââ html.vim -> xml.vim âââ js.vim -> xml.vim âââ xml.vim instead of this : $ ...
3
votes
1answer
176 views

Process a file that starts with a BOM (FF FE)

I received a .csv file with the FF FE BOM: $ head -n1 dotan.csv | hd 00000000 ff fe 41 00 64 00 20 00 67 00 72 00 6f 00 75 00 |..A.d. .g.r.o.u.| When using awk to parse it I'm getting a bunch of ...
3
votes
2answers
184 views

cp: what quotation marks are used in error messages?

I'm currently fiddling around with a dying harddisk, and while trying to cp data from it, I get errors like cp: error reading ‘brokenFile’: Input/output error # comparison: backtick: `, apostrophe ' ...
3
votes
2answers
146 views

MP3 tags Cyrillic chars

I have some mp3 files with Cyrillic chars on file names and also on their tags. I'm using audacious to play them. See the file info below: Is it possible to change some encoding to show the ...