The tag has no wiki summary.

learn more… | top users | synonyms

23
votes
2answers
19k views

How can I set VIM's default encoding to UTF-8?

I'd like to contribute to an open source project providing translated strings. One of their requirements is that contributors must use UTF-8 as the encoding for the PO files. I'm using VIM 7.3 on ...
20
votes
3answers
12k views

What charset encoding is used for filenames and paths on Linux?

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec ...
17
votes
3answers
6k views

bulk rename (or correctly display) files with special characters

I have a bunch of directories and subdirectories that contain files with special characters, like this file: robbie@phil:~$ ls test�sktest.txt test?sktest.txt Find reveals an escape sequence: ...
15
votes
2answers
10k views
+100

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
14
votes
3answers
3k views

Understanding Unix file name encoding

I have a hard time understanding how the file name encoding works. On unix.SE I find contradicting explanations. File names are stored as characters To quote another answer: Several questions about ...
11
votes
1answer
310 views

Command to retrieve the list of characters in a given character class in the current locale

What could be way to retrieve a list of all the characters in a given character class (like blank, alpha, digit...) in the current locale. For instance, LC_ALL=en_GB.UTF-8 that-command blank ...
11
votes
4answers
281 views

support for utf-8 encoding with lpr

While trying to send a text file to the printer via lpr from xterm, the content was corrupted beyond recognition, the cause of which was ultimately traced to the encoding of the file. If I instead ...
10
votes
2answers
13k views

ssh and character encoding

When I ssh into my VPS, I have irssi running in screen. When someone sends a unicode character (such as © or €), irssi displays garbage when I use it via the screen in a ssh session. If I connect to ...
9
votes
3answers
2k views

identify files with non-ASCII or non-printable characters in file name

In a directory size 80GB with approx 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there: An easy ...
9
votes
1answer
6k views

Foreign characters won't display in SSH

I'm having some problems with using ssh and a remote server (which I do not have admin access to) - Specifically, there are a couple of folders on there with Korean and Cyrillic text. When I display ...
8
votes
2answers
919 views

Why are wc -m and wc -c different?

As a C programmer, I was surprised to see that wc -c (which count the number of bytes), and wc -m (which counts the number of characters) output very different results for a long, text file of mine. I ...
8
votes
2answers
15k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding... The html meta tag states ...
8
votes
2answers
1k views

What is the ^M character called?

TexPad is creating it. I know that it is under some deadkey. I just cannot remember it is name. The blue character: I just want to mass remove them from my document. How can you type it?
8
votes
2answers
6k views

Printing decimal to ascii character, my command does not output as intended

I wanted to output a string of all the ascii characters with the following command for i in `seq 32 127`; do printf "%c" $i; done The output of the above command is: ...
8
votes
2answers
3k views

Several questions about file-system character encoding on linux

Due to a lot of file exchange works between Windows (GBK encoding) and Linux (UTF-8 encoding), it will encounter character encoding issues easily, such as: zip/tar files whose name contains chinese ...
7
votes
2answers
414 views

Does OS X have its own line format?

Obviously there are at least two newline types: DOS and Unix. But does OS X have its own plaintext 'format'? I opened a text file in nano and was surprised to see: [ Read 26793 lines (Converted from ...
7
votes
4answers
4k views

How to print all printable ASCII chars in CLI?

How can I list all the printable ASCII characters in the terminal?
7
votes
3answers
10k views

How to specify characters using hexadecimal codes in `grep`?

I am using following command to grep character set range for hexadecimal code 0900 (instead of अ) to 097F (instead of व). How I can use hexadecimal code in place of अ and व? bzcat archive.bz2 | grep ...
7
votes
2answers
4k views

Effect of $LANG on terminal

I'm trying to learn how the $LANG variable behaves with gnome-terminal (and its character encoding preference option). I've been using iso8859-1 (latin1) as my main character-set and all my filenames ...
7
votes
1answer
5k views

How to fix character encoding in bash (arch linux)

I have some issues with the encoding of german umlauts in bash. It mostly is working fine, but when i am typing a invalid command, the error message generated by bash contains strange escape sequences ...
6
votes
1answer
661 views

`^M` at the end of each line of text files generated under Windows

I was wondering why if you open a textfile made in Windows notepad under unix you will find that it has ^M where there should be a new line? My understanding is that in Windows, every line is ended ...
6
votes
3answers
672 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
6
votes
2answers
151 views

What is the connection between a gedit bug and a Unix-&-Linux Q/A href?

While answering a Unix-&-Linux question, I observed that Gedit and two other editors, Leafpad and Medit (I tested 12 editors altogether) exhibit the a certain bug. As it turns out, the bug is ...
6
votes
3answers
5k views

Why do some characters show as squares in Chrome?

For example in the dev tools I get something like: Some of these squares are at the end of lines, initially I thought they were carriage returns but it turns out they aren't. Also, squares appear ...
6
votes
2answers
4k views

special character in filename (\#033OA)

I've got a slight problem with a very stubborn error during an rsync. It's caused by a file with a special character in its filename. There's been others but I could sort that out by doing some ...
6
votes
2answers
804 views

Same file, different filename due to encoding problem?

I was about to diff a backup from it's source to manually verify that the data is correct. Some chars, like åäö, is not shown correctly on the original data but as the clients (over samba) interpret ...
6
votes
2answers
2k views

working with filenames in a different encoding over ssh

I'm ssh'ing to a remote system where a different encoding for the filenames (and for the users' locales) has been used. And this causes some problems. Problems solved by matching the locale settings ...
5
votes
3answers
4k views

Most common encoding for strings in C++ in Linux (and Unix?)

For creating a C++ program that is source code level portable between Windows and Linux and handles internationalization well, there are IMHO three main encodings to consider: The encoding of the ...
5
votes
3answers
5k views

Characters encodings supported by more, cat and less

I have a text file encoded as following according to file: ISO-8859 text, with CRLF line terminators This file contains French's text with accents. My shell is able to display accent and emacs ...
5
votes
1answer
586 views

why does my emacs whitespace mode add “” to my buffer?

I'm using Putty to ssh into a RedHat box, and I'm running GNU emacs inside the putty ssh window. When I turn on whitespace-mode using M-x whitespace mode, I notice I see a "Â" before each space or ...
5
votes
1answer
96 views

Some apps doesn't accept some characters from the «Compose» key

The problem is that the compose key works fine, but some application doesn't accept some characters from it. E.g. I can paste the character ∞ in Emacs (Compose+8+8), but this won't work in FireFox, ...
5
votes
1answer
2k views

Convert a .docx to a .pdf with pandoc

I am trying to convert a .docx received by mail to a correct pdf by using pandoc (I am using GNU/Linux). I have an error concerning characters encoding : $ pandoc file.docx -o file.pdf pandoc: ...
5
votes
1answer
63 views

Can not use `cut -c` (`--characters`) with UTF-8?

The command cut has an option -c to work on characters, instead of bytes with the option -b. But that does not seem to work, in en_US.UTF-8 locale: The second byte gives the second ASCII character ...
5
votes
1answer
2k views

terminal: displaying special characters

In some programs, such as htop the lines and frames are not displayed correctly. Instead, they are displayed as - and /. But on another machine, they are displayed correctly as proper lines: I ...
5
votes
2answers
819 views

Lowercase all but the first (Uppercase) letter from UPPERCASE in cyrillic

To make all letters a lower case except the first letter. The first letter would look like "Uppercase" after I changed (from UPPERCASE in cyrillic). The rest (not UPPERCASED) leave unchanged. I'm ...
5
votes
3answers
5k views

How to fix Russian letters in a Wine application when adjusting LANG does not help?

An application with a Russian UI shows question marks instead the most of its text (some elements are rendered ok though). The system locale is English (en_US, en_IE), I've tried Ubuntu ant XUbuntu ...
5
votes
1answer
2k views

How to use grep/ack with files in arbitrary encoding?

On my Linux desktop I have a UTF-8 locale. When I try to search some KOI8-R encoded files with grep (ack), it fails. If I manually encode the pattern into KOI8-R and pass that as an argument, it ...
5
votes
2answers
535 views

How to fix diacritics in PDF forms in Evince?

I happen to live in Czech Republic and we have such letters like ě, š, č, ť, etc in the alphabet and use them everywhere, including addresses. Today I was filling a PDF form in Evince and noticed that ...
5
votes
1answer
466 views

How to display Chinese characters correctly on remote Red-Hat machine?

I am using Ubuntu14.04 to connect to a remote host. Which its version is: Linux version 2.6.32-431.11.5.el6.yyyzzz.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jul 3 ...
5
votes
1answer
141 views

Greek characters in mutt's sidebar (mailboxes names sourced via offlineimap)

Mutt's sidebar, does not read properly Greek characters (mailboxes named by using Greek characters). Anyhow, there is no such problem inside both the index and the pager, where Greek ...
4
votes
2answers
1k views

Encoding changed after running cat

After running cat on some files I get the infamous 'bushes': ␌⎺└⎻⎼␊⎽⎽␊␍ ├▒⎼ °␋┌␊. B␊├├␊⎼ ┤⎽␊ ├▒⎼'⎽ ≥ which I believe are result of encoding being changed. Is there any way to recover the terminal ...
4
votes
1answer
2k views

How can I change locale/encoding to avoid getting weird characters in terminal?

I have installed tree on my ubuntu box, when i connect to it from Putty and I launch tree i get this : $ tree âââ html.vim -> xml.vim âââ js.vim -> xml.vim âââ xml.vim instead of this : $ ...
4
votes
4answers
4k views

How to convert hex chars to normal chars?

I tried, but I'm stuck at "escaping" the "sed's": sed -i 's/\\x0/NUL/g' $1 sed -i 's/\\x1/SOH/g' $1 sed -i 's/\\x2/STX/g' $1 sed -i 's/\\x3/ETX/g' $1 sed -i 's/\\x4/EOT/g' $1 sed -i 's/\\x5/ENQ/g' ...
4
votes
2answers
4k views

Converting text into ASCII/ISO-8859-1

I'm looking for some tool that can convert text, ideally from UTF-8 (but ISO-8859-2 and WINDOWS-1250 would be fine) into ASCII/ISO-8859-1? I have seen some online transliteration tools but I need ...
4
votes
2answers
647 views

What would break if the C locale was UTF-8 instead of ASCII?

The C locale is defined to use the ASCII charset and POSIX does not provide a way to use a charset without changing the locale as well. What would happen if the encoding of C were switched to UTF-8 ...
4
votes
3answers
3k views

Set Gnome terminal's default charset to utf8

Currently, I opened my gnome terminal and the default charset is gb2312. How can set the default charset to utf8 in Gnome terminal? I checked the preference setting but seemed not to find it. ...
4
votes
1answer
896 views

How to produce a file encoded in Mac OS Roman

Inside a bash script I have a line that creates and then inside a loop, append strings to a text file, something like echo $myTextString >> file.txt I need this final file to be formatted ...
4
votes
4answers
3k views

How to convert to HTML code?

Are there any scripts that can convert between text (e.g. <hi>) and the html entities version (&lt;hi&gt;) like this website does? Or at least a PHP file?
4
votes
2answers
205 views

Strange encoding in Slackware

I am a student and learning about operating systems, I was modifying a file using vi, when I accidentally did something (I don't remember what) that change the codification/encoding and now this is ...
4
votes
2answers
899 views

tr complains of “Illegal byte sequence”

I'm brand new to UNIX and I am using Kirk McElhearn's "The Mac OS X Command Line" to teach myself some commands. I am attempting to use tr and grep so that I can search for text strings in a regular ...