Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
7
votes
2answers
31k views

Bash/Korn shell script edited on Windows throws error '…^M: not found'

I wrote a Bash script in Windows using Notepad++. cxStopAllServicesOnSERVER.sh #!/usr/bin/bash cd "/some/path" echo "Hi. Logs can be found at "`pwd`"/cxStartStopLogger.log" echo "["`date`"]*** ...
25
votes
3answers
11k views

Understanding Unix file name encoding

I have a hard time understanding how the file name encoding works. On unix.SE I find contradicting explanations. File names are stored as characters To quote another answer: Several questions about ...
45
votes
3answers
39k views

What charset encoding is used for filenames and paths on Linux?

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec ...
2
votes
1answer
501 views

Unexpected non-null encoding of /proc/<pid>/cmdline

I am parsing the /proc/pid/cmdline value for a number of processes on my Linux system (Ubuntu 16.04) and have found that while most of the entries are null-encoded, as expected, at least one uses ...
2
votes
3answers
1k views

How to determine the character encoding that a terminal uses in a C/C++ program?

I've noticed that SyncTERM uses a different character encoding than the default MacOS terminal emulator, and they're incompatible with one another. For example, say you want to print a block ...
12
votes
2answers
11k views

Can vim display ASCII characters only, and treat other bytes as binary data?

I already know vim -b, however, depending on the locale used, it displays multi-byte characters (like UTF-8) as single letters. How can I ask vim to only display ASCII printable characters, and ...
1
vote
1answer
11k views

Script failing with “command not found: ^M”

When I try to run the following script in zsh, via the command /bin/zsh ~/.set_color_scheme.sh I get the following error: command not found: ^M The script has u+x permissions and it used to work on ...
50
votes
6answers
53k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
20
votes
4answers
12k views

bulk rename (or correctly display) files with special characters

I have a bunch of directories and subdirectories that contain files with special characters, like this file: robbie@phil:~$ ls test�sktest.txt test?sktest.txt Find reveals an escape sequence: ...
18
votes
3answers
56k views

Characters encodings supported by more, cat and less

I have a text file encoded as following according to file: ISO-8859 text, with CRLF line terminators This file contains French's text with accents. My shell is able to display accent and emacs in ...
12
votes
2answers
6k views

Several questions about file-system character encoding on linux

Due to a lot of file exchange works between Windows (GBK encoding) and Linux (UTF-8 encoding), it will encounter character encoding issues easily, such as: zip/tar files whose name contains chinese ...
6
votes
3answers
8k views

“mv” file with garbled name by inode number?

I have several files with encoding issues in their file names (German umlauts, burned on CD with Windows, read by Windows and synced to Linux with Seafile. Something, somewhere went wrong...). Bash ...
9
votes
2answers
9k views

Specify encoding with libreoffice --convert-to csv

Excel files can be converted to CSV using: $ libreoffice --convert-to csv --headless --outdir dir file.xlsx Everything appears to work just fine. The encoding, though, is set to something wonky. ...
7
votes
1answer
7k views

How to use grep/ack with files in arbitrary encoding?

On my Linux desktop I have a UTF-8 locale. When I try to search some KOI8-R encoded files with grep (ack), it fails. If I manually encode the pattern into KOI8-R and pass that as an argument, it works....
9
votes
1answer
721 views

Some apps doesn't accept some characters from the «Compose» key

The problem is that the compose key works fine, but some application doesn't accept some characters from it. E.g. I can type the character ∞ in Emacs (Compose+8+8), but this won't work in FireFox, ...
8
votes
2answers
16k views

terminal: non-english characters

I used to be able to copy&paste (or type if I load respective keyboard layout) any kind of characters (eg. é ö ñ ß و 好) in my console/terminal. Now I bought a new computer and it does not work ...
8
votes
2answers
3k views

Same file, different filename due to encoding problem?

I was about to diff a backup from it's source to manually verify that the data is correct. Some chars, like åäö, is not shown correctly on the original data but as the clients (over samba) interpret ...
5
votes
2answers
2k views

Encoding changed after running cat

After running cat on some files I get the infamous 'bushes': ␌⎺└⎻⎼␊⎽⎽␊␍ ├▒⎼ °␋┌␊. B␊├├␊⎼ ┤⎽␊ ├▒⎼'⎽ ≥ which I believe are result of encoding being changed. Is there any way to recover the terminal ...
4
votes
2answers
14k views

Curl JSON encoded in UTF-8

Important note: I'm using Cygwin I retrieve JSON from a file that I alter before sending it to a server using curl. By now everything is ok except one thing, when the server receive the JSON, every ...
3
votes
2answers
447 views

Generate collating order of a list of individual characters

In some cases, the collating order of each individual character needs to be known (to be used). It is commonly expressed in the character class of a regex like [b-d]. That character class will match ...
7
votes
2answers
7k views

working with filenames in a different encoding over ssh

I'm ssh'ing to a remote system where a different encoding for the filenames (and for the users' locales) has been used. And this causes some problems. Problems solved by matching the locale settings ...
2
votes
1answer
459 views

Fish shell shows dark-grey “⎔ characters in prompt

I'm pretty certain this is some foible of my SSH client (RoyalTS for Windows) but, having just installed and changed to fish shell, my prompt is preceded by two dark-grey ⎠characters. It doesn't ...
62
votes
2answers
61k views

How can I set VIM's default encoding to UTF-8?

I'd like to contribute to an open source project providing translated strings. One of their requirements is that contributors must use UTF-8 as the encoding for the PO files. I'm using VIM 7.3 on ...
44
votes
4answers
99k views

echo bytes to a file

I'm trying to connect my rasberry Pi to some display using the i2c bus. To get started I wanted to manually write stuff, bytes in particular to a file. How do you write specific bytes to a file? I ...
45
votes
3answers
94k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same files in Jedit, it warns me of invalid UTF-8 encoding... The HTML meta tag states "charset=ISO-8859-...
23
votes
4answers
20k views

identify files with non-ASCII or non-printable characters in file name

In a directory size 80GB with approximately 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there: An ...
21
votes
4answers
47k views

How to change encoding from Non-ISO extended-ASCII text, with CRLF line terminators to UTF-8?

I have a txt file : $ file -i x.txt x.txt: text/plain; charset=unknown-8bit $ file x.txt x.txt: Non-ISO extended-ASCII text, with CRLF line terminators And there are some characters that are ...
18
votes
2answers
1k views

Command to retrieve the list of characters in a given character class in the current locale

What could be a way to retrieve a list of all the characters in a given character class (like blank, alpha, digit...) in the current locale. For instance, LC_ALL=en_GB.UTF-8 that-command blank ...
24
votes
2answers
34k views

tr complains of “Illegal byte sequence”

I'm brand new to UNIX and I am using Kirk McElhearn's "The Mac OS X Command Line" to teach myself some commands. I am attempting to use tr and grep so that I can search for text strings in a regular ...
22
votes
7answers
34k views

How to print all printable ASCII chars in CLI?

How can I list all the printable ASCII characters in the terminal?
17
votes
6answers
44k views

How can I correctly decompress a ZIP archive of files with Hebrew names?

Someone sent me a ZIP file containing files with Hebrew names (and created on Windows, not sure with which tool). I use LXDE on Debian Stretch. The Gnome archive manager manages to unzip the file, but ...
26
votes
4answers
59k views

How to specify characters using hexadecimal codes in `grep`?

I am using following command to grep character set range for hexadecimal code 0900 (instead of अ) to 097F (instead of व). How I can use hexadecimal code in place of अ and व? bzcat archive.bz2 | grep -...
19
votes
2answers
16k views

Convert a .docx to a .pdf with pandoc

I am trying to convert a .docx received by mail to a correct pdf by using pandoc (I am using GNU/Linux). I have an error concerning characters encoding : $ pandoc file.docx -o file.pdf pandoc: ...
15
votes
3answers
4k views

Can not use `cut -c` (`--characters`) with UTF-8?

The command cut has an option -c to work on characters, instead of bytes with the option -b. But that does not seem to work, in en_US.UTF-8 locale: The second byte gives the second ASCII character (...
6
votes
1answer
937 views

Greek characters in mutt's sidebar (mailboxes names sourced via offlineimap)

Mutt's sidebar, does not read properly Greek characters (mailboxes named by using Greek characters). Anyhow, there is no such problem inside both the index and the pager, where Greek characters/words/...
5
votes
4answers
4k views

Prevent tail from modifying the charset of the terminal

In a Linux terminal (CentOS) I am using the command tail --follow=name my-rolling-file.log in order to see the logs of my application. Sometimes in the log, there is some binary data dumped (I dump ...
11
votes
2answers
17k views

terminal: displaying special characters

In some programs, such as htop the lines and frames are not displayed correctly. Instead, they are displayed as - and /. But on another machine, they are displayed correctly as proper lines: I am ...
9
votes
2answers
9k views

Process a file that starts with a BOM (FF FE)

I received a .csv file with the FF FE BOM: $ head -n1 dotan.csv | hd 00000000 ff fe 41 00 64 00 20 00 67 00 72 00 6f 00 75 00 |..A.d. .g.r.o.u.| When using awk to parse it I'm getting a bunch of ...
7
votes
3answers
21k views

How to convert unknown-8bit file to utf8

I have a .srt file that displays as gibberish when I open it in gEdit in ubuntu. So I want to convert it to utf8 to be able to read it. When I try to figure out what the encoding it give: file -i x....
5
votes
6answers
2k views

Rename folder with odd characters

I have a folder on my Mac called "␀␀␀␀HFS+ Private Data". I'm trying to delete it but it contains a bunch of odd characters that are choking unlink, rm and mv, making it difficult to remove it and ...
5
votes
2answers
2k views

What does “position” mean in iconv error message?

I get error when converting a large file: $ iconv -f GB2312 -t UTF-8 2001.txt -o 2001_u.txt iconv: illegal input sequence at position 245256667 What does the position mean in this error? I tried it ...
3
votes
2answers
2k views

MP3 tags Cyrillic chars

I have some mp3 files with Cyrillic chars on file names and also on their tags. I'm using audacious to play them. See the file info below: Is it possible to change some encoding to show the contents ...
9
votes
2answers
8k views

How can I change locale/encoding to avoid getting weird characters in terminal?

I have installed tree on my ubuntu box, when i connect to it from Putty and I launch tree i get this : $ tree âââ html.vim -> xml.vim âââ js.vim -> xml.vim âââ xml.vim instead of this : $ ...
8
votes
3answers
23k views

How to display Chinese characters correctly on remote Red-Hat machine?

I am using Ubuntu14.04 to connect to a remote host. Which its version is: Linux version 2.6.32-431.11.5.el6.yyyzzz.x86_64 (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jul 3 09:42:...
6
votes
4answers
16k views

How do I re-encode a mixed encoded text file

I've got a log file that is ASCII, except for a few UTF-8 characters (which I can fix for a future version). For the moment, I need to figure out how to get this file to a viewable/searchable/...
6
votes
1answer
4k views

Untar filenames in a character encoding different from encoding used in the filesystem

I occasionally get tarballs where the filenames are encoded in ISO-8859-1 or some other pre-Unicode scheme. My system uses UTF-8, so when I untar these archives with the usual options (tar xvf foo....
4
votes
1answer
2k views

Encoding of a zip file

When I extract some file (mp3) with japanese or korean characters within its filename on Windows, they appear normally, but when I extract it on Linux (Ubuntu/Mint) they appear as something like this ...
3
votes
2answers
534 views

What are these rectangular characters and where is this notation defined

Using Linux (Fedora, specifically) as my primary workstation, I noticed, at times, strange non-printable characters like in the following screenshots (bottom-left in the first image and upper-left in ...
5
votes
2answers
4k views

Lowercase all but the first (Uppercase) letter from UPPERCASE in cyrillic

To make all letters a lower case except the first letter. The first letter would look like "Uppercase" after I changed (from UPPERCASE in cyrillic). The rest (not UPPERCASED) leave unchanged. I'm ...
4
votes
1answer
2k views

Problems with umlauts (can't type in terminal + can't open files)

On my laptop I have Linux Mint 18.3 installed and right now I am experiencing the following problem: As a German I happen to have some files (or directories) that contain umlauts in their names (which ...