Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

learn more… | top users | synonyms

3
votes
2answers
27 views

Problem with reading text file encoded in Western encoding (ISO-8859-1)

I'm having a problem with encoding of ISO-8859-1 text file (subtitles in Polish language), which looks something like that: Mieszka³ sam,|¿adnej ¿ony, dzieci. It should be : "Mieszkał sam, żadnej ...
3
votes
2answers
983 views

Unix character set conversion

I'm confused by character-sets in Unix. I have a CSV file downloaded via SFTP: $ file -ib myfile text/plain; charset=us-ascii The purpose for this character-set quest is that the data within file ...
1
vote
1answer
32 views

store file with invalid characters

Some files we get from a customer could not be processed properly because they were declared as US-ASCII but contained invalid characters. In order to validate a software fix, I am trying to copy ...
0
votes
0answers
12 views

View mac shared folder from linux (mint/ubuntu)

When i open shared folder hosted on mac (local network). All folder name's display like this: –ü–∞–ø–∫–∞ ¬´–û–±—â–∏–µ¬ª Mac has internal encoding (MAC) Linux has UTF-8 ╰─$ locale LANG=en_US....
0
votes
2answers
368 views

How does rsync encode filenames?

By using strace on the active rsync process, I can see output like: read(4, "9\0\0\7\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1\10\0\1"..., 16384) = 61 write(3, "\357\265mD0e\233:\337\...
3
votes
1answer
130 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
0
votes
1answer
35 views

Unexpected appearance of â in man

In few of the man pages I've often seen below typical appearance of the character â in stead of an apostrophe, say in man who, -T, -w, --mesg add userâs message status as +, - or ? Why would that ...
6
votes
4answers
6k views

How to convert to HTML code?

Are there any scripts that can convert between text (e.g. <hi>) and the html entities version (&lt;hi&gt;) like this website does? Or at least a PHP file?
10
votes
2answers
12k views

Printing decimal to ascii character, my command does not output as intended

I wanted to output a string of all the ascii characters with the following command for i in `seq 32 127`; do printf "%c" $i; done The output of the above command is: ...
5
votes
2answers
817 views

File name encoding wrong in KDE, but correct in GTK/tty1/

I'm on a new Linux (installed yesterday). When I access files from old partitions their file names have incorrect encoding (ä instead of ä, same for other umlauts) inside KDE apps like dolphin and ...
1
vote
1answer
56 views

How to convert GBK to UTF-8 in a mixed encoding directory?

Background : You can skip this section, if you are not interested in. I usually backup my MicroSD card of my cell phone, with command sudo dd if=/dev/sdc1 of=~/Document/Cell\ Phone\ Files/...
6
votes
4answers
17k views

echo bytes to a file

I'm trying to connect my rasberry Pi to some display using the i2c bus. To get started I wanted to manually write stuff, bytes in particular to a file. How do you write specific bytes to a file? I ...
11
votes
2answers
438 views

Which character encodings are supported by posix?

POSIX defines the behavior of tools such as grep, awk, sed, etc which work against text files. Since it is a text file, I think there is the problem(s) of character encoding. Question: What is the ...
2
votes
1answer
57 views

Fish shell shows dark-grey “⎔ characters in prompt

I'm pretty certain this is some foible of my SSH client (RoyalTS for Windows) but, having just installed and changed to fish shell, my prompt is preceded by two dark-grey ⎠characters. It doesn't ...
18
votes
4answers
10k views

bulk rename (or correctly display) files with special characters

I have a bunch of directories and subdirectories that contain files with special characters, like this file: robbie@phil:~$ ls test�sktest.txt test?sktest.txt Find reveals an escape sequence: ...
1
vote
1answer
22 views

Understanding what is happening when I dump a terminal character sequence with Ctrl-v?

If I want to bind a key-mapping to a function or widget in zsh I have learnt that I first have to hit Ctrl+v - at a prompt, then enter the key sequence I want to use, then use the output in my key-...
7
votes
2answers
4k views

working with filenames in a different encoding over ssh

I'm ssh'ing to a remote system where a different encoding for the filenames (and for the users' locales) has been used. And this causes some problems. Problems solved by matching the locale settings ...
2
votes
2answers
405 views

UTF-8 characters are not displayed correctly in Debian

Short description of my problem: I ran into an issue lately where I am unable to make bash/nano/irssi/etc display "special" UTF-8 characters like the german umlauts (äüö), the euro sign (€) and some ...
3
votes
2answers
311 views

cp: what quotation marks are used in error messages?

I'm currently fiddling around with a dying harddisk, and while trying to cp data from it, I get errors like cp: error reading ‘brokenFile’: Input/output error # comparison: backtick: `, apostrophe ' ...
7
votes
2answers
4k views

terminal: displaying special characters

In some programs, such as htop the lines and frames are not displayed correctly. Instead, they are displayed as - and /. But on another machine, they are displayed correctly as proper lines: I ...
12
votes
4answers
17k views

Why do some characters show as squares in Chrome?

For example in the dev tools I get something like: Some of these squares are at the end of lines, initially I thought they were carriage returns but it turns out they aren't. Also, squares appear ...
1
vote
1answer
157 views

Why doesn't this sed substitution for a non-ASCII byte work?

I tried this substitution with GNU sed on OS X (4.2.2 installed through Homebrew). But it doesn't work. printf "Hello\x92 World" | gsed -r "s/[\x92]/'/g" The expected output is: Hello' World The ...
0
votes
0answers
42 views

Special characters are not recognized even with UTF-8 in KDE

So, I have Manjaro KDE pre 16 and I have some character encoding issues. Characters like ã á and such are not recognised and are represented by the ? sign. And all programs seem to use this encoding ...
0
votes
0answers
144 views

Fix broken windows cyrillic filename encoding

I have the following folder: $ ls -1 --color=never ?? ??¨??_?¥¬®­áâà æ¨®­­ë© ¬ â¥à¨ « ¨ ¢ë¡®àª¨ ?१¥­â æ¨¨ ??¨??_?á­®¢­ë¥ ¨áâ®ç­¨ª¨ ?ª§ ¬¥­ ??¨??_?§ ??¨?? ­ £®á. íª§ ¬¥­ _ТВиМС_13-14.zip As you ...
1
vote
0answers
54 views

certain characters get converted when adding to file

I have a script I am writing, that includes passing across a connection string to socat. When I paste this into the command line it is fine, but when I open a file and paste it into there (pico ...
5
votes
5answers
467 views

Rename folder with odd characters

I have a folder on my Mac called "␀␀␀␀HFS+ Private Data". I'm trying to delete it but it contains a bunch of odd characters that are choking unlink, rm and mv, making it difficult to remove it and ...
0
votes
1answer
101 views

How can I get rid of the byte-order mark 

I have created an html webpage when I run it at the bottom of the page there's some unwanted characters uncoding . I tried to get rid of it through vi using set nobomb but it still appears in the ...
1
vote
1answer
254 views

Octals 302 240 together seem to correspond to non-breaking space

By looking at a particular line of a text file (say, the 1123th, see below), it seems that there is a non-breaking space, but I am not sure: $ cat myfile.csv | sed -n 1123p | cut -f2 Lisztes feher $ ...
4
votes
1answer
146 views

Is it possible to convert linux salted sha512 password hash to LDAP format?

We have an LDAP server which stores passwords and other user data. The server is not used for authentication of client machines though but only for authentication of client apps. So users change their ...
3
votes
3answers
2k views

How to convert html entities to readable text?

I want html number entities like &#x119; and want to convert it to real character. I have emails mostly from linkedin that look like this: chcia&#x142;abym zapyta&#x107;, czy rozwa&...
2
votes
1answer
97 views

Why is this find command not returning filenames containing non-ASCII characters only?

I'm trying to determine the root cause of why this find command is not working; it shouldn't match the file called this_should_not_match below: $ > find . -type f -name "*[^ -~]*" ./__º╚t ./...
3
votes
3answers
380 views

does head input > output copy all invisible characters to the new file?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
5
votes
1answer
30 views

Can I set both (in and out) charsets in `less`?

I can tell less to output characters in UTF-8: export LESSCHARSET=UTF-8 But then it tries to read files as UTF-8 as well. Can I tell it to read files as ISO-8859-2 (latin2) but display them as UTF-...
1
vote
0answers
65 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
2
votes
0answers
35 views

Generating 'ASCII-art' banners with arrows

I recently discovered figlet which generates ASCII-art banners. Joy! ... but, alas, I want a banner with an arrow on it. Now... $ figlet unicode → arrow _ _ //\ ...
0
votes
0answers
171 views

unzipping files with non-ASCII charactes in the file names

On the Linux distribution that I use, I have been frustrated with the unzip command since every time I have unzipped files where the file names contain non-ASCII characters, the command mangles the ...
1
vote
1answer
35 views

USB LABEL in different language

If the USB LABEL (name) is in different langauage (Hindi or Chinese) how do i get to know which language it is coded to pass to the iconv or is there any way we can know the language which has been ...
1
vote
3answers
1k views

Check whether text contains non-ASCII characters in a shell script

I have a file, I want to determine if it contains only English ASCII characters. I thought this is a very common question, but when I googled it, there is no direct answer or related. According to ...
4
votes
5answers
5k views

How do I rename files with strange characters

I have file with greek or cyrillic characters. It is not owned by me, but by the web server user (www). I cannot use the shell as the web server user (www) or as root, but I've used a script (...
2
votes
0answers
143 views

some cyrillic file names got broken after moving from server to server

I moved my website from one server to another and encounter a problem with cyrillic filenames. Some cyrillic URLs in wordpress are fine and some are now broken. It used to be OK on previous server. ...
5
votes
1answer
113 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called mirror.de/�%...
2
votes
2answers
1k views

What encoding does my Konsole support?

How can I check what encoding (ASCII, UTF-8, UTF-16 etc) my Terminal uses? It shouldn't matter, but I am using Konsole on KDE3.5.
4
votes
2answers
502 views

Specify encoding with libreoffice --convert-to csv

Excel files can be converted to CSV using: $ libreoffice --convert-to csv --headless --outdir dir file.xlsx Everything appears to work just fine. The encoding, though, is set to something wonky. ...
0
votes
0answers
96 views

Iceweasel content encoding errors (caused by Debian 8.3 upgrade)

I recently updated my system from Debian Mate 8.2 + Iceweasel 38.5 to Debian Mate 8.3 + Iceweasel 38.6 Almost immediatly after this, I noticed that my browsing was considerably slower but I thought it ...
1
vote
1answer
86 views

Encoding of /proc/<pid>/cmdline files

I am thinking about working with the /proc/<pid>/cmdline files, but I couldn't find any documentation about the file encoding. The only piece of information I could find is located in the man ...
0
votes
0answers
31 views

info to get using last digits of ip address

What is the product/registered owners id based on the last digits of IP#: 85ff:fec9:d41b My facebook account shows a login from that IP# and I am accessing the owner info and login info for my own ...
13
votes
3answers
647 views

How to convert an emoticon specified by a U+xxxxx code to utf-8?

Emoticons seem to be specified using a format of U+xxxxx wherein each x is a hexadecimal digit. For example, U+1F615 is the official Unicode Consortium code for the "confused face" 😕 As I am ...
0
votes
1answer
301 views

iconv can't convert circled numbers to/from Japanese encodings

The unicode range of circled digits (U+2460 .. U+2468) cannot be converted to, or from, any of the Japanese encodings (EUC-JP, Shift-JIS, ISO-2022-JP), even though they exist there, and I run across ...
2
votes
1answer
1k views

How can I correctly decompress a ZIP archive of files with Hebrew names?

(Question self-migrated from superuser.com) Someone sent me a ZIP file containing files with Hebrew names (and created on Windows, not sure with which tool). I use LXDE on Debian Stretch. The Gnome ...
1
vote
0answers
221 views

curl --data-urlencode and underscores

I've been using curl -XPOST to post some links to a Telegram channel via a bot api/key, the urls are in form of https://site/x/pre_encoded_string, where pre_encoded_string is in form (real samples) ...