Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

learn more… | top users | synonyms

-1
votes
1answer
16 views

Possible encode error on open window titlte [on hold]

I'm having a problem after updating, upgrading and rebooting my system. When i open a program and its window shows up, the title only shows some little boxes. For example, if i open a terminal, there ...
1
vote
1answer
17 views

Understanding what is happening when I dump a terminal character sequence with Ctrl-v?

If I want to bind a key-mapping to a function or widget in zsh I have learnt that I first have to hit Ctrl+v - at a prompt, then enter the key sequence I want to use, then use the output in my ...
2
votes
2answers
125 views

UTF-8 characters are not displayed correctly in Debian

Short description of my problem: I ran into an issue lately where I am unable to make bash/nano/irssi/etc display "special" UTF-8 characters like the german umlauts (äüö), the euro sign (€) and some ...
0
votes
0answers
29 views

Special characters are not recognized even with UTF-8 in KDE

So, I have Manjaro KDE pre 16 and I have some character encoding issues. Characters like ã á and such are not recognised and are represented by the ? sign. And all programs seem to use this encoding ...
0
votes
0answers
63 views

Fix broken windows cyrillic filename encoding

I have the following folder: $ ls -1 --color=never ?? ??¨??_?¥¬®­áâà æ¨®­­ë© ¬ â¥à¨ « ¨ ¢ë¡®àª¨ ?१¥­â æ¨¨ ??¨??_?á­®¢­ë¥ ¨áâ®ç­¨ª¨ ?ª§ ¬¥­ ??¨??_?§ ??¨?? ­ £®á. íª§ ¬¥­ _ТВиМС_13-14.zip As you ...
1
vote
0answers
51 views

certain characters get converted when adding to file

I have a script I am writing, that includes passing across a connection string to socat. When I paste this into the command line it is fine, but when I open a file and paste it into there (pico ...
0
votes
1answer
43 views

How can I get rid of the byte-order mark 

I have created an html webpage when I run it at the bottom of the page there's some unwanted characters uncoding . I tried to get rid of it through vi using set nobomb but it still appears in the ...
1
vote
1answer
200 views

Octals 302 240 together seem to correspond to non-breaking space

By looking at a particular line of a text file (say, the 1123th, see below), it seems that there is a non-breaking space, but I am not sure: $ cat myfile.csv | sed -n 1123p | cut -f2 Lisztes feher $ ...
2
votes
1answer
92 views

Why is this find command not returning filenames containing non-ASCII characters only?

I'm trying to determine the root cause of why this find command is not working; it shouldn't match the file called this_should_not_match below: $ > find . -type f -name "*[^ -~]*" ./__º╚t ...
3
votes
1answer
91 views

Is it possible to convert linux salted sha512 password hash to LDAP format?

We have an LDAP server which stores passwords and other user data. The server is not used for authentication of client machines though but only for authentication of client apps. So users change their ...
1
vote
0answers
56 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
3
votes
3answers
371 views

does head input > output copy all invisible characters to the new file?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
1
vote
0answers
31 views

Generating 'ASCII-art' banners with arrows

I recently discovered figlet which generates ASCII-art banners. Joy! ... but, alas, I want a banner with an arrow on it. Now... $ figlet unicode → arrow _ _ //\ ...
0
votes
0answers
124 views

unzipping files with non-ASCII charactes in the file names

On the Linux distribution that I use, I have been frustrated with the unzip command since every time I have unzipped files where the file names contain non-ASCII characters, the command mangles the ...
1
vote
1answer
33 views

USB LABEL in different language

If the USB LABEL (name) is in different langauage (Hindi or Chinese) how do i get to know which language it is coded to pass to the iconv or is there any way we can know the language which has been ...
5
votes
1answer
30 views

Can I set both (in and out) charsets in `less`?

I can tell less to output characters in UTF-8: export LESSCHARSET=UTF-8 But then it tries to read files as UTF-8 as well. Can I tell it to read files as ISO-8859-2 (latin2) but display them as ...
5
votes
1answer
97 views

How to fix file name encoding

I scrapped a site with wget. That site is in German and some of that pages had Ü,ü,Ö,ö,Ä,ä,ß in the URL. Now some files have a very weird name. For example one file is called ...
4
votes
2answers
327 views

Specify encoding with libreoffice --convert-to csv

Excel files can be converted to CSV using: $ libreoffice --convert-to csv --headless --outdir dir file.xlsx Everything appears to work just fine. The encoding, though, is set to something wonky. ...
0
votes
0answers
90 views

Iceweasel content encoding errors (caused by Debian 8.3 upgrade)

I recently updated my system from Debian Mate 8.2 + Iceweasel 38.5 to Debian Mate 8.3 + Iceweasel 38.6 Almost immediatly after this, I noticed that my browsing was considerably slower but I thought it ...
1
vote
1answer
54 views

Encoding of /proc/<pid>/cmdline files

I am thinking about working with the /proc/<pid>/cmdline files, but I couldn't find any documentation about the file encoding. The only piece of information I could find is located in the man ...
0
votes
0answers
28 views

info to get using last digits of ip address

What is the product/registered owners id based on the last digits of IP#: 85ff:fec9:d41b My facebook account shows a login from that IP# and I am accessing the owner info and login info for my own ...
5
votes
5answers
330 views

Rename folder with odd characters

I have a folder on my Mac called "␀␀␀␀HFS+ Private Data". I'm trying to delete it but it contains a bunch of odd characters that are choking unlink, rm and mv, making it difficult to remove it and ...
13
votes
3answers
615 views

How to convert an emoticon specified by a U+xxxxx code to utf-8?

Emoticons seem to be specified using a format of U+xxxxx wherein each x is a hexadecimal digit. For example, U+1F615 is the official Unicode Consortium code for the "confused face" 😕 As I am ...
2
votes
1answer
679 views

How can I correctly decompress a ZIP archive of files with Hebrew names?

(Question self-migrated from superuser.com) Someone sent me a ZIP file containing files with Hebrew names (and created on Windows, not sure with which tool). I use LXDE on Debian Stretch. The Gnome ...
1
vote
0answers
184 views

curl --data-urlencode and underscores

I've been using curl -XPOST to post some links to a Telegram channel via a bot api/key, the urls are in form of https://site/x/pre_encoded_string, where pre_encoded_string is in form (real samples) ...
2
votes
0answers
96 views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent ...
3
votes
3answers
274 views

What is the default encoding for javac in Linux?

javac says If the -encoding option is not specified, then the platform default converter is used. I guess the default encoding in Linux is UTF-8, how to get the default encoding in Linux?
2
votes
1answer
121 views

how to force shell script characters encoding from within the script

I have a few shell scripts with UTF8-encoded characters inside and i want to be sure that they are decoded correctly regardless of the machine locale settings. Is it possible to force the shell (bash ...
1
vote
1answer
36 views

How to write a luit command that outputs a file

I need help formatting a luit command so I can write a file that I'm trying to fix the encoding for. What I have write now is luit -encoding gbk cat santi.txt, but I would like this to have the output ...
6
votes
1answer
59 views

How can I make `file` output line break type and encoding for all file types?

I often find myself in the need to check the encoding and the line break type of a file. The file utility seems to be a perfect fit. Unfortunately it doesn't output the desired information for all ...
3
votes
2answers
57 views

What are these rectangular characters and where is this notation defined

Using Linux (Fedora, specifically) as my primary workstation, I noticed, at times, strange non-printable characters like in the following screenshots (bottom-left in the first image and upper-left in ...
0
votes
1answer
20 views

Keybord-key and the resulting character. Who is responsible (in the OS) [duplicate]

Pressing a key on the keybord results in different types of characters in different OS's. e.g. pressing the up arrow in windows will result in a nother sequence of characters then pressing the same ...
0
votes
0answers
20 views

Character encoding keyboard suddenly changes

Now and then I lose the Swedish encoding on my Linux platform Kubuntu / just suddenly the character set on the keyboard is changed. For instance when I press the key for the Swedish letter ä, got it ...
0
votes
0answers
32 views

Why is LXDE displaying only boxes for font - and how to fix

A picture is worth 1000 words: This is fedora 20 running LXDE using X2GOServer, with X2GO client on windows. I am on a VPS at ramnode. I assume it's an issue with encoding, but I just can't ...
2
votes
1answer
14 views

What encoding am I using? $LANG doesn't have an encoding

It seems like typically: echo $LANG results in something like this: en_US.UTF-8 What encoding is used when the result does not specify an encoding? echo $LANG en_US How do I figure out ...
2
votes
0answers
101 views

some cyrillic file names got broken after moving from server to server

I moved my website from one server to another and encounter a problem with cyrillic filenames. Some cyrillic URLs in wordpress are fine and some are now broken. It used to be OK on previous server. ...
0
votes
0answers
18 views

iconv adding redundant Shift-in 0x0F

we are using the below command to convert the codepage from utf8 to cp935 iconv filename -f utf8 -t Cp935 > out.filename the end file will be in Cp935 encoding for Chinese. At times this ...
2
votes
1answer
147 views

UTF-8 not working over SSH

I have configured my terminal on OSX to show emoji properly. When I open up a terminal I can type and see emoji properly. My locale settings for OSX are shown below. When I start a tmux session it ...
1
vote
1answer
347 views

Unix command to convert character encoding in a .csv file

I need a Unix command to convert a .csv file that is in UNICODE format to ANSI format. The file is imported from the Cognos environment and I am unable to make any changes with the format in cognos ...
2
votes
1answer
60 views

Running grep for a string with non-rendering characters

So I have a log.txt files which I can search in Notepad++ on Windows, however due to the sheer number of them, I'd like to be able to run grep on them. However, the data is formatted with ...
4
votes
3answers
1k views

“mv” file with garbled name by inode number?

I have several files with encoding issues in their file names (German umlauts, burned on CD with Windows, read by Windows and synced to Linux with Seafile. Something, somewhere went wrong...). Bash ...
3
votes
1answer
155 views

How to use grep with files in arbitrary encoding, without locale installed and without sudo?

I have utf-8 locale on my computer, and i want to search in KOI8-R encoded files. But i don't have KOI8-R locale installed. And i don't have sudo rights to install this locale. Is it possible to ...
0
votes
0answers
33 views

Polish–Vietnamese mojibake in a .txt file, using LibreOffice, mousepad, and vim

I have a trouble with some .txt file. When I try open it with mousepad, vim or LibreOffice Writer in every case some Polish letters are changing to another, incorrectly (i.e. correct char ś and ć are ...
5
votes
1answer
543 views

Untar filenames in a character encoding different from encoding used in the filesystem

I occasionally get tarballs where the filenames are encoded in ISO-8859-1 or some other pre-Unicode scheme. My system uses UTF-8, so when I untar these archives with the usual options (tar xvf ...
2
votes
1answer
214 views

Encoding problem on aix system when logging using log4j2

I am using log4j2 to log message on aix system. In log when I use command less I can see: ...|LU?N ...|NOV�K ...|ZEM�NEK when I copy same file on my ubunty system and use same program I can see: ...
3
votes
0answers
99 views

Get consistent encoding for all files in directory

I have a directory containing lots of csv files from various vendors with two different encodings: ASCII Text / UTF-8 UCS2 / UTF-16 little endian I'd like to use grep, awk, sed and other utilities ...
6
votes
3answers
1k views

Convert an ASCII file with octal escapes for UTF-8 codes to UTF-8

I have a text file which is an ASCII file itself, but contains octal escape sequences representing codes in utf-8: \350\207\252\345\212\250\346\216 Is there some program or command that can ...
0
votes
0answers
228 views

Nothing helps Gnome Terminal to not default to ANSI_x3.4-1968

I would really like gnome-terminal to default to a UTF-8 encoding, but whatever I do, I can't get it to do it. I don't have sudo access, so I cannot edit /etc/environment. I'm on a university-managed ...
2
votes
3answers
527 views

Why are file names surrounded with “â” sign?

After adding alias rm='rm -i' to my ~/.bashrc file (because, when I removed a file, it wasn't asking for confirmation), file names are surrounded with "â" signs as in the example below: rm: cannot ...
-1
votes
1answer
80 views

Usefulness of --from-code in command iconv

Is the --from-code option of the iconv command useful (http://man7.org/linux/man-pages/man3/iconv.3.html)? I mean, I don't see any difference if I specify this parameter (--from-code 'iso-8859-1' or ...