Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

49 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
3answers
1k views

Is it possible to convert linux salted sha512 password hash to LDAP format?

We have an LDAP server which stores passwords and other user data. The server is not used for authentication of client machines though but only for authentication of client apps. So users change their ...
4
votes
1answer
2k views

How to tell wget to download files with url encoded names?

I'm trying to download an entire website using wget and this is the command I use: wget --recursive --no-clobber --page-requisites --convert-links --domains example.com --no-parent http://www....
3
votes
0answers
85 views

Mounting old cyrillic HFS partition in modern Linux

I'm looking for an easiest way to transfer files to my old Macintosh LCII (to avoid messing with floppies or CD burning). I figured that I can probably get its old SCSI harddrive to mount on my modern ...
3
votes
0answers
701 views

some cyrillic file names got broken after moving from server to server

I moved my website from one server to another and encounter a problem with cyrillic filenames. Some cyrillic URLs in wordpress are fine and some are now broken. It used to be OK on previous server. ...
3
votes
1answer
558 views

Change Text File Encoding without knowning the source encoding

I want to change the charset encoding for a file in unix with a single command but since this will be an automated process it's impossible for me to know the source encoding. So I want a command that ...
3
votes
1answer
444 views

How to add EBCDIC support on debian/ubuntu

I have to add EBCSIC text support to deal with IBM OS text messages. I have succesfuly added EBCDIC support to a CentOS box by adding lang and encoding to /etc/i18n/sysconfig. I searched in the docs ...
2
votes
1answer
138 views

SFTP remote readdir Failure - national characters in file name

I have created a directory with national characters in it's name. I can see and browse this directory via ssh # cd /root # ls СтранноеРусскоеИмя but can't do it via SFTP: sftp> cd /root sftp>...
2
votes
1answer
263 views

Can't get iso-8859-1 to work for 'umlauts' (Debian / Plesk)

I think I have configured ISO-8859-1 at all necessary locations, but still, my pages are displaying black '?' instead of German umlauts! My server runs Plesk 17.8.11 / Debian 8 / Apache 2 / PHP 7.1 (...
2
votes
1answer
93 views

Encoding problems with German man pages

On the machine I am working on I have English man pages under /usr/share/man/and German ones under /usr/share/man/de. The former are encoded with us-ascii, the latter ones with utf-8. Unfortunately ...
2
votes
1answer
180 views

Why is sed failing to follow collate order?

On this command: $ echo "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" | > sed 's/[a-z]//g' ABCDEFGHIJKLMNOPQRSTUVWXYZ It seems correct, doesn't it? However, the locale in effect is ...
2
votes
1answer
742 views

Why does Vim show these strange characters sometimes?

Why is Vim showing these strange characters? I'm running Arch, and locale -a gives this: C en_US.utf8 POSIX ➜$> I've also tried :set fileencoding=utf-8 But, that didn't help either.
2
votes
0answers
896 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
2
votes
0answers
298 views

Generating 'ASCII-art' banners with arrows

I recently discovered figlet which generates ASCII-art banners. Joy! ... but, alas, I want a banner with an arrow on it. Now... $ figlet unicode → arrow _ _ //\ ...
2
votes
1answer
178 views

Understanding script and font relation (character-map)

When I use charactermap, I found that if I change font, only Latin script changes. Character by all other scripts sounds independent with change in font! So, I want to know what is relationship ...
2
votes
1answer
885 views

Rename files with non-utf8 characters using detox

I am trying to use detox in the following way detox -s utf_8-only -r -v -n to detect invalid filenames on my system. The command performs mostly ok. Unfortunately it falsely detects also German ...
2
votes
0answers
554 views

iconv cannot replace Ø

It appears that iconv cannot, for example, replace the letter Ø. It was also noted in the second answer to https://stackoverflow.com/questions/3371697/replacing-accented-characters-php I have two ...
2
votes
1answer
67 views

Non-standard logins

As part of a sort of esoteric "hack" I'm putting together, I've modified my /etc/passwd and /etc/shadow files to result in a user called ニエット ("Niet") with a shell pointing to a custom script I have ...
2
votes
0answers
880 views

Character encoding issue with my linux install?

I just installed Ubuntu 12.04.4 64 bits, all seemed to be ok, when I started to work on console I noticed: that I couldn't get some accented characters typed in to the console, for example, if I ...
2
votes
1answer
78 views

Can't obtain the connected forms of some unicode grapheme clusters with combining diacritical marks

In Arabic and some other languages a character has two or three forms which will find its appropriate one automatically when you type a word (according to its previous and next characters). How can ...
2
votes
0answers
127 views

how to guess and rename file from deleted rar has invalid encoding

i have many files and folders that extracted from many rar from jdownloader, and deleted after extract, some of it names are: �������[�ς݂��ς݂� - �Ȃ񂾂��ꂭ������ (invalid encoding) �R�e���} (invalid ...
2
votes
0answers
126 views

Search in evolution does not find umlaut in different encoding

I use Evolution 3.6.4 on Fedora 18 x86_64, my main language is German. I got a mail (on my imapx account) encoded in iso-8859-15 with the word "hügelig" in it. The mail is displayed correctly in ...
1
vote
1answer
523 views

Tmux is displaying strange characters

I'm having an issue with tmux where it's displaying these strange characters I've found a few posts the seemed to be talking about the same issue and they suggested that I need to change my character ...
1
vote
1answer
680 views

Unknown encoding: POSIX

I'm trying to run xrdb ~/.Xresources but I'm getting the following error: ~/dotfiles/.Xresources:0: warning: Unknown encoding: POSIX Any suggestions? Xft.hinting: true Xft.hintstyle: hintfull Xft....
1
vote
2answers
151 views

Unable to display Greek letters on Mutt 1.7

I recently installed mutt on Linux Mint 18 using apt. I configured it and it worked great for all my three accounts. Then I realized I had mutt version 1.5 while version 1.7 is the latest one. I ...
1
vote
0answers
127 views

certain characters get converted when adding to file

I have a script I am writing, that includes passing across a connection string to socat. When I paste this into the command line it is fine, but when I open a file and paste it into there (pico ...
1
vote
0answers
2k views

curl --data-urlencode and underscores

I've been using curl -XPOST to post some links to a Telegram channel via a bot api/key, the urls are in form of https://site/x/pre_encoded_string, where pre_encoded_string is in form (real samples) ...
1
vote
1answer
400 views

Can not get the file names with convmv

I have just uploaded some files on a FreeBSD server. Some files have the ??? into their names. The files came from a windows machine, so I try this: $ convmv -f cp850 -t utf-8 * Starting a dry run ...
1
vote
0answers
58 views

After rebooting server, service start without special chars

I wrote a simple Java server. This service (java server) connects with a mySQL database. I am connecting to this service with a socket connection. Everything works well when I start my service by ...
1
vote
0answers
285 views

How do I configure medit to display UTF-8?

My environment is set to use UTF-8. I produce a file in UTF-8 encoding. It works correctly on the terminal. For example: $ echo ééé > test $ cat test $ echo é | od -t x1 0000000 c3 a9 0a ...
1
vote
1answer
452 views

Why doesn't this sed substitution for a non-ASCII byte work?

I tried this substitution with GNU sed on OS X (4.2.2 installed through Homebrew). But it doesn't work. printf "Hello\x92 World" | gsed -r "s/[\x92]/'/g" The expected output is: Hello' World The ...
1
vote
2answers
232 views

Emacs shows octal escapes for some characters in a UTF-8 file

I copied some math formulas into some files under gedit, using the default settings of gedit from websites, such as math.stackexchange.com, wikipedia, ... When I open the files now under emacs, the ...
1
vote
0answers
834 views

Convert ASCII-8BIT to UTF-8 using OSX' .bash_profile

When running certain rails scripts in OSX' Terminal, I get warnings such as the following: unable to convert "\xA9" from ASCII-8BIT to UTF-8 for tmp/cache/assets/CC7/4D0/sprockets%...
1
vote
0answers
421 views

Force mysqldump encoding

I have two mysql servers, viewed from the same mysql client, right_server is rendering the french characters correctly, wrong_server is not. Encoding is the same : $ mysql -h right_server -NBe "SHOW ...
1
vote
1answer
370 views

How do I edit text in Cyrillic / Russian on OpenBSD?

I'm trying to make it possible to easily edit text in Cyrillic / Russian on a remote OpenBSD 5.2 amd64. I'm using emacs-21.4p18-no_x11, which doesn't show any Cyrillic characters whatsoever when I ...
0
votes
0answers
18 views

How can i find out which characters that are transliterated with iconv translit

Once a day my server needs to process a huge csv text file. It has to be converted from UTF8 to ISO8859-1 using iconv. Sometimes there has been special characters like ń in the file and this is ...
0
votes
0answers
23 views

I want to convert encoding using iconv

I want to use the iconv command (Linux) to switch to a multi-type encoding file. (Encoding type that wants to be changed) UTF-16 UTF-16BE UTF-16LE CP949 EUC-KR EUC-JP US-ASCII Big5 ISO-2022-JP ISO-...
0
votes
0answers
77 views

Set default file encoding to ASCII?

Years ago, most text editing applications had the option to set the default file encoding to US-ASCII. Now I can't find any recent editor on Linux that allows you to choose ASCII. If the purpose is to ...
0
votes
0answers
242 views

How can I import csv file with special characters?

I am trying to import CSV sheet to Postgres database using PG-ADMIN. I am able to import sheet but after importing sheet due to special characters values are changed. For example: Before Importing: ...
0
votes
0answers
212 views

How to open windows txt with cyrillic text?

How to open Windows text file that have Cyrillic text ? If I open with mousepad there is understandable text. If I type file -bi test.txt text/plain; charset=iso-8859-1
0
votes
0answers
99 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
0
votes
1answer
205 views

Filename displays with ? char when `ls` while with junk character when `ls | grep fran`?

What is the configuration that defines how file name with unrecognizable character be replaced with some other character. For example in my Linux system same file is displayed differently in UTF8 ...
0
votes
1answer
94 views

Different output when executing script manually and via cron/jenkins job

I have a shell script which opens sqlplus and select some data from db containing chinese characters. When I execute it manually, the output contains those characters. When I do it via cron or jenkins ...
0
votes
2answers
292 views

How to change encoding for an unnamed buffer in VIM?

Is it possible? I only know how it can be done for a named buffer (file). It should be reopened with ++enc=<encoding> option. But an unnamed buffer cannot be reopened. I executed :set ...
0
votes
1answer
243 views

force bash script to match filenames with special characters in different encodings

I have a bash script to compare filenames in different locations. They have special characters in the filenames that are in two different encodings, so my script doesn't match those particular files. ...
0
votes
0answers
258 views

Encoding problems on a VPS Ubuntu 16.04

This is greasy. I've checked this only with Polish characters. There's a virtual machine I SSH into. The only exception is when I type something in my terminal. Then it can echo it right. And there's ...
0
votes
0answers
1k views

Doctrine OCI8 charset behavior between php-cli and php-fpm

My Oracle database is in ISO8859-1 (not by choice). I'm struggling with segfault from php-fpm for a couple of days now. To dig out the source of it I've set two parallel environnements with the ...
0
votes
1answer
340 views

make `tar` archive from NTFS with non-ascii file names

Loaded a live disk into a Windows based machine with NTFS files. Being a Linux user for some years, using tar came to my mind instantly and made some backup archives from partitions and stored them ...
0
votes
1answer
165 views

jumbled character set

In my Linux env, for some unknown! reason when I process a raw unformatted file which I copied from Win7prof (was encoded in utf-16), the resulting file is all jumbled. It will print OK in Linux ...
0
votes
1answer
62 views

Is a POSIX-conformant implementation required to support other than default locales?

I would like to implement a minimal POSIX-conformant make utility. However, the specification mentions that the implementation shall be affected by the following environment variables: LANG, LC_ALL, ...