Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
0
votes
1answer
40 views

Why Windows text files encoding crash after I open them in Linux system?

I use Debian and sometimes I need to mount Windows disks because I want to read some *.txt file from Windows. Sometimes it's okay, but sometimes after I edit or read file using Debian system and then ...
4
votes
3answers
553 views

What is the difference between a byte and a character (at least *nixwise)?

I understand that any character is comprised of one or more byte/s. If I am not mistaken, at least in *nix operating systems, a character will generally (or totally?) be comprised of only one byte. ...
0
votes
2answers
49 views

$'\r': command not found — file encoding on CentosOS trough cPanel for cloud service

I am writing a silly script trough cPanel on a Centoss remote server, no so much I can do about it, the script need changes from time to time, and upload trough ftp each time get annoying because of ...
2
votes
2answers
273 views

What is `â<80><98>` and how to avoid it?

I do a cd /folder/ && find . -not \( -path ./exclude_folder -prune \) > /log.log and get find: â<80><98>./qs/www/ergebnisse/validitaet/0df21b8a-e227-47b2-aaa5-9f54d1f9b8fd.txtâ&...
1
vote
2answers
41 views

How to specify [any symbol] in SED?

I am using the SED command to strip out foreign language and other non-keyboard characters from very large text files: example: sed 's/[^a-zA-Z0-9]//g' The above command keeps any line that contains ...
0
votes
2answers
799 views

Terminal doesnt work anymore: _p9k_init_params:72: character not in range manjaro%

Im on Manjaro GNOME. I installed this package from AUR via Add/Remove Software: https://aur.archlinux.org/packages/github-desktop/ It told me that I need to restart as it /installed/reinstalled some ...
2
votes
1answer
32 views

Formatting from ISO-8859-1 to Windows-1251

I want to fix the encoding in a string "Ïåðåïðîøèòü Ñûðîåæêèíà" and get the correct UTF-8 string "Перепрошить Сыроежкина". This website told me that I need to convert from ISO-8859-...
1
vote
0answers
30 views

Ubuntu: Display SOH character in Terminal

When using the terminal with Ubuntu 20.04.1 LTS, the SOH character is not being displayed when outputted from a program. The same program also successfully prints the output to a file which correctly ...
2
votes
1answer
50 views

thunderbird: force encoding for all composed emails to unicode

I am using thunderbird 68.8. on Debian 10. I have received an email in HTML format, with what looks to be greek ISO encoding (iso-8859-7) When I click reply, the composed email is automatically set to ...
1
vote
0answers
27 views

Web browsers missing characters

I regularly update my Linux system (mint 19.1 Tessa). After a recent update many of the websites I visit appear to have missing characters, at least visually. I've tried on Firefox and Chromium based ...
2
votes
2answers
33 views

Is there a way to extend iconv to support more encodings?

The iconv command is used to convert from one character encoding to another. You can see the list of all the encodings it supports with iconv -l. Is there a way to extend iconv by adding more ...
-1
votes
1answer
44 views

how to remove non-standard end of line characters in bash? [closed]

Sometimes, when passing the name of files from a config.txt into a pipe, I get weird problems of non-existing files. I fear it is due to some non-standard end of line characters. I am using kate as ...
2
votes
1answer
82 views

Writing Vietnamese in Groff

Using ms macro, is there some way to support more Vietnamese accented characters in Groff? I looked through the page but there doesn't seem to have enough accented characters. https://www.gnu.org/...
0
votes
1answer
21 views

Converting UTF8 characters to slash escaped numeric values

My system language is not Japanese, but my user language/encoding is. Using shell things in terminal emulators 'directly' works okay, but it seems that 'behind the scene' stuff does not. 保存 = Save for ...
1
vote
1answer
58 views

Decode Cyrillic strings

How to decode strings on a text files like this: Âèòàëèé This analyzer (https://2cyr.com/decode/?lang=en) detected the source encoding as WINDOWS-1251 + displayed as ISO-8859-1, and successfully ...
2
votes
2answers
62 views

How to properly decode utf7 file names obtained from an IMAP server

I have a few directories that I downloaded from an IMAP server. Example: $ find . -maxdepth 1 . ./&BCEEPwQwBDw- ./&BCMENAQwBDsENQQ9BD0ESwQ1- ./inbox ./&BCcENQRABD0EPgQyBDgEOgQ4- ./&...
0
votes
1answer
25 views

Problems in linux xterm/rxvt/console under SSH after migrating to UTF-8 [duplicate]

I use bash and have been using Latin1 encoding my whole life, since I've always had problems when trying to migrate to the advised UTF-8. I finally gave up, and I am migrating everything to UTF-8. ...
1
vote
1answer
42 views

Missing one character in ANSI art on Linux

This continuation of Why this ANSI Art doesn't render properly on Linux terminal I have one more issue with ANSI art on Linux terminal that renders properly in ANSI art editor (like PabloDraw). The ...
1
vote
1answer
70 views

Why this ANSI Art doesn't render properly on Linux terminal

I'm trying to investigate why my library don't render this ANSI art properly: I though that the problem is in my ANSI escape processing but it seems that it's the same in real terminal (tested in ...
2
votes
1answer
45 views

Mutt subject is garbled when using japanese/chinese characters

When using mutt command: mutt -e "set content_type=text/plain" -e "my_hdr From:abc@gmail.com" -s "取样更多 サンプル 样品 text" -c "取样更多" -a "サンプル取样更多" Now ...
1
vote
1answer
80 views

How do I properly convert the file to UTF-16LE encoding without strange characters appearing in the file?

I'm having some peculiarities with the dictionary file of .dsl format I'm trying to convert. It's essentially a text file with the dictionary pairs. The dictionary software I use is GoldenDict. It ...
4
votes
1answer
152 views

Is there any ligature-aware alternative for “pdfgrep” in command line?

I always use "pdfgrep" to search inside of multiple PDF files from the command line. But I met a problem: This ligature character "fi" (see https://www.compart.com/en/unicode/U+FB01)...
0
votes
1answer
189 views

convert encoding of id3 tags of mp3 file

My mp3 file has id3 tags which can be displayed in a terminal on my debian buster linux for example by using the program id3tool or id3. If my shell (xfce4-terminal) has a set default character ...
0
votes
0answers
96 views

Question marks in my kali linux shell

I am trying to do active reconnaissance in Kali Linux (4.19.0-kali3-amd64) using xprobe2 tool. So I typed the command xprobe2 espn.com (for example). I know that espn.com use "OpenBSD" ...
2
votes
1answer
688 views

How is data encoded in pipes/STDOUT/STDIN?

I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes. Here are some general assumptions I have: Pipes deal with binary, ...
0
votes
1answer
20 views

How to fix malformatted special characters in Pop OS, that occoured after an system crash?

This afternoon my computer (operating Pop OS!) was not reacting to keyboard or mouse interactions, I tried everything but couldn't find out why this happened. So I did a cold start, I assume the ...
0
votes
1answer
48 views

How to write and read in a different encoding from the terminal?

I'm using terminator on debian, and its enconding is set to UTF-8. In most cases this isn't a problem since almost everything that's recent has the encoding set to utf-8 aswell. When I connect to a ...
0
votes
0answers
8 views

Using MYSQL_PARAMETERS to update mariadb table

I have an array of struct data that I want to update a table in mariadb, but i can't get around the Swedish characters ( å. ä , ö ), they display as '?'-chars or blank. My table consists of 32 rows ( ...
0
votes
1answer
23 views

Change from two different encodings to UTF-8

awk -F : '$1 ~ /[[:digit:]]+[[:alnum:]]*[[:digit:]]+/ && ($3>6200) {print $5" --- "$1" --- "$3;count++} END{print"\n----------\nSuma znalezionych rekordów:"count"\n----------\n"}' /etc/...
3
votes
2answers
68 views

File name character encoding gets confused when called with find -exec

I'd like to simplify the example, but any further simplification removes the problem … I already stripped my script down to this: #!/bin/sh echo "$1" | sed 's/[^[:alnum:]]//g' This does what I ...
0
votes
1answer
166 views

How to turn mojibake text to readable form?

I got mojibake text (from MP3 ID3 fields) that need to be converted to the right form. I tried to: echo 'c2 8d 44 e2 80 99 c2 b2 c2 8d 44 e2 80 99 c2 b2' | xxd -p -r | iconv -f sjis -t utf8; echo ...
1
vote
2answers
42 views

Vim is displaying medium dash as �~@~S and long dash as �~@~R

I'm using vim on a shared Bluehost server. When I try to type —, Vim inserts �~@~R. And when I try to type –, Vim inserts �~@~S. The problem is not specific to any particular file. It seems to ...
0
votes
0answers
24 views

Double lined borders on the console

Is there an easy way how to programatically check if double lined borders are supported on the given terminal? Everything works fine if I have xterm or xterm256, but when I switch to tty only ASCII ...
0
votes
0answers
30 views

Changing language settings from de_de@EURO to de_DE.UTF-8 results in UMLAUT errors in ERP FORM MASKS and OUTPUTS

I ran into a kind of bigger problem. In my company we have an In-house ERP-server (SUSE ENTERPRISE). I wanted to write a bash script that adds users automatically. However, I needed the signs "\" (=...
0
votes
1answer
132 views

Replace character in file based on position

I work on the encoding of a file, from unknown-8bit to UTF-8. but I got an error message. iconv: illegal input sequence at position 136828 So I did a search in the file with the position like this : ...
1
vote
0answers
56 views

Broken pipe symbol from SQL changing in Linux

I am migrating Solaris code to Linux and have problem with broken pipe symbol used inside sql scripts. There are multiple SQL scripts (and output it produces) with broken pipe symbol (¦) which works ...
1
vote
1answer
106 views

Mozilla won't show special characters (such as chinese, korean, japanese, etc…) just numbers in squares on Debian

I just installed Debian 10.3 and when I try to read special characters i see what is shown on the picture below. What am I supposed to do? I've read some solutions, but they were only for arch linux, ...
0
votes
2answers
1k views

How to fix strange characters in my .txt file?

I edited a .txt file in Visual Studio on Windows and then copied that into a HPC server. The file looks fine to me at first, But when I opened it in the linux environment, weird characters showed up (...
1
vote
1answer
121 views

Cannot convert from utf16be to utf8

I'm working on a script for aix 7.2 and I'm getting an error when I try to convert from utf16be to utf8. I'm using iconv to convert. Using the follow the script: for file in *.OPEN do echo "...
1
vote
0answers
241 views

Unknown character encoding for zip file in archive manager

Forenote: I am quite new to Linux, and am really just a beginner. I have downloaded a zip file from a respectable-looking site with firefox; it should contain stuff to install an application. The zip ...
0
votes
0answers
52 views

Converting text file encodings

I have been having a horrible time with text files that mostly have cp1252 encoding, but some don't. When I try to iterate over them in Python, I'll hit a file with some other weird encoding and the ...
0
votes
1answer
123 views

What should ASCII character 26 do?

I have code where I display ANSI art, I've found that last 128 bytes hold SAUCE meta data. The problem I have is just before the SAUCE data there is one character that looks out of place, that is ...
0
votes
0answers
128 views

How to find which encoding algorithm is used in the file

I have a file with encoded content. I am trying to figure out what encoding algorithm is being used to encode the file contents. Is there a tool or command that can help me find which encoding ...
1
vote
0answers
33 views

If I know the corresponding encoded bytes to some characters, how can I determine matching charsets?

I have a few files where the filenames obviously use the wrong charset. As soon as I know which charset is being used, I can use convmv to convert it to UTF-8. From the rest of the filename, I can ...
0
votes
3answers
376 views

Creating/using my own 'custom' character

Is it possible to create your own ascii character that you can use? If so, how would it be done? Now my use case would be something like this: Let's say I created the character #12345, and I am ...
4
votes
3answers
94 views

How to find character locations of a string in a file?

I need to search for a string (a sequence of characters) in a file with a certain encoding, typically utf8, but return the character offsets (not byte offsets) of the results. So this is a search ...
2
votes
1answer
982 views

SFTP remote readdir Failure - national characters in file name

I have created a directory with national characters in it's name. I can see and browse this directory via ssh # cd /root # ls СтранноеРусскоеИмя but can't do it via SFTP: sftp> cd /root sftp>...
-1
votes
1answer
59 views

How do we get the readable string from big endian hex string

How do we get the readable string from hex string 0049006e00740072006f of BE which means as normal word, ie. we dont have to swap the byte pair per word tried with xxd in vain, cause I dont know ...
0
votes
1answer
1k views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
2
votes
1answer
683 views

Disabling utf-8 in all terminals?

This question pertains to my Ubuntu 18 system, as follows ... % uname -a Linux myhost.name.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I don't ...

1
2 3 4 5
8