Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
0
votes
0answers
49 views

Question marks in my kali linux shell

I am trying to do active reconnaissance in Kali Linux (4.19.0-kali3-amd64) using xprobe2 tool. So I typed the command xprobe2 espn.com (for example). I know that espn.com use "OpenBSD" ...
2
votes
1answer
279 views

How is data encoded in pipes/STDOUT/STDIN?

I have recently been looking into various text encodings and I am not able to find any good sources on how data is encoded in pipes. Here are some general assumptions I have: Pipes deal with binary, ...
0
votes
1answer
9 views

How to fix malformatted special characters in Pop OS, that occoured after an system crash?

This afternoon my computer (operating Pop OS!) was not reacting to keyboard or mouse interactions, I tried everything but couldn't find out why this happened. So I did a cold start, I assume the ...
0
votes
0answers
17 views

freeipa commands are erroring intermittently with ipa: ERROR: System encoding must be UTF-8

This happens sometimes, then other times there are no problems at all: kahn@host:~$ ipa user-status kahn ipa: ERROR: System encoding must be UTF-8, 'iso8859-1' is not supported. Set LC_ALL="C.UTF-...
0
votes
0answers
26 views

JavaScript: Escape (encode?) special characters in filenames [migrated]

I have a JavaScript that runs an external program (mediainfo) on local files. However, some of my files have characters that mess up my code. The code: objMedInfo = JSON.parse(proc.execSync('...
0
votes
1answer
36 views

How to write and read in a different encoding from the terminal?

I'm using terminator on debian, and its enconding is set to UTF-8. In most cases this isn't a problem since almost everything that's recent has the encoding set to utf-8 aswell. When I connect to a ...
0
votes
0answers
7 views

Using MYSQL_PARAMETERS to update mariadb table

I have an array of struct data that I want to update a table in mariadb, but i can't get around the Swedish characters ( å. ä , ö ), they display as '?'-chars or blank. My table consists of 32 rows ( ...
0
votes
1answer
16 views

Change from two different encodings to UTF-8

awk -F : '$1 ~ /[[:digit:]]+[[:alnum:]]*[[:digit:]]+/ && ($3>6200) {print $5" --- "$1" --- "$3;count++} END{print"\n----------\nSuma znalezionych rekordów:"count"\n----------\n"}' /etc/...
3
votes
2answers
46 views

File name character encoding gets confused when called with find -exec

I'd like to simplify the example, but any further simplification removes the problem … I already stripped my script down to this: #!/bin/sh echo "$1" | sed 's/[^[:alnum:]]//g' This does what I ...
0
votes
1answer
47 views

How to turn mojibake text to readable form?

I got mojibake text (from MP3 ID3 fields) that need to be converted to the right form. I tried to: echo 'c2 8d 44 e2 80 99 c2 b2 c2 8d 44 e2 80 99 c2 b2' | xxd -p -r | iconv -f sjis -t utf8; echo ...
1
vote
2answers
28 views

Vim is displaying medium dash as �~@~S and long dash as �~@~R

I'm using vim on a shared Bluehost server. When I try to type —, Vim inserts �~@~R. And when I try to type –, Vim inserts �~@~S. The problem is not specific to any particular file. It seems to ...
0
votes
0answers
15 views

Double lined borders on the console

Is there an easy way how to programatically check if double lined borders are supported on the given terminal? Everything works fine if I have xterm or xterm256, but when I switch to tty only ASCII ...
0
votes
0answers
26 views

Changing language settings from de_de@EURO to de_DE.UTF-8 results in UMLAUT errors in ERP FORM MASKS and OUTPUTS

I ran into a kind of bigger problem. In my company we have an In-house ERP-server (SUSE ENTERPRISE). I wanted to write a bash script that adds users automatically. However, I needed the signs "\" (=...
0
votes
1answer
57 views

Replace character in file based on position

I work on the encoding of a file, from unknown-8bit to UTF-8. but I got an error message. iconv: illegal input sequence at position 136828 So I did a search in the file with the position like this : ...
1
vote
0answers
43 views

Broken pipe symbol from SQL changing in Linux

I am migrating Solaris code to Linux and have problem with broken pipe symbol used inside sql scripts. There are multiple SQL scripts (and output it produces) with broken pipe symbol (¦) which works ...
1
vote
1answer
38 views

Mozilla won't show special characters (such as chinese, korean, japanese, etc…) just numbers in squares on Debian

I just installed Debian 10.3 and when I try to read special characters i see what is shown on the picture below. What am I supposed to do? I've read some solutions, but they were only for arch linux, ...
0
votes
2answers
299 views

How to fix strange characters in my .txt file?

I edited a .txt file in Visual Studio on Windows and then copied that into a HPC server. The file looks fine to me at first, But when I opened it in the linux environment, weird characters showed up (...
1
vote
1answer
79 views

Cannot convert from utf16be to utf8

I'm working on a script for aix 7.2 and I'm getting an error when I try to convert from utf16be to utf8. I'm using iconv to convert. Using the follow the script: for file in *.OPEN do echo "...
1
vote
0answers
114 views

Unknown character encoding for zip file in archive manager

Forenote: I am quite new to Linux, and am really just a beginner. I have downloaded a zip file from a respectable-looking site with firefox; it should contain stuff to install an application. The zip ...
0
votes
0answers
48 views

Converting text file encodings

I have been having a horrible time with text files that mostly have cp1252 encoding, but some don't. When I try to iterate over them in Python, I'll hit a file with some other weird encoding and the ...
0
votes
1answer
50 views

What should ASCII character 26 do?

I have code where I display ANSI art, I've found that last 128 bytes hold SAUCE meta data. The problem I have is just before the SAUCE data there is one character that looks out of place, that is ...
0
votes
0answers
56 views

How to find which encoding algorithm is used in the file

I have a file with encoded content. I am trying to figure out what encoding algorithm is being used to encode the file contents. Is there a tool or command that can help me find which encoding ...
1
vote
0answers
32 views

If I know the corresponding encoded bytes to some characters, how can I determine matching charsets?

I have a few files where the filenames obviously use the wrong charset. As soon as I know which charset is being used, I can use convmv to convert it to UTF-8. From the rest of the filename, I can ...
0
votes
0answers
27 views

How can i find out which characters that are transliterated with iconv translit

Once a day my server needs to process a huge csv text file. It has to be converted from UTF8 to ISO8859-1 using iconv. Sometimes there has been special characters like ń in the file and this is ...
0
votes
0answers
134 views

I want to convert encoding using iconv

I want to use the iconv command (Linux) to switch to a multi-type encoding file. (Encoding type that wants to be changed) UTF-16 UTF-16BE UTF-16LE CP949 EUC-KR EUC-JP US-ASCII Big5 ISO-2022-JP ISO-...
0
votes
3answers
190 views

Creating/using my own 'custom' character

Is it possible to create your own ascii character that you can use? If so, how would it be done? Now my use case would be something like this: Let's say I created the character #12345, and I am ...
4
votes
3answers
86 views

How to find character locations of a string in a file?

I need to search for a string (a sequence of characters) in a file with a certain encoding, typically utf8, but return the character offsets (not byte offsets) of the results. So this is a search ...
2
votes
1answer
648 views

SFTP remote readdir Failure - national characters in file name

I have created a directory with national characters in it's name. I can see and browse this directory via ssh # cd /root # ls СтранноеРусскоеИмя but can't do it via SFTP: sftp> cd /root sftp>...
-1
votes
1answer
46 views

How do we get the readable string from big endian hex string

How do we get the readable string from hex string 0049006e00740072006f of BE which means as normal word, ie. we dont have to swap the byte pair per word tried with xxd in vain, cause I dont know ...
0
votes
1answer
940 views

Distinguish ascii from UTF-8 characters in the same file

On Ubuntu 18.04, I created a dummy text file with just one UTF-8 character, è. The other characters are all ascii: $ cat dummytext Hello Helloè This is the resulting hexdump: $ hexdump -C dummyfile ...
2
votes
1answer
469 views

Disabling utf-8 in all terminals?

This question pertains to my Ubuntu 18 system, as follows ... % uname -a Linux myhost.name.net 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux I don't ...
0
votes
0answers
115 views

Set default file encoding to ASCII?

Years ago, most text editing applications had the option to set the default file encoding to US-ASCII. Now I can't find any recent editor on Linux that allows you to choose ASCII. If the purpose is to ...
2
votes
2answers
1k views

ls won't display Cyrillic characters properly

I've got a problem with Cyrillic alphabet. My terminal will display Cyrillic normally as well most of my programs the only exception I've found so far is ls. Instead of normal Cyrillic it displays ????...
-1
votes
1answer
133 views

Can't see the contents of a data file in terminal

I have downloaded a number of input files from the NASDAQ website (for example ftp://emi.nasdaq.com/ITCH/01302018.NASDAQ_ITCH50.gz) After decompressing (I double click on a Mac) I can't see the file ...
0
votes
0answers
473 views

How can I import csv file with special characters?

I am trying to import CSV sheet to Postgres database using PG-ADMIN. I am able to import sheet but after importing sheet due to special characters values are changed. For example: Before Importing: ...
1
vote
1answer
73 views

(Possible) inconsistent behavior of grep and less

I have a utf-8 file containing some Turkish text inside. (My system is MacOSX) $ file -I foo.merge $foo.merge: text/plain; charset=utf-8 When I try to see some Turkish specific characters by using ...
0
votes
0answers
422 views

How to open windows txt with cyrillic text?

How to open Windows text file that have Cyrillic text ? If I open with mousepad there is understandable text. If I type file -bi test.txt text/plain; charset=iso-8859-1
6
votes
3answers
5k views

Strange character in a file

I have an UTF-8 file that contains a strange character -- visible to me just as <96> This is how it appears on vi and how it appears on gedit and how it appears under LibreOffice and that ...
3
votes
1answer
83 views

UTF8 Character Makes File Inaccessible

If I run: scp me@example.com:/home/me/cömmön_file.jpg /home/me/ from my remote server I get: scp: /home/me/cömmön_file.jpg: No such file or directory If I swap out the utf8 characters though ...
0
votes
3answers
180 views

Demonstrate file encoding

I am developing some training and would like to demonstrate file encoding. What I am trying to accomplish is create a text file with one type of encoding that show up as nonsense when read by Linux. ...
0
votes
1answer
754 views

Convert text file to non-iso extended ascii file

I have a file which is Non-ISO Extended ASCII type. I can't see its content just by opening it normally via cat or vim. However, I'm able to see its content in readable form with this command... od -...
0
votes
1answer
556 views

Choosing mousepad encoding

I have some text file with windows-1251 encoding (rus), but when I open it in mousepad there is no such encoding in the drop-down list, and there is no utf-8 also. The question is: how to add ...
0
votes
0answers
153 views

file command gives incorrect encoding type

I have a csv file in hand. When I ran 'file -i filename', it shows that it's encoded as us-ascii. But when I ran cat filename | csvcut -t -e us-ascii, I got an error: "Your file is not "us-ascii" ...
2
votes
1answer
414 views

Can't get iso-8859-1 to work for 'umlauts' (Debian / Plesk)

I think I have configured ISO-8859-1 at all necessary locations, but still, my pages are displaying black '?' instead of German umlauts! My server runs Plesk 17.8.11 / Debian 8 / Apache 2 / PHP 7.1 (...
3
votes
1answer
287 views

CONVMV and cyrillic filenames

I am trying to convert filenames in russian zipfile to UTF-8 using convmv. Original filename: "æ óá¡¿½∞¡δ¼ áα«¼áΓ«¼.jpg" (æ\ óá¡¿½∞¡δ¼\ áα«¼áΓ«¼.jpg with slashes) This analyzer (https://2cyr.com/...
0
votes
1answer
83 views

Is a POSIX-conformant implementation required to support other than default locales?

I would like to implement a minimal POSIX-conformant make utility. However, the specification mentions that the implementation shall be affected by the following environment variables: LANG, LC_ALL, ...
2
votes
2answers
229 views

How to generate a ascii char in a string with Bash / ZSH correctly?

I'm trying to generate a string of 10 followed by a NULL (\0) followed by a 10. However echo "10\010" does not seem to work (I'm guessing it generates two chars - a 10 followed by \010. I'm not sure ...
0
votes
1answer
101 views

AIX x Linux line endings

Using Meld to compare two identical files in terms of contents, it give a warning like this: ... There was a problem opening the file "...\CHXXXX_...\psart20p.pc". There was an encoding ...
0
votes
1answer
263 views

Filename displays with ? char when `ls` while with junk character when `ls | grep fran`?

What is the configuration that defines how file name with unrecognizable character be replaced with some other character. For example in my Linux system same file is displayed differently in UTF8 ...
2
votes
1answer
166 views

Encoding problems with German man pages

On the machine I am working on I have English man pages under /usr/share/man/and German ones under /usr/share/man/de. The former are encoded with us-ascii, the latter ones with utf-8. Unfortunately ...

1
2 3 4 5
7