2022 Developer Survey is open! Take survey.

Questions tagged [character-encoding]

Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.

Filter by
Sorted by
Tagged with
2 votes
2 answers
101 views

replacement for tr with utf-8 capabilities

in order to isolate the last word in any line of a poem (to have a list of all rhymes), I put together several snippets of code obtaining this awk '{print $NF}' input.txt | tr 'A-Z' 'a-z' | tr -sc 'a-...
user avatar
  • 297
0 votes
1 answer
26 views

Characters do not display properly in terminal (st)

When typing German umlauts (ä,ö,ü) into the terminal (I am using st on Arch Linux, $XTERM is st-256color), it displays only <ffffffff>. Locale seems to be set properly. Output of locale is ...
user avatar
  • 3
0 votes
0 answers
17 views

Something like file command which also checks file headers for encodings?

I'm using the file command to detect mime type and text encoding, and it's working great in general. But one situation it doesn't work so great for is with a HTML file. It's returning us-ascii, but if ...
user avatar
2 votes
1 answer
46 views

VIM shows ^@ every other character and ^M^@ at the end of line

Sorry if it's answered somewhere, I have no idea how to look for it. I received a series of reports from a bank that I'm supposed to process and they seem to be... badly encoded? First two lines in ...
user avatar
  • 975
0 votes
1 answer
57 views

Fix character encoding chaos

I‘m building a script to extract some data from a website with broken character encoding: The html header claims it‘s iso-8859-1, but it‘s not wgetting the file shows me it‘s actually utf-8, but with ...
user avatar
  • 11.2k
0 votes
1 answer
60 views

Ubuntu and Debian have different character encoding for filenames

I am mounting a NFS filesystem to a Debian, an Amazon Linux and an Ubuntu file system and they are showing different character encoding for file name. An example file is displayed as follows in the ...
user avatar
  • 115
0 votes
1 answer
49 views

Unable to change file encoding to utf-8

I modified a UTF-8 encoded xml file using vi editor and saved it. I'm on Redhat Linux 7.9 I checked the file encoding after the changes and found it to be us-ascii file --mime-encoding tmpfiles/08/...
user avatar
  • 297
0 votes
0 answers
17 views

How to copy shared library into initrd?

My initrd cannot convert text file from UTF-16 to UTF-8 by "iconv" and it seems to be caused by there is no the folder "/main/lib/x86_64-linux-gnu/gconv/" and the file "/main/...
user avatar
  • 21
0 votes
4 answers
104 views

>= unary operator expected or -eq integer expression expected

monterey2="12.0" macOS=$(sw_vers -productVersion) if [ "$macOS" >= "$monterey2" ] ; then echo "Monterey" else echo "not Monterey" fi exit 0 ...
user avatar
  • 1
2 votes
1 answer
61 views

How to find a locale name for a given character encoding?

The man page for setlocale seems to say that the language code and the character encoding name are sufficient to set the appropriate locale: A locale name is typically of the form language[_territory]...
user avatar
3 votes
2 answers
218 views

How to substitute non printable characters with space character in a file

I have read nearly every answer about this topic on this website or Stackoverflow but didn't manage to solve the issue below. When I copy the text from a PDF file and paste it into a text file file....
user avatar
1 vote
1 answer
121 views

How to fix the UTF-8 character encoded filenames which don't look good in sub directories

I have filenames like Käyttöohje.pdf. This should be Käyttöohje.pdf. I can do the conversion of all files in sub directories with the command: convmv -f utf8 -t iso-8859-1 -r --notest * This ...
user avatar
  • 13
5 votes
1 answer
313 views

View file names in hex?

I have some files with special characters like accented letters. They are valid names, but for some reason when they are copied across the network to a drive, maybe in another format, the name still ...
user avatar
  • 435
0 votes
0 answers
27 views

Weird characters in output file [duplicate]

I opened the output file created from a script in my terminal and the output file is fine, shows the output i want. cat 2_val.txt 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 ...
user avatar
  • 55
0 votes
2 answers
172 views

How do you execute ls -la when encoded as hex?

There used to be a really bad joke from the internet where you would copy something starting with \x0B\x2E... (just an example) and when you execute this command the terminal would interpret it as ...
user avatar
3 votes
1 answer
193 views

ASCII to ANSI convertion does not work

Trying to convert an ASCII file to ANSI on Linux: $ file -bi relocate.xml text/xml; charset=us-ascii $ iconv -f us-ascii -t ms-ansi < relocate.xml | file -bi - text/xml; charset=us-ascii $ cat ...
user avatar
  • 1,237
1 vote
1 answer
37 views

Why is the special character escaped in the command line

I have an AlmaLinux server, in which I struggle with some encoding issue. For example, if I type this in the CLI: mkdir asdfü ls ./ I get 'asdf'$'\303\274' How can I fix this?
user avatar
  • 463
0 votes
0 answers
33 views

File with Multiple ISO encodings

I have an Arabic file with multiple ISO encodings; encoded in ISO-8859-6 and a few characters encoded in ISO-8859-1 (Because they don't have an equivalent in ISO-8859-6). I want to convert this whole ...
user avatar
  • 167
3 votes
1 answer
479 views

Problems with UTF-8 when attaching to a tmux session over ssh

Scenario: I have a server, on which a tmux session is running. Let's call that session tmux_session in the rest of this post. Attaching to that session can be done in two, slightly different ways: By ...
user avatar
  • 10.3k
0 votes
0 answers
26 views

Problem with utf-8 encode

locale-command return the following: LANG=ru_RU.UTF-8 LC_CTYPE="ru_RU.UTF-8" LC_NUMERIC="ru_RU.UTF-8" LC_TIME="ru_RU.UTF-8" LC_COLLATE="ru_RU.UTF-8" LC_MONETARY=...
user avatar
  • 123
0 votes
0 answers
32 views

Is there an output character to indicate the end of a command in bash from stdout?

I am trying to build a web-app where i can run a bash terminal directly from the browser. I have got the basic functionality working where i fire up a bash process and redirect the stdout to a ...
user avatar
  • 161
2 votes
1 answer
44 views

Can I determine at runtime the available NLS character sets?

I can find the list of the kernel's supported filesystem types in /proc/filesystems. Is there a way to determine which character sets I'm able to use with the mount options iocharset or nls, other ...
user avatar
  • 7,769
0 votes
0 answers
65 views

Unicode Characters Not Displaying after scp to Server

I want to upload some html files to my webserver (running Debian 10 buster) from my laptop (running Ubuntu 20.04). I did so by using scp to transfer the files as a tarball to the remote server. For ...
user avatar
0 votes
1 answer
220 views

cryptsetup with veracrypt container: set encoding?

I'm trying to unlock (and later mount) a veracrypt container like this: sudo cryptsetup --type tcrypt --veracrypt open /home/me/my_container.vc my_container sudo mount /dev/mapper/my_container /media/...
user avatar
  • 103
0 votes
0 answers
143 views

Strace displaying results in ASCII only at process ending and not runtime

I'm searching a way for strace to print the content of the write(...) syscall to ASCII and not useless bytes ! The Strace command I use : sudo strace -e write=1 -e trace=write -s9999 -p 551 2>&...
user avatar
  • 123
1 vote
0 answers
40 views

extracting filenames in ubuntu with non-latin encoding which were archived in windows

This issue has been asked quite a lot (e.g q1, q2), but is still far from solved , at least in Ubuntu 20.04 with the default zip tools (zip, file-roller). The zip files are created in Windows, and in ...
user avatar
  • 11
1 vote
1 answer
1k views

Finding text file's encoding

The iconv command may change file encodings. But is there a command to find the file encoding of a certain text file. Also, I am trying to find what file encoding standards each country use. So that I ...
user avatar
  • 113
1 vote
1 answer
211 views

symbols display wrong in `screen` - ascii or utf

I don't know since when and why : $ mkdir -p a/b/c x/y/{1,2,3} $ tree . ├── a │   └── b │   └── c └── x └── y ├── 1 ├── 2 └── 3 8 directories, 0 files when I run ...
user avatar
1 vote
1 answer
296 views

How to replace Unicode code numbers by UTF-8 characters in a number of files?

I have 500 files that contain escaped Unicode characters like so: Albert Vel\u00e1zquez V\u00e9ronique Ekin etc. The data was queried from a server by a script through XMLHttpRequests and I assume ...
user avatar
  • 7,487
2 votes
1 answer
61 views

How are characters encoded in stty's output?

I ran stty --all on a terminal which had been reconfigured by a badly behaved process which exited before putting my terminal back to its original settings. Part of the output reads: eol = M-^?;. What ...
user avatar
  • 125
0 votes
1 answer
143 views

File is having lots of special characters. How to replace?

I have a file with record like: CENTURY®KETTLEBELLS COLEMANTT40°BAG AlphaPro2¾TDLE Lot™Stretch M6×30 It shows normally in notepad files but in terminal when i cat it, it shows: CENTURY▒KETTLEBELL ...
user avatar
  • 13
17 votes
1 answer
2k views

Why did {} start appearing as äå in Terminal.app?

I'm running CentOS 7.9 and today my terminal is showing weird characters. Some letters look fine, but some symbols appear as some non-English characters. For example I made a text file with the ...
user avatar
  • 493
0 votes
1 answer
100 views

Why Windows text files encoding crash after I open them in Linux system?

I use Debian and sometimes I need to mount Windows disks because I want to read some *.txt file from Windows. Sometimes it's okay, but sometimes after I edit or read file using Debian system and then ...
user avatar
5 votes
3 answers
2k views

What is the difference between a byte and a character (at least *nixwise)?

I understand that any character is comprised of one or more byte/s. If I am not mistaken, at least in *nix operating systems, a character will generally (or totally?) be comprised of only one byte. ...
user avatar
0 votes
2 answers
75 views

$'\r': command not found — file encoding on CentosOS trough cPanel for cloud service

I am writing a silly script trough cPanel on a Centoss remote server, no so much I can do about it, the script need changes from time to time, and upload trough ftp each time get annoying because of ...
user avatar
  • 11
2 votes
2 answers
889 views

What is `â<80><98>` and how to avoid it?

I do a cd /folder/ && find . -not \( -path ./exclude_folder -prune \) > /log.log and get find: â<80><98>./qs/www/ergebnisse/validitaet/0df21b8a-e227-47b2-aaa5-9f54d1f9b8fd.txtâ&...
user avatar
  • 278
1 vote
2 answers
215 views

How to specify [any symbol] in SED?

I am using the SED command to strip out foreign language and other non-keyboard characters from very large text files: example: sed 's/[^a-zA-Z0-9]//g' The above command keeps any line that contains ...
user avatar
3 votes
2 answers
2k views

Terminal doesn't work anymore: _p9k_init_params:72: character not in range manjaro%

I'm on Manjaro GNOME. I installed this package from AUR via Add/Remove Software: https://aur.archlinux.org/packages/github-desktop/ It told me that I need to restart as it /installed/reinstalled some ...
user avatar
  • 177
3 votes
1 answer
398 views

Formatting from ISO-8859-1 to Windows-1251

I want to fix the encoding in a string "Ïåðåïðîøèòü Ñûðîåæêèíà" and get the correct UTF-8 string "Перепрошить Сыроежкина". This website told me that I need to convert from ISO-8859-...
user avatar
1 vote
0 answers
273 views

Ubuntu: Display SOH character in Terminal

When using the terminal with Ubuntu 20.04.1 LTS, the SOH character is not being displayed when outputted from a program. The same program also successfully prints the output to a file which correctly ...
user avatar
  • 119
2 votes
1 answer
348 views

thunderbird: force encoding for all composed emails to unicode

I am using thunderbird 68.8. on Debian 10. I have received an email in HTML format, with what looks to be greek ISO encoding (iso-8859-7) When I click reply, the composed email is automatically set to ...
user avatar
1 vote
0 answers
113 views

Web browsers missing characters

I regularly update my Linux system (mint 19.1 Tessa). After a recent update many of the websites I visit appear to have missing characters, at least visually. I've tried on Firefox and Chromium based ...
user avatar
2 votes
2 answers
255 views

Is there a way to extend iconv to support more encodings?

The iconv command is used to convert from one character encoding to another. You can see the list of all the encodings it supports with iconv -l. Is there a way to extend iconv by adding more ...
user avatar
  • 1,205
-1 votes
1 answer
54 views

how to remove non-standard end of line characters in bash? [closed]

Sometimes, when passing the name of files from a config.txt into a pipe, I get weird problems of non-existing files. I fear it is due to some non-standard end of line characters. I am using kate as ...
user avatar
  • 295
2 votes
2 answers
182 views

Writing Vietnamese in Groff

Using ms macro, is there some way to support more Vietnamese accented characters in Groff? I looked through the page but there doesn't seem to have enough accented characters. https://www.gnu.org/...
user avatar
0 votes
1 answer
114 views

Converting UTF8 characters to slash escaped numeric values

My system language is not Japanese, but my user language/encoding is. Using shell things in terminal emulators 'directly' works okay, but it seems that 'behind the scene' stuff does not. 保存 = Save for ...
user avatar
1 vote
1 answer
140 views

Decode Cyrillic strings

How to decode strings on a text files like this: Âèòàëèé This analyzer (https://2cyr.com/decode/?lang=en) detected the source encoding as WINDOWS-1251 + displayed as ISO-8859-1, and successfully ...
user avatar
  • 23
3 votes
3 answers
548 views

How to properly decode utf7 file names obtained from an IMAP server

I have a few directories that I downloaded from an IMAP server. Example: $ find . -maxdepth 1 . ./&BCEEPwQwBDw- ./&BCMENAQwBDsENQQ9BD0ESwQ1- ./inbox ./&BCcENQRABD0EPgQyBDgEOgQ4- ./&...
user avatar
0 votes
1 answer
73 views

Problems in linux xterm/rxvt/console under SSH after migrating to UTF-8 [duplicate]

I use bash and have been using Latin1 encoding my whole life, since I've always had problems when trying to migrate to the advised UTF-8. I finally gave up, and I am migrating everything to UTF-8. ...
user avatar
1 vote
1 answer
94 views

Missing one character in ANSI art on Linux

This continuation of Why this ANSI Art doesn't render properly on Linux terminal I have one more issue with ANSI art on Linux terminal that renders properly in ANSI art editor (like PabloDraw). The ...
user avatar
  • 8,730

1
2 3 4 5
8