Skip to main content
60 votes
Accepted

Why did {} start appearing as äå in Terminal.app?

I can reproduce it with the xterm terminal emulator (version 366), if I do: $ printf '\e[?42h\e(H'; cat chars.txt; printf '\e(B\e[?42l' !É#$%Ü&*()_+äå Where: \e[?42h. Enables National ...
Stéphane Chazelas's user avatar
30 votes
Accepted

Wget returning binary instead of html?

This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure ...
Gilles 'SO- stop being evil''s user avatar
27 votes
Accepted

How can I get Japanese characters to show properly in Firefox on Arch?

It should be enough to install the great noto fonts bundles: sudo pacman -S noto-fonts-cjk noto-fonts-emoji noto-fonts The restart firefox and you should be abe to see them. Personally, I also ...
terdon's user avatar
  • 244k
27 votes
Accepted

Strange character in a file

This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). ...
Michael Homer's user avatar
22 votes
Accepted

Which character encodings are supported by posix?

There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00. What POSIX does require is that all characters from its Portable Character ...
Michael Homer's user avatar
21 votes
Accepted

How can I identify a strange character?

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë: $ printf "\353\n" | ...
Stephen Kitt's user avatar
20 votes
Accepted

How to get UTF8 from a hex variable?

With xxd (usually shipped with vim) $ echo 5374c3a97068616e650a | xxd -p -r Stéphane If your locale's charset (see output of locale charmap) is not UTF-8 (but can represent all the characters in the ...
Stéphane Chazelas's user avatar
18 votes
Accepted

Convert binary encoding that head and Notepad can read to UTF-8

"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job. The file utility doesn't give useful information when it doesn't recognize the file format. It could be ...
Thomas Dickey's user avatar
17 votes

How can I correctly decompress a ZIP archive of files with Hebrew names?

I had success with the command 7z x <source.zip>. Version: p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,[...]) Potentially relevant environment: LANG=en_US.UTF-8 LC_ALL=...
vsz's user avatar
  • 171
15 votes

Wget returning binary instead of html?

This file is still compressed with gzip. You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it. mv file.dat file.dat.gz gunzip file.dat....
Chris Davies's user avatar
13 votes

Why do some characters show as squares in Chrome?

There's a better way to determine what font you're missing instead of blindly installing font packages. For example I did the following to resolve missing fonts: I received an email with two unknown ...
2bluesc's user avatar
  • 449
13 votes

Can not use `cut -c` (`--characters`) with UTF-8?

colrm (part of util-linux, should be already installed on most distributions) seems to handle internationalization much better : $ echo 'αβγ' | colrm 3 αβ $ echo 'αβγ' | colrm 2 α Beware of the ...
Skippy le Grand Gourou's user avatar
12 votes

echo bytes to a file

Shell In shell you can use printf: printf "%b" '\x66\x6f\x6f' > file.bin Note: %b - Print the associated argument while interpreting backslash escapes. Perl With perl, it is even simpler: ...
kenorb's user avatar
  • 21.2k
12 votes

How to handle accents with enscript?

The usual character encoding in Linux is UTF-8; however, enscript does not support UTF-8. You need to convert the text to an encoding supported by enscript, such as ISO 8859-1: echo "élisée" | iconv -...
AlexP's user avatar
  • 10.5k
12 votes
Accepted

How to translate Unicode characters?

Both GNU and BSD sed are multibyte-aware in appropriate locales, and the y command is analogous to tr: $ echo hello | sed -e 'y/abcdefghijklmnopqrstuvwxyz/abcdefghijklmnopqrstuvwxyz/' hello This ...
Michael Homer's user avatar
11 votes
Accepted

Why do my file names look 'normal' in Linux but not remotely on Windows?

But here comes Windows! Windows has nothing whatsoever to do with this. You could reproduce this same exact behaviour with a local instance of (say) GNOME Terminal, with appropriately selected ...
JdeBP's user avatar
  • 69.3k
11 votes
Accepted

corrupted directories, problem in deleting them

ls will print non-ASCII characters (or rather, characters not supported in the current locale) as ?. This is one of the reasons why parsing the output of ls is a bad thing to do. The output from ls ...
Kusalananda's user avatar
  • 338k
10 votes
Accepted

sed on cygwin can only replace one character?

Most probably, that file is encoded in UTF-16, that is with 2 or 4 bytes per characters, probably even with a Byte-Order-Mark at the beginning. The characters that are shown in your sample (all ASCII ...
Stéphane Chazelas's user avatar
10 votes
Accepted

What's the right way to base64 encode a binary file on CentOS 7?

$ echo foo |base64 Zm9vCg== $ echo foo |base64 |wc -c 9 Note the trailing newline in the output of base64, it's the ninth character here. For longer input, it'll produce more than one line, as it ...
ilkkachu's user avatar
  • 141k
10 votes

How is data encoded in pipes/STDOUT/STDIN?

I'll address each of your points below: Pipes deal with binary, and are agnostic to the encoding Correct. Applications on each side of the pipe (including STDOUT/STDIN) should have consensus on the ...
Andy Dalton's user avatar
  • 14.1k
10 votes

What is the difference between a byte and a character (at least *nixwise)?

POSIXly, emphasis mine: 3.87 Character A sequence of one or more bytes representing a single graphic symbol or control code. In practice, the exact meaning depends on the locale in effect, e.g. ...
ilkkachu's user avatar
  • 141k
10 votes
Accepted

Problems with UTF-8 when attaching to a tmux session over ssh

You're checking the locale settings inside the tmux sessions, but not those that tmux itself receives. server.ltd likely doesn't have AcceptEnv LANG LC_* in its sshd_config or/and you don't have ...
Stéphane Chazelas's user avatar
10 votes
Accepted

View file names in hex?

Pipe the file names to od or a similar tool: printf '%s\n' * | od -t x1 -a $ ls Accentué bar foo $ printf '%s\n' * | od -t x1 -a 0000000 41 63 63 65 6e 74 75 c3 a9 0a 62 61 72 0a 66 ...
Stephen Kitt's user avatar
9 votes

Encoding of cyrillic filenames in zip files

I have found a solution on the OpenNET.ru forum, a popular Russian-language resource that is dedicated to open-source software and technologies since 1996. A post on OpenNET suggests that Info-ZIP, ...
Dima Chubarov's user avatar
9 votes

How to print a variable that contains unprintable characters?

Some various approaches at giving visual representations of strings: POSIX $ printf %s "$IFS" | od -vtc -to1 0000000 \t \n \0 040 011 012 000 0000004 $ printf '%s\n' "$IFS" | LC_ALL=C ...
Stéphane Chazelas's user avatar
9 votes
Accepted

Unexpected non-null encoding of /proc/<pid>/cmdline

at least one uses spaces for delimiters Incorrect. If you look at the end of the pseudo-file on FreeBSD/TrueOS, where you can encounter exactly the same behaviour with Chromium, you will find a &#...
JdeBP's user avatar
  • 69.3k
9 votes

What is "modifier" in locale name?

There is no single unified meaning for the modifier. For example, in the early 2000s, when parts of the EU transitioned from their own national currencies to the Euro, the @euro modifier was used to ...
Jörg W Mittag's user avatar
8 votes

What is the ^M character called?

Code sed -i 's/^M//' filename.txt While typing ^M in the command, do not use ^M as that only inserts what is displayed, not what causes it to be displayed; use CtrlV CtrlM.
Kashyap's user avatar
  • 79
8 votes

Why do some characters show as squares in Chrome?

installing the noto font from google, did it for me. yay -S noto-fonts Now, reload the font cache: fc-cache -vf
Manuel Schmitzberger's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible