72 votes
Accepted

What is the ^M character called?

It is known as carriage return. If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r. Inserting 0x0D in a hex editor will do the task. ...
user avatar
  • 36.8k
60 votes
Accepted

Why did {} start appearing as äå in Terminal.app?

I can reproduce it with the xterm terminal emulator (version 366), if I do: $ printf '\e[?42h\e(H'; cat chars.txt; printf '\e(B\e[?42l' !É#$%Ü&*()_+äå Where: \e[?42h. Enables National ...
user avatar
45 votes
Accepted

Filtering invalid utf8

If you want to use grep, you can do: grep -axv '.*' file in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).
user avatar
  • 11.3k
44 votes

Converting a UTF-8 file to ASCII (best-effort)

This will work for some things: iconv -f utf-8 -t ascii//TRANSLIT echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be ...
user avatar
42 votes

tr complains of “Illegal byte sequence”

grep is a text processing tool. It expects their input to be text files. It seems that the same goes for tr on macOS (even though tr is supposed to support binary files). Computers store data as ...
user avatar
36 votes
Accepted

Generating a random password; why isn't this portable?

It's your locale and tr problem. Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird: $ </dev/urandom LC_ALL=vi_VN.tcvn ...
user avatar
  • 144k
29 votes
Accepted

Wget returning binary instead of html?

This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure ...
user avatar
27 votes
Accepted

Strange character in a file

This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). ...
user avatar
26 votes
Accepted

find(1): how is the star wildcard implemented for it to fail on some filenames?

That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c): b = fnmatch (...
user avatar
  • 14.6k
25 votes
Accepted

Can vim display ASCII characters only, and treat other bytes as binary data?

When using vim -b, this displays all high characters as <xx>: set encoding=latin1 set isprint= set display+=uhex Any single-byte encoding will work, vim uses ASCII for all lower chars and has ...
user avatar
  • 516
25 votes
Accepted

How to convert html entities to readable text?

With Free recode (formerly known as GNU recode): recode html < file If you don't have recode or HTML::Entities and only need to decode &#x<hex>; entities, you could do it by hand with: ...
user avatar
25 votes
Accepted

How can I correctly decompress a ZIP archive of files with Hebrew names?

It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc). Is there another decompression utility that will decompress my files with the correct names? I'm ...
user avatar
23 votes

tr complains of “Illegal byte sequence”

I suppose that your charmap from the locales is UTF-8, so that you'll have problems on binary files. Just switch to C locale: LC_ALL=C tr '\r' '\n' < target-file | LC_ALL=C grep search-string
user avatar
  • 11.3k
22 votes

How to convert an emoticon specified by a U+xxxxx code to utf-8?

UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding. To get ...
user avatar
22 votes
Accepted

Which character encodings are supported by posix?

There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00. What POSIX does require is that all characters from its Portable Character ...
user avatar
22 votes
Accepted

How can I identify a strange character?

Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë: $ printf "\353\n" | ...
user avatar
19 votes
Accepted

iconv illegal input sequence- why?

The file is encoded in ISO-8859-1, not in UTF-8: $ hd 0606461.txt | grep -B1 '^0002c520' 0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin| 0002c520 e6 0d 0a 2d 2d 48 6f ...
user avatar
  • 11.3k
19 votes
Accepted

Can not use `cut -c` (`--characters`) with UTF-8?

You haven't said which cut you're using, but since you've mentioned the GNU long option --characters I'll assume it's that one. In that case, note this passage from info coreutils 'cut invocation': ‘-...
user avatar
18 votes
Accepted

Process a file that starts with a BOM (FF FE)

From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8: iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt
user avatar
  • 144k
18 votes
Accepted

Converting a UTF-8 file to ASCII (best-effort)

konwert utf8-ascii It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g. ...
user avatar
18 votes
Accepted

Convert binary encoding that head and Notepad can read to UTF-8

"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job. The file utility doesn't give useful information when it doesn't recognize the file format. It could be ...
user avatar
18 votes
Accepted

How can I get Japanese characters to show properly in Firefox on Arch?

It should be enough to install the great noto fonts bundles: sudo pacman -S noto-fonts-cjk noto-fonts-emoji noto-fonts The restart firefox and you should be abe to see them. Personally, I also ...
user avatar
  • 217k
17 votes
Accepted

Octals 302 240 together seem to correspond to non-breaking space

It's the UTF-8 encoding of the U+00A0 Unicode character: $ unicode U+00A0 U+00A0 NO-BREAK SPACE UTF-8: c2 a0 UTF-16BE: 00a0 Decimal: &#160; Octal: \0240   Category: Zs (Separator, Space) Bidi: CS ...
user avatar
17 votes
Accepted

How to get UTF8 from a hex variable?

With xxd (usually shipped with vim) $ echo 5374c3a97068616e650a | xxd -p -r Stéphane If your locale's charset (see output of locale charmap) is not UTF-8 (but can represent all the characters in the ...
user avatar
16 votes

How can I correctly decompress a ZIP archive of files with Hebrew names?

I had success with the command 7z x <source.zip>. Version: p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,[...]) Potentially relevant environment: LANG=en_US.UTF-8 LC_ALL=...
user avatar
  • 161
16 votes
Accepted

Specify encoding with libreoffice --convert-to csv

Apparently LibreOffice tries to use ISO-8859-1 by default, which is causing the problem. In response to this bug report, a new parameter --infilter has been added. The following command produces U+...
user avatar
  • 571
15 votes

Convert a .docx to a .pdf with pandoc

This still comes up on google searches so I wanted to put this on the record: pandoc could not read docx when this question was asked (the error comes from trying to read a binary file) but since ...
user avatar
  • 151
15 votes
Accepted

"mv" file with garbled name by inode number?

You could try: find . -inum 12321475 -exec mv {} new-filename \; or find . -inum 12321475 -print0 | xargs -0 mv -t new-filename Generally I prefer xargs over exec. Google for why. It's tricky ...
user avatar
15 votes

Wget returning binary instead of html?

This file is still compressed with gzip. You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it. mv file.dat file.dat.gz gunzip file.dat....
user avatar
  • 90.3k
14 votes

Prevent tail from modifying the charset of the terminal

You can translate special characters (binary data) into ordinary characters that are safe to display by piping your tail command into cat -v: tail --follow=name my-rolling-file.log | cat -v The -v (...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible