72
votes
Accepted
What is the ^M character called?
It is known as carriage return.
If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r.
Inserting 0x0D in a hex editor will do the task.
...
- 36.8k
60
votes
Accepted
Why did {} start appearing as äå in Terminal.app?
I can reproduce it with the xterm terminal emulator (version 366), if I do:
$ printf '\e[?42h\e(H'; cat chars.txt; printf '\e(B\e[?42l'
!É#$%Ü&*()_+äå
Where:
\e[?42h. Enables National ...
- 470k
45
votes
Accepted
Filtering invalid utf8
If you want to use grep, you can do:
grep -axv '.*' file
in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).
- 11.3k
44
votes
Converting a UTF-8 file to ASCII (best-effort)
This will work for some things:
iconv -f utf-8 -t ascii//TRANSLIT
echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be ...
- 867
42
votes
tr complains of “Illegal byte sequence”
grep is a text processing tool. It expects their input to be text files. It seems that the same goes for tr on macOS (even though tr is supposed to support binary files).
Computers store data as ...
36
votes
Accepted
Generating a random password; why isn't this portable?
It's your locale and tr problem.
Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird:
$ </dev/urandom LC_ALL=vi_VN.tcvn ...
- 144k
29
votes
Accepted
Wget returning binary instead of html?
This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure ...
27
votes
Accepted
Strange character in a file
This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). ...
- 70.6k
26
votes
Accepted
find(1): how is the star wildcard implemented for it to fail on some filenames?
That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c):
b = fnmatch (...
- 14.6k
25
votes
Accepted
Can vim display ASCII characters only, and treat other bytes as binary data?
When using vim -b, this displays all high characters as <xx>:
set encoding=latin1
set isprint=
set display+=uhex
Any single-byte encoding will work, vim uses ASCII for all lower chars and has ...
- 516
25
votes
Accepted
How to convert html entities to readable text?
With Free recode (formerly known as GNU recode):
recode html < file
If you don't have recode or HTML::Entities and only need to decode &#x<hex>; entities, you could do it by hand with:
...
- 470k
25
votes
Accepted
How can I correctly decompress a ZIP archive of files with Hebrew names?
It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc).
Is there another decompression utility that will decompress my files with the correct names? I'm ...
- 70.6k
23
votes
tr complains of “Illegal byte sequence”
I suppose that your charmap from the locales is UTF-8, so that you'll have problems on binary files. Just switch to C locale:
LC_ALL=C tr '\r' '\n' < target-file | LC_ALL=C grep search-string
- 11.3k
22
votes
How to convert an emoticon specified by a U+xxxxx code to utf-8?
UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding.
To get ...
- 470k
22
votes
Accepted
Which character encodings are supported by posix?
There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00.
What POSIX does require is that all characters from its Portable Character ...
- 70.6k
22
votes
Accepted
How can I identify a strange character?
Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:
$ printf "\353\n" | ...
- 358k
19
votes
Accepted
iconv illegal input sequence- why?
The file is encoded in ISO-8859-1, not in UTF-8:
$ hd 0606461.txt | grep -B1 '^0002c520'
0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin|
0002c520 e6 0d 0a 2d 2d 48 6f ...
- 11.3k
19
votes
Accepted
Can not use `cut -c` (`--characters`) with UTF-8?
You haven't said which cut you're using, but since you've mentioned the GNU long option --characters I'll assume it's that one. In that case, note this passage from info coreutils 'cut invocation':
‘-...
- 70.6k
18
votes
Accepted
Process a file that starts with a BOM (FF FE)
From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8:
iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt
- 144k
18
votes
Accepted
Converting a UTF-8 file to ASCII (best-effort)
konwert utf8-ascii
It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g.
...
- 1,763
18
votes
Accepted
Convert binary encoding that head and Notepad can read to UTF-8
"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job.
The file utility doesn't give useful information when it doesn't recognize the file format. It could be ...
- 71.1k
18
votes
Accepted
How can I get Japanese characters to show properly in Firefox on Arch?
It should be enough to install the great noto fonts bundles:
sudo pacman -S noto-fonts-cjk noto-fonts-emoji noto-fonts
The restart firefox and you should be abe to see them. Personally, I also ...
- 217k
17
votes
Accepted
Octals 302 240 together seem to correspond to non-breaking space
It's the UTF-8 encoding of the U+00A0 Unicode character:
$ unicode U+00A0
U+00A0 NO-BREAK SPACE
UTF-8: c2 a0 UTF-16BE: 00a0 Decimal:   Octal: \0240
Category: Zs (Separator, Space)
Bidi: CS ...
- 470k
17
votes
Accepted
How to get UTF8 from a hex variable?
With xxd (usually shipped with vim)
$ echo 5374c3a97068616e650a | xxd -p -r
Stéphane
If your locale's charset (see output of locale charmap) is not UTF-8 (but can represent all the characters in the ...
- 470k
16
votes
How can I correctly decompress a ZIP archive of files with Hebrew names?
I had success with the command 7z x <source.zip>.
Version:
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,[...])
Potentially relevant environment:
LANG=en_US.UTF-8
LC_ALL=...
- 161
16
votes
Accepted
Specify encoding with libreoffice --convert-to csv
Apparently LibreOffice tries to use ISO-8859-1 by default, which is causing the problem.
In response to this bug report, a new parameter --infilter has been added. The following command produces U+...
- 571
15
votes
Convert a .docx to a .pdf with pandoc
This still comes up on google searches so I wanted to put this on the record: pandoc could not read docx when this question was asked (the error comes from trying to read a binary file) but since ...
- 151
15
votes
Accepted
"mv" file with garbled name by inode number?
You could try:
find . -inum 12321475 -exec mv {} new-filename \;
or
find . -inum 12321475 -print0 | xargs -0 mv -t new-filename
Generally I prefer xargs over exec. Google for why. It's tricky ...
- 38.2k
15
votes
Wget returning binary instead of html?
This file is still compressed with gzip.
You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it.
mv file.dat file.dat.gz
gunzip file.dat....
- 90.3k
14
votes
Prevent tail from modifying the charset of the terminal
You can translate special characters (binary data)
into ordinary characters that are safe to display
by piping your tail command into cat -v:
tail --follow=name my-rolling-file.log | cat -v
The -v (...
- 9,774
Only top scored, non community-wiki answers of a minimum length are eligible