86
You have to take your codes into quotes:
echo -n -e '\x66\x6f\x6f' > byteFileForNow
cause otherwise shell replaces \x to x before it goes to echo -e.
ps. double escape will also work:
echo -n -e \\x66\\x6f\\x6f > byteFileForNow
66
It is known as carriage return.
If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r.
Inserting 0x0D in a hex editor will do the task.
How do I remove it?
You can remove it using the command
perl -p -i -e "s/\r//g" filename
As the OP suggested in the comments of this answer here, you can even ...
60
I can reproduce it with the xterm terminal emulator (version 366), if I do:
$ printf '\e[?42h\e(H'; cat chars.txt; printf '\e(B\e[?42l'
!É#$%Ü&*()_+äå
Where:
\e[?42h. Enables National Replacement Character sets
\e(H selects a Swedish charset for the G0 set (CP1106).
The other ones undo the settings.
To restore the terminal to a sane state, you can ...
45
If you want to use grep, you can do:
grep -axv '.*' file
in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).
41
Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names:
LC_ALL=C find . -name '*[! -~]*'
(The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.)
The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...
40
file tells you “Non-ISO extended-ASCII text” because it detects that this is:
most likely a “text” file from the lack of control characters (byte values 0–31) other than line breaks;
“extended-ASCII” because there are characters outside the ASCII range (byte values ≥128);
“non-ISO” because there are characters in the 128–159 range (ISO 8859 reserves this ...
answered Jan 8 '14 at 0:42
Gilles 'SO- stop being evil'
703k170 gold badges1472 silver badges1963 bronze badges
39
grep is a text processing tool. It expects their input to be text files. It seems that the same goes for tr on macOS (even though tr is supposed to support binary files).
Computers store data as sequences of bytes. A text is a sequence of characters. There are several ways to encode characters as bytes, called character encodings. The de facto standard ...
answered Jul 9 '14 at 0:06
Gilles 'SO- stop being evil'
703k170 gold badges1472 silver badges1963 bronze badges
39
This will work for some things:
iconv -f utf-8 -t ascii//TRANSLIT
echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.
iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. Also, the IGNORE option will ...
36
It's your locale and tr problem.
Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird:
$ </dev/urandom LC_ALL=vi_VN.tcvn tr -dc '[:print:]' | head -c 64
`�pv���Z����c�ox"�O���%�YR��F�>��췔��ovȪ������^,<H ���>
The shell will print multi-byte characters correctly, but GNU ...
29
This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure out that the file is compressed as well):
file output.html
wget -O - … | file -
The server (I guessed it from the content you showed) is sending gzipped data ...
answered Nov 6 '16 at 23:58
Gilles 'SO- stop being evil'
703k170 gold badges1472 silver badges1963 bronze badges
27
This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). That isn't a useful character for any modern system, but it's unlikely to be harmful that it's there.
The original source for this was likely a byte 0x96 in ...
26
That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c):
b = fnmatch (str, base, flags) == 0;
(...)
return b;
This code tests the return value of fnmatch for equality with 0, but does not check for errors; this results in any ...
24
Your shell can display accents etc because it is probably using UTF-8. Since the file in question is a different encoding, less more and cat are trying to read it as UTF and fail. You can check your current encoding with
echo $LANG
You have two choices, you can either change your default encoding, or change the file to UTF-8. To change your encoding, open ...
23
This fixed the problem for me. Install the dejavu fonts.
sudo pacman -S ttf-dejavu
23
With Free recode (formerly known as GNU recode):
recode html < file
If you don't have recode or HTML::Entities and only need to decode &#x<hex>; entities, you could do it by hand with:
perl -Mopen=locale -pe 's/&#x([\da-f]+);/chr hex $1/gie'
22
When using vim -b, this displays all high characters as <xx>:
set encoding=latin1
set isprint=
set display+=uhex
Any single-byte encoding will work, vim uses ASCII for all lower chars and has them hard-coded as printable. Setting isprint to empty will mark everything else as non-printable. Setting uhex will display them as hexadecimal.
Here is how ...
22
UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding.
To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just:
$ printf '\U1F615\n'
😕
$...
22
Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:
$ printf "\353\n" | iconv -f ISO-8859-1
ë
Other candidates would be δ in code page 437, Ù in code page 850...
od -x’s output is confusing in this case because of endianness; a ...
21
It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc).
Is there another decompression utility that will decompress my files with the correct names? I'm not aware of a zip utility that supports these code pages natively. 7z has some understanding of encodings, but I believe it has to be an encoding your system ...
20
There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00.
What POSIX does require is that all characters from its Portable Character Set exist. The Portable Character Set contains the printable ASCII characters, space, BEL, backspace, tab, carriage return, newline, vertical tab, form feed, ...
19
The file is encoded in ISO-8859-1, not in UTF-8:
$ hd 0606461.txt | grep -B1 '^0002c520'
0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin|
0002c520 e6 0d 0a 2d 2d 48 6f 72 61 63 65 2e 0d 0a 0d 0a |...--Horace.....|
And the byte "e6" alone is not a valid UTF-8 sequence.
So, use iconv -f latin1 -t ascii//TRANSLIT file.
18
From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8:
iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt
18
I suppose that your charmap from the locales is UTF-8, so that you'll have problems on binary files. Just switch to C locale:
LC_ALL=C tr '\r' '\n' < target-file | LC_ALL=C grep search-string
18
You haven't said which cut you're using, but since you've mentioned the GNU long option --characters I'll assume it's that one. In that case, note this passage from info coreutils 'cut invocation':
‘-c character-list’
‘--characters=character-list’
Select for printing only the characters in positions listed in character-list. The same as -b for now, ...
17
In the documentation here, .docx is not listed as a compatible input:
Pandoc is a Haskell library for converting from one markup format to
another, and a command-line tool that uses this library. It can read
markdown and (subsets of) Textile, reStructuredText, HTML, LaTeX,
MediaWiki markup, Haddock markup, OPML, and DocBook; and it can write
plain ...
17
konwert utf8-ascii
It will do best-effort conversion, depending on the conversion tables. If you know approximately the input language, there are language specific filters giving better results, e.g.
konwert utf8-xmetodo
is the conversion of Esperanto into the x-metodo representation,
konwert UTF8-tex
will try to do TeX representation of diacritics, ...
17
It's the UTF-8 encoding of the U+00A0 Unicode character:
$ unicode U+00A0
U+00A0 NO-BREAK SPACE
UTF-8: c2 a0 UTF-16BE: 00a0 Decimal:   Octal: \0240
Category: Zs (Separator, Space)
Bidi: CS (Common Number Separator)
Decomposition: <noBreak> 0020
$ locale charmap
UTF-8
$ printf '\ua0' | od -to1
0000000 302 240
0000002
UTF-8 is an encoding ...
answered Mar 25 '16 at 17:28
Stéphane Chazelas
429k76 gold badges846 silver badges1275 bronze badges
17
"binary" isn't an encoding (character-set name). iconv needs an encoding name to do its job.
The file utility doesn't give useful information when it doesn't recognize the file format. It could be UTF-16 for example, without a byte-encoding-mark (BOM). notepad reads that. The same applies to UTF-8 (and head would display that since your terminal may be ...
17
It should be enough to install the great noto fonts bundles:
sudo pacman -S noto-fonts-cjk noto-fonts-emoji noto-fonts
The restart firefox and you should be abe to see them. Personally, I also installed the following from AUR:
yaourt -S ttf-freefont ttf-ms-fonts ttf-linux-libertine ttf-dejavu ttf-inconsolata ttf-ubuntu-font-family
I doubt those will help ...
15
You could try:
find . -inum 12321475 -exec mv {} new-filename \;
or
find . -inum 12321475 -print0 | xargs -0 mv -t new-filename
Generally I prefer xargs over exec. Google for why. It's tricky though. See Find -exec + vs find | xargs. Which one to choose?
Only top voted, non community-wiki answers of a minimum length are eligible
