66
You have to take your codes into quotes:
echo -n -e '\x66\x6f\x6f' > byteFileForNow
cause otherwise shell replaces \x to x before it goes to echo -e.
ps. double escape will also work:
echo -n -e \\x66\\x6f\\x6f > byteFileForNow
64
When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on.
By default, the detection process is crude. Every file that you open ...
53
It is known as carriage return.
If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r.
Inserting 0x0D in a hex editor will do the task.
How to remove?
You can remove it using the command perl -p -i -e "s/\r//g" filename.
As the OP suggested in the comments of this answer here, you can even try a ...
34
If you want to use grep, you can do:
grep -axv '.*' file
in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).
34
It's your locale and tr problem.
Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird:
$ </dev/urandom LC_ALL=vi_VN.tcvn tr -dc '[:print:]' | head -c 64
`�pv���Z����c�ox"�O���%�YR��F�>��췔��ovȪ������^,<H ���>
The shell will print multi-byte characters correctly, but GNU ...
33
file tells you “Non-ISO extended-ASCII text” because it detects that this is:
most likely a “text” file from the lack of control characters (byte values 0–31) other than line breaks;
“extended-ASCII” because there are characters outside the ASCII range (byte values ≥128);
“non-ISO” because there are characters in the 128–159 range (ISO 8859 reserves this ...
answered Jan 8 '14 at 0:42
Gilles 'SO- stop being evil'
590k143 gold badges1212 silver badges1721 bronze badges
31
Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names:
LC_ALL=C find . -name '*[! -~]*'
(The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.)
The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...
29
grep is a text processing tool. It expects their input to be text files. It seems that the same goes for tr on macOS (even though tr is supposed to support binary files).
Computers store data as sequences of bytes. A text is a sequence of characters. There are several ways to encode characters as bytes, called character encodings. The de facto standard ...
answered Jul 9 '14 at 0:06
Gilles 'SO- stop being evil'
590k143 gold badges1212 silver badges1721 bronze badges
28
This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure out that the file is compressed as well):
file output.html
wget -O - … | file -
The server (I guessed it from the content you showed) is sending gzipped data ...
answered Nov 6 '16 at 23:58
Gilles 'SO- stop being evil'
590k143 gold badges1212 silver badges1721 bronze badges
27
This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). That isn't a useful character for any modern system, but it's unlikely to be harmful that it's there.
The original source for this was likely a byte 0x96 in ...
25
Short answer: restrictions imposed in Unix/Linux/BSD kernel, namei() function. Encoding takes place in user level programs like xterm, firefox or ls.
I think you're starting from incorrect premises. A file name in Unix is a string of bytes with arbitrary values. A few values, 0x0 (ASCII Nul) and 0x2f (ASCII '/') are just not allowed, not as part of a ...
25
This will work for some things:
iconv -f utf-8 -t ascii//TRANSLIT
echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.
iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. Also, the IGNORE option will ...
25
That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c):
b = fnmatch (str, base, flags) == 0;
(...)
return b;
This code tests the return value of fnmatch for equality with 0, but does not check for errors; this results in any ...
22
Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:
$ printf "\353\n" | iconv -f ISO-8859-1
ë
Other candidates would be δ in code page 437, Ù in code page 850...
od -x’s output is confusing in this case because of endianness; a ...
21
Look at this question.
Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in utf-8 encoding.
grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]"
and
grep '[अ-व]'
are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of devanagari script (that is, the matching is NOT "any ...
20
The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...
20
UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding.
To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just:
$ printf '\U1F615\n'
😕
$...
19
This fixed the problem for me. Install the dejavu fonts.
sudo pacman -S ttf-dejavu
19
There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00.
What POSIX does require is that all characters from its Portable Character Set exist. The Portable Character Set contains the printable ASCII characters, space, BEL, backspace, tab, carriage return, newline, vertical tab, form feed, ...
18
With GNU recode:
recode html < file
If you don't have recode or HTML::Entities and only need to decode &#x<hex>; entities, you could do it by hand with:
perl -Mopen=locale -pe 's/&#x([\da-f]+);/chr hex $1/gie'
17
The thing is, the kernel doesn't care one bit how the applications interpret the data it is given as a filename.
Let's imagine I have a C application that deals with exclusively UTF-16 strings. And I enter, via a properly configured input method, the ∯ symbol (Unicode 0x222F) into the "Save As" prompt/dialog.
If the application doesn't do any form of ...
17
Change the character translation in PuTTY to UTF-8.
17
Your shell can display accents etc because it is probably using UTF-8. Since the file in question is a different encoding, less more and cat are trying to read it as UTF and fail. You can check your current encoding with
echo $LANG
You have two choices, you can either change your default encoding, or change the file to UTF-8. To change your encoding, open ...
16
At a guess,
Your locale uses UTF-8 encoding, and
About 10% of your file consists of characters which require more than one octet to encode into UTF-8.
By the way, from man wc:
-c, --bytes
print the byte counts
-m, --chars
print the character counts
16
In the documentation here, .docx is not listed as a compatible input:
Pandoc is a Haskell library for converting from one markup format to
another, and a command-line tool that uses this library. It can read
markdown and (subsets of) Textile, reStructuredText, HTML, LaTeX,
MediaWiki markup, Haddock markup, OPML, and DocBook; and it can write
plain ...
16
When using vim -b, this displays all high characters as <xx>:
set encoding=latin1
set isprint=
set display+=uhex
Any single-byte encoding will work, vim uses ASCII for all lower chars and has them hard-coded as printable. Setting isprint to empty will mark everything else as non-printable. Setting uhex will display them as hexadecimal.
Here is how ...
16
The file is encoded in ISO-8859-1, not in UTF-8:
$ hd 0606461.txt | grep -B1 '^0002c520'
0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin|
0002c520 e6 0d 0a 2d 2d 48 6f 72 61 63 65 2e 0d 0a 0d 0a |...--Horace.....|
And the byte "e6" alone is not a valid UTF-8 sequence.
So, use iconv -f latin1 -t ascii//TRANSLIT file.
15
It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc).
Is there another decompression utility that will decompress my files with the correct names? I'm not aware of a zip utility that supports these code pages natively. 7z has some understanding of encodings, but I believe it has to be an encoding your system ...
15
It's the UTF-8 encoding of the U+00A0 Unicode character:
$ unicode U+00A0
U+00A0 NO-BREAK SPACE
UTF-8: c2 a0 UTF-16BE: 00a0 Decimal:   Octal: \0240
Category: Zs (Separator, Space)
Bidi: CS (Common Number Separator)
Decomposition: <noBreak> 0020
$ locale charmap
UTF-8
$ printf '\ua0' | od -to1
0000000 302 240
0000002
UTF-8 is an encoding ...
answered Mar 25 '16 at 17:28
Stéphane Chazelas
345k59 gold badges674 silver badges1052 bronze badges
15
This file is still compressed with gzip.
You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it.
mv file.dat file.dat.gz
gunzip file.dat.gz
Or
zcat file.dat
Only top voted, non community-wiki answers of a minimum length are eligible
