79
You have to take your codes into quotes:
echo -n -e '\x66\x6f\x6f' > byteFileForNow
cause otherwise shell replaces \x to x before it goes to echo -e.
ps. double escape will also work:
echo -n -e \\x66\\x6f\\x6f > byteFileForNow
59
It is known as carriage return.
If you're using vim you can enter insert mode and type CTRL-v CTRL-m. That ^M is the keyboard equivalent to \r.
Inserting 0x0D in a hex editor will do the task.
How do I remove it?
You can remove it using the command
perl -p -i -e "s/\r//g" filename
As the OP suggested in the comments of this answer here, you can even ...
39
If you want to use grep, you can do:
grep -axv '.*' file
in UTF-8 locales to get the lines that have at least an invalid UTF-8 sequence (this works with GNU Grep at least).
38
file tells you “Non-ISO extended-ASCII text” because it detects that this is:
most likely a “text” file from the lack of control characters (byte values 0–31) other than line breaks;
“extended-ASCII” because there are characters outside the ASCII range (byte values ≥128);
“non-ISO” because there are characters in the 128–159 range (ISO 8859 reserves this ...
answered Jan 8 '14 at 0:42
Gilles 'SO- stop being evil'
649k157 gold badges1346 silver badges1855 bronze badges
37
Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names:
LC_ALL=C find . -name '*[! -~]*'
(The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.)
The hint for LC_ALL=C is required (actually, LC_CTYPE=C and ...
35
grep is a text processing tool. It expects their input to be text files. It seems that the same goes for tr on macOS (even though tr is supposed to support binary files).
Computers store data as sequences of bytes. A text is a sequence of characters. There are several ways to encode characters as bytes, called character encodings. The de facto standard ...
answered Jul 9 '14 at 0:06
Gilles 'SO- stop being evil'
649k157 gold badges1346 silver badges1855 bronze badges
35
It's your locale and tr problem.
Currently, GNU tr fully supports only single-byte characters. So in locales using multibyte encodings, the output can be weird:
$ </dev/urandom LC_ALL=vi_VN.tcvn tr -dc '[:print:]' | head -c 64
`�pv���Z����c�ox"�O���%�YR��F�>��췔��ovȪ������^,<H ���>
The shell will print multi-byte characters correctly, but GNU ...
33
This will work for some things:
iconv -f utf-8 -t ascii//TRANSLIT
echo ĥéĺłœ π | iconv -f utf-8 -t ascii//TRANSLIT returns helloe ?. Any characters that iconv doesn’t know how to convert will be replaced with question marks.
iconv is POSIX, but I don’t know if all systems have the TRANSLIT option. It works for me on Linux. Also, the IGNORE option will ...
29
This is a gzip compressed file. You can find this out by running the file command, which figures out the file format from magic numbers in the data (this is how programs such as Text Wrangler figure out that the file is compressed as well):
file output.html
wget -O - … | file -
The server (I guessed it from the content you showed) is sending gzipped data ...
answered Nov 6 '16 at 23:58
Gilles 'SO- stop being evil'
649k157 gold badges1346 silver badges1855 bronze badges
27
This file contains bytes C2 96, which are the UTF-8 encoding of codepoint U+0096. That codepoint is one of the C1 control characters commonly called SPA "Start of Guarded Area" (or "Protected Area"). That isn't a useful character for any modern system, but it's unlikely to be harmful that it's there.
The original source for this was likely a byte 0x96 in ...
26
That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c):
b = fnmatch (str, base, flags) == 0;
(...)
return b;
This code tests the return value of fnmatch for equality with 0, but does not check for errors; this results in any ...
23
Look at grep: Find all lines that contain Japanese kanjis.
Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in UTF-8 encoding.
grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]"
and
grep '[अ-व]'
are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of Devanagari script ...
22
Your file contains two bytes, EB and 0A in hex. It’s likely that the file is using a character set with one byte per character, such as ISO-8859-1; in that character set, EB is ë:
$ printf "\353\n" | iconv -f ISO-8859-1
ë
Other candidates would be δ in code page 437, Ù in code page 850...
od -x’s output is confusing in this case because of endianness; a ...
21
This fixed the problem for me. Install the dejavu fonts.
sudo pacman -S ttf-dejavu
21
The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all ...
21
Your shell can display accents etc because it is probably using UTF-8. Since the file in question is a different encoding, less more and cat are trying to read it as UTF and fail. You can check your current encoding with
echo $LANG
You have two choices, you can either change your default encoding, or change the file to UTF-8. To change your encoding, open ...
21
With Free recode (formerly known as GNU recode):
recode html < file
If you don't have recode or HTML::Entities and only need to decode &#x<hex>; entities, you could do it by hand with:
perl -Mopen=locale -pe 's/&#x([\da-f]+);/chr hex $1/gie'
21
UTF-8 is a variable length encoding of Unicode. It is designed to be superset of ASCII. See Wikipedia for details of the encoding. \x00 \x01 \xF6 \x15 would be UCS-4BE or UTF-32BE encoding.
To get from the Unicode code point to the UTF-8 encoding, assuming the locale's charmap is UTF-8 (see the output of locale charmap), it's just:
$ printf '\U1F615\n'
😕
$...
20
When using vim -b, this displays all high characters as <xx>:
set encoding=latin1
set isprint=
set display+=uhex
Any single-byte encoding will work, vim uses ASCII for all lower chars and has them hard-coded as printable. Setting isprint to empty will mark everything else as non-printable. Setting uhex will display them as hexadecimal.
Here is how ...
20
It sounds like the filenames are encoded in one of Windows' proprietary codepages (CP862, 1255, etc).
Is there another decompression utility that will decompress my files with the correct names? I'm not aware of a zip utility that supports these code pages natively. 7z has some understanding of encodings, but I believe it has to be an encoding your system ...
20
There is no specific character encoding mandated by POSIX. The only character in a fixed position is null, which must be 00.
What POSIX does require is that all characters from its Portable Character Set exist. The Portable Character Set contains the printable ASCII characters, space, BEL, backspace, tab, carriage return, newline, vertical tab, form feed, ...
19
The file is encoded in ISO-8859-1, not in UTF-8:
$ hd 0606461.txt | grep -B1 '^0002c520'
0002c510 64 75 6d 20 66 65 72 69 65 6e 74 20 72 75 69 6e |dum ferient ruin|
0002c520 e6 0d 0a 2d 2d 48 6f 72 61 63 65 2e 0d 0a 0d 0a |...--Horace.....|
And the byte "e6" alone is not a valid UTF-8 sequence.
So, use iconv -f latin1 -t ascii//TRANSLIT file.
17
Change the character translation in PuTTY to UTF-8.
17
In the documentation here, .docx is not listed as a compatible input:
Pandoc is a Haskell library for converting from one markup format to
another, and a command-line tool that uses this library. It can read
markdown and (subsets of) Textile, reStructuredText, HTML, LaTeX,
MediaWiki markup, Haddock markup, OPML, and DocBook; and it can write
plain ...
17
From this wikipedia article, FF FE means UTF16LE. So you should tell iconv to convert from UTF16LE to UTF8:
iconv -f UTF-16LE -t UTF-8 dotan.csv > fixed.txt
16
At a guess,
Your locale uses UTF-8 encoding, and
About 10% of your file consists of characters which require more than one octet to encode into UTF-8.
By the way, from man wc:
-c, --bytes
print the byte counts
-m, --chars
print the character counts
16
It's the UTF-8 encoding of the U+00A0 Unicode character:
$ unicode U+00A0
U+00A0 NO-BREAK SPACE
UTF-8: c2 a0 UTF-16BE: 00a0 Decimal:   Octal: \0240
Category: Zs (Separator, Space)
Bidi: CS (Common Number Separator)
Decomposition: <noBreak> 0020
$ locale charmap
UTF-8
$ printf '\ua0' | od -to1
0000000 302 240
0000002
UTF-8 is an encoding ...
answered Mar 25 '16 at 17:28
Stéphane Chazelas
388k70 gold badges765 silver badges1175 bronze badges
15
You can't directly print the ascii codes by using the printf "%c" $i like in C.
You have to first convert the decimal value of i into its octal value and then you have to print it using using printf and putting \ in front of their respective octal values.
To print A, you have to convert the decimal 65 into octal, i.e. 101, and then you have to print that ...
15
After snooping around various forums and some Stack-overflow post I found the issue.
Here is an explanation to better understand and 'see' the root cause of the errors.
First off if you are writing code on Windows, better arm yourself with
Notepad++. It's basically the Swiss Army knife of text-editors in my
opinion.
Now to have a look at the EOL (...
15
This file is still compressed with gzip.
You can see that the first two bytes 0x1f8b match the gzip signature. So to read the data you need to uncompress it.
mv file.dat file.dat.gz
gunzip file.dat.gz
Or
zcat file.dat
Only top voted, non community-wiki answers of a minimum length are eligible
