if grep -q "�" out.txt
    then
        echo "working"
    else
        cat out.txt
fi

Basically, if the file "out.txt" contains "�" anywhere in the file I would like it to echo "working" AND if the file "out.txt" does NOT contain "�" anywhere in the file then I would like it to cat out.txt

EDIT: So here's what I'm doing. I'm trying to brute force an openssl decrypt.

openssl enc returns 0 on success, non-zero otherwise. Note: you will get false positives because AES/CBC can only determine if "decryption works" based on getting the padding right. So the file decrypts but it will not be the correct password so it will have gibberish in it. A common character in the gibberish is "�". So I want the do loop to keep going if the output contains "�".

Heres my git link https://github.com/Raphaeangelo/OpenSSLCracker Heres the script

while read line
do
openssl aes-256-cbc -d -a -in $1 -pass pass:$line -out out.txt 2>out.txt >/dev/null && printf "==================================================\n"
if grep -q "�" out.txt
    then
        :
    else
        cat out.txt && printf "\n==================================================" && printf "\npassword is $line\n" && read -p "press return key to continue..." < /dev/tty; 
fi
done < ./password.txt

its still showing me output with the � charicter in it

UPDATE: Solved

printf "Working..."

while read line
do
openssl aes-256-cbc -d -a -in $1 -pass pass:$line -out out.txt 2>out.txt >/dev/null
if file out.txt | grep -q 'out.txt: ASCII text'
    then
        printf "\n==================================================\n\n" && cat out.txt && printf "\n==================================================" && printf "\npassword is $line\n" && read -p "press return key to continue..." < /dev/tty;
    else
        : 
fi
done < ./password.txt
  • It looks correct, it should work (btw, I have no font for your unicode character to see, but none of them has any special meaning). grep long understands unicode (which makes it much slower, so to search for ascii strings, a LANG=C grep is a huge performance improvement). – peterh Oct 11 at 2:16
  • I might have to delete this and post another question because I'm sure I'm completely confusing everyone here. – Stuart Sloan Oct 11 at 2:43
  • @Stuart Sloan the title of your question is How to grep for unicode � in a bash script is this really what you want? to extract the unicode? please clarify so we can help! – Goro Oct 11 at 2:47
  • @Goro I've made the edit to my original post. I hope it makes sense. Please let me know if it doesn't and I will try an clarify. – Stuart Sloan Oct 11 at 2:55
  • @Goro um.... what? – Stuart Sloan Oct 11 at 3:09
up vote 24 down vote accepted

grep is the wrong tool for the job.

You see the � U+FFFD REPLACEMENT CHARACTER not because it’s literally in the file content, but because you looked at a binary file with a tool that is supposed to handle only text-based input. The standard way to handle invalid input (i.e., random binary data) is to replace everything that is not valid in the current locale (most probably UTF-8) with U+FFFD before it hits the screen.

That means it is very likely that a literal \xEF\xBF\xBD (the UTF-8 byte sequence for the U+FFFD character) never occurs in the file. grep is completely right in telling you, there is none.

One way to detect whether a file contains some unknown binary is with the file(1) command:

$ head -c 100 /dev/urandom > rubbish.bin
$ file rubbish.bin
rubbish.bin: data

For any unknown file type it will simply say data. Try

$ file out.txt | grep '^out.txt: data$'

to check whether the file really contains any arbitrary binary and thus most likely rubbish.

If you want to make sure that out.txt is a UTF-8 encoded text file only, you can alternatively use iconv:

$ iconv -f utf-8 -t utf-16 out.txt >/dev/null
  • You are exactly right! unfortunately, I'm still getting some (less than before) rubbish in the output. – Stuart Sloan Oct 11 at 7:36
  • Possibly file detects some other content type for those files. If you 100% always only expect UTF-8 encoded text files, you can check with iconv, if a file is valid UTF-8: iconv -f utf-8 -t utf-16 out.txt >/dev/null. If iconv cannot convert the file due to invalid UTF-8 sequences, it will return with a non-zero exit code. – Boldewyn Oct 11 at 7:44
  • 1
    The file command was right! You helped me solve my issue thanks! – Stuart Sloan Oct 11 at 8:03
  • 2
    Of course that grep "is the tool for the job", try grep -axv '.*' badchars.txt. That will print any line that contains any invalid Unicode Character. – Isaac Oct 11 at 11:44
  • iconv -c ... will omit invalid input characters silently if you don’t care about them. You can omit the output encoding if you want to use the current locale’s encoding or specify something that makes more sense than UTF-16 in your context. – David Foerster Oct 11 at 14:34

This very early answer was for the original post which was:

How to grep for unicode � in a bash script

if grep -q "�" out.txt
    then
        echo "working"
    else
        cat out.txt  fi

Basically, if the file "out.txt" contains "�" anywhere in the file I would like it to echo "working" AND if the file "out.txt" does NOT contain "�" anywhere in the file then I would like it to cat out.txt

Try

grep -oP "[^\x00-\x7F]"

with an if .. then statement as follows:

if grep -oP "[^\x00-\x7F]" file.txt; then
    echo "grep found something ..."
else
    echo "Nothing found!"
fi

Explanation💡:

  • -P, --perl-regexp: PATTERN is a Perl regular expression
  • -o, --only-matching: show only the part of a line matching PATTERN
  • [^\x00-\x7F] is a regex to match a single non-ASCII character.
  • [[:ascii:]] - matches a single ASCII char
  • [^[:ascii:]] - matches a single non-ASCII char

in bash

LC_COLLATE=C grep -o '[^ -~]' file
  • 3
    This will break (have a false positive) as soon as someone doesn't speak English... – Kevin Oct 11 at 5:18
  • or if someone tries discussing à la carte, emoji, Pokémon, or anything else not strictly confined to 7bit ASCII. Better look for anything in 00-1F, except 09 0A 0D (tab, linefeed, carriage return). – Alcaro Oct 11 at 10:48
  • This is a very ver bad idea. This will reject any valid Unicode char above the ASCII range, just only a little more than a million valid characters. Amazing. Try: printf '%b' "$(printf '\\U%x' {128..131})" | grep -oP "[^\x00-\x7F]" Just 4 valid Unicode characters that your code reject. :-( – Isaac Oct 11 at 11:49
  • This is an extremely misleading answer. Please read in my answer why the simplistic approach of limiting only to ASCII fails grossly. – Isaac Oct 14 at 1:40

Both present answers are extremely misleading and basically wrong.

To test, Get this two files (from a very well regarded developer: Markus Kuhn ):

$ wget https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt
$ wget https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

Demo

The first UTF-8-demo.txt is a file designed to show how well UTF-8 is able to present many languages, math, braille and many other useful types of characters. Take a look with a text editor (that understand utf-8) and you will see a lot of examples and no .

The test that one answer propose: to limit the character range to \x00-\x7F will reject almost everything inside this file.
That is very wrong and will not remove any as there is none in that file.

Using the test recommended in that answer will remove 72.5 % of the file:

$ grep -oP "[^\x00-\x7F]" UTF-8-demo.txt | tr -d '\n' | wc -c
10192
$ cat UTF-8-demo.txt | wc -c
14058

That is (for most practical purposes) the whole file. A file very well designed to show perfectly valid characters.

Test

The second file is designed to try several border cases to confirm that utf-8 readers are doing a good job. It contains inside many characters that will cause a '�' to be shown. But the other answer recommendation (the selected one) to use file fails grossly with this file. Only removing a zero byte (\0) (which technically is valid ASCII) and a \x7f byte (DEL - delete) (which is clearly an ASCII character as well) will make all the file valid for the file command:

$ cat UTF-8-test.txt | tr -d '\0\177' > a.txt
$ file a.txt 
a.txt: Non-ISO extended-ASCII text, with LF, NEL line terminators

Not only does file fail to detect the many incorrect characters, but also fail to detect and report that it is an UTF-8 encoded file.

And yes, file is able to detect and report UTF-8 encoded text:

$ echo "ééakjfhhjhfakjfhfhaéá" | file -
/dev/stdin: UTF-8 Unicode text

Also, file fails to report as ASCII most of the control characters in the range 1 to 31. It (file) reports some ranges as data:

$ printf '%b' "$(printf '\\U%x' {1..6})" | file -
/dev/stdin: data

Others as ASCII text:

$ printf '%b' "$(printf '\\U%x' 7 {9..12})" | file -
/dev/stdin: ASCII text

As the printable character range (with newlines):

$ printf '%b' "$(printf '\\U%x' {32..126} 10)" | file -
/dev/stdin: ASCII text

But some ranges may cause weird results:

$ printf '%b' "$(printf '\\U%x' {14..26})" | file -
/dev/stdin: Atari MSA archive data, 4113 sectors per track, starting track: 5141, ending track: 5655

The program file is not a tool to detect text, but to detect magic numbers in executable programs or files.

The ranges file detect, and the corresponding type reported I found were:

  • One byte values, mostly ascii:

    {1..6} {14..26} {28..31} 127   :data
    {128..132} {134..159}          :Non-ISO extended-ASCII text
    133                            :ASCII text, with LF, NEL line terminators
    27                             :ASCII text, with escape sequences
    13                             :ASCII text, with CR, LF line terminators
    8                              :ASCII text, with overstriking
    7 {9..12} {32..126}            :ASCII text
    {160..255}                     :ISO-8859 text
    
  • Utf-8 encoded ranges:

    {1..6} {14..26} {28..31} 127   :data
    27                             :ASCII text, with escape sequences
    13                             :ASCII text, with CR, LF line terminators
    8                              :ASCII text, with overstriking
    7 {9..12} {32..126}            :ASCII text
    {128..132} {134..159}          :UTF-8 Unicode text
    133                            :UTF-8 Unicode text, with LF, NEL line terminators
    {160..255}                     :UTF-8 Unicode text
    {256..5120}                    :UTF-8 Unicode text
    

One possible solution lies below.


Previous Answer.

The Unicode value for the character you are posting is:

$ printf '%x\n' "'�"
fffd

Yes, that is a Unicode Character 'REPLACEMENT CHARACTER' (U+FFFD). That is a character used to replace any invalid Unicode character found in the text. It is a "visual aid", not a real character. To find and list every full line that contains invalid UNICODE characters use:

grep -axv '.*' out.txt 

but if you only want to detect if any character is invalid, use:

grep -qaxv '.*' out.txt; echo $?

If the result is 1 the file is clean, otherwise will be zero 0.


If what you were asking was: how to find the character, then, use this:

➤ a='Basically, if the file "out.txt" contains "�" anywhere in the file I'
➤ echo "$a" | grep -oP $(printf %b \\Ufffd)
�

Or if your system process correctly UTF-8 text, simply:

➤ echo "$a" | grep -oP '�'
�

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.