7

I have a file, I want to determine if it contains only English ASCII characters.

I thought this is a very common question, but when I googled it, there is no direct answer or related.

According to some other questions, this is what I got so far, but this doesn't work. No matter ASCII or Non-ASCII chars, it will always determine as non-ascii.

if (LC_ALL=C; [[ $TEXT = *[[:ascii:]]* ]]) then
  echo "Contain Non-ASCII"
fi

BTW, This is a shell script.

9

Your code tests whether the value of the variable TEXT contains at least one ASCII character. If you want to test whether $TEXT contains non-ASCII characters, you need to invert the set. By the way, [:ascii:] is locale-independent, so you don't need to set LC_CTYPE.

if [[ $TEXT = *[![:ascii:]]* ]]; then
  echo "Contain Non-ASCII"
fi

Note that [:ascii:] is a bash (and zsh) feature. If you want your script to work in ksh or plain sh, you will need to make use of the locale. Here's a portable method:

LC_CTYPE=C
case $TEXT in
  *[![:cntrl:][:print:]]*) echo "Contain Non-ASCII";;
esac

If you want to check the file contents, use grep.

if LC_ALL=C grep -q '[^[:cntrl:][:print:]]' somefile; then
  echo "Contain Non-ASCII"
fi
5
  • 1
    Fine! How to tell where it is?
    – Notinlist
    Apr 7 '18 at 14:58
  • yeah, how can i print the non-ascii char that was found ? Apr 26 '18 at 10:13
  • @DavidGidony Remove the -q option from the grep call so that it prints the offending lines, maybe add -n to print the line number. Apr 26 '18 at 10:32
  • The example with grep doesn't work for me. It always says Contains Non-ASCII. Where as the following command shows me that it has Non-ASCII characters: grep --color='auto' -P -n "[\x80-\xFF]" somefile or LANG=c sed 's,[\x00-\x7F],,g' somefile > somefile_without_ascii creates a new non-zero byte file. Also this (from yourself about 5 years prior to this answer) stackoverflow.com/a/3208902/319542 works! :) Jun 24 at 16:17
  • 1
    @PraveenLobo I fixed a mistake in the grep command. Jun 24 at 16:35
4

The file command can tell you the type of a file (ASCII, unicode, binary, etc.).

$ file my_file.txt
my_file.txt: ASCII text

So you can just check to see that the output contains the word "ASCII" and this should work:

if [[ file my_file.txt | grep -i ascii ]] ...
1
  • 2
    FYI - Caution grepping for "ASCII" alone. Other things than plain ascii may come from "file". For instance: Non-ISO extended-ASCII text
    – BRebey
    Nov 1 '19 at 17:44
-1
TEXT=SOMETEXTHERE

RESULT=$(echo $TEXT | LC_COLLATE=C grep -r '[^ -~]')
if [ -z "$RESULT" ]; then
    echo "ALL ASCII"
else
    echo "Contain Non-ASCII"
fi

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.