Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 1 down vote favorite

I need to strip non-ASCII characters off a file. I was using the command -

/usr/xpg4/bin/tr -cd '\0-\177' <non-ASCII_file.dat >ASCII_file.dat

Though it worked in the past, it is not working now. It is now stripping off all the alphabet along with non-ASCII characters. The only characters being left in the ASCII file are numbers and special characters in the ASCII set.

OS is Solaris 9.

What could be causing this issue? Is the Octal range being interpreted incorrectly? Something to do with my environment?

share|improve this question
How about strings <non-ASCII_file.dat >ASCII_file.txt – Johan Mar 19 at 10:27
1  
Does it make a difference if you do LC_ALL=C /usr/xpg4/bin/tr -cd '\0-\177'? – Stephane Chazelas Mar 19 at 11:12
If you have Ruby available stackoverflow.com/q/1268289/789593 might help. – N.N. Mar 19 at 11:30
@StephaneChazelas That did the trick. What is LC_ALL and what does setting it to 'C' signify and how does it affect tr? And could you please elaborate this in the answer section? – Sri M. Mar 19 at 12:33

1 Answer

up vote 3 down vote accepted

Behaviour of character ranges depend on the locale, that is the internationalisation settings. Different locales have different order for characters. For instance in a French locale (and most locales where there is a â character), â will be after a and before b.

The C locale is one that is not language specific (or US English specific when it has to make a choice), in that locale, characters are bytes and they sort by their byte value.

The locales area that tr is concerned about are LC_CTYPE to define the type of character, and LC_COLLATE to define the order of characters. Note that nowadays the characters have variable number of bytes as utf-8 is becoming more and more common as the default character set.

Those can be specified using environment variables of the same name. LC_ALL however overrides them all. So to be sure to get the behavior you want, you have to either unset LC_ALL and set the ones you like or simpler, just set LC_ALL:

LC_ALL=C tr -cd '\0-\177'

Or:

LC_ALL=C tr -d '\200-\377'

That also works for utf-8 data because utf-8 is a superset of ASCII and all the non-ASCII characters have the eighth bit set in all their bytes.

share|improve this answer
Good point about UTF-8. Which raises the question, "Why does anyone do anything other than UTF-8?" – Bruce Ediger Mar 19 at 14:25
@BruceEdiger Legacy, and binary files. – Gilles Mar 19 at 22:50
@BruceEdiger: History for one, not to mention other uses as e.g. SMS where one can cram a lot more then 140 characters in 140 8 bit bytes (octets). It depends on use. If you have an application that are only going to process ASCII - adding support for UTF would mean raising the complexity of the code by N levels. Especially with lower languages as C. It's void to e.g. say your application support UTF-8, but only for the 127 first code points. – Sukminder Mar 20 at 0:25
I should have written a clearer question: "Why does anyone do any other encoding of Unicode other than UTF-8?" Living as I do in the middle of North America, I write personal use programs for ASCII only, so I understand the history and "other uses" aspects. – Bruce Ediger Mar 20 at 2:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.