1

I am using the SED command to strip out foreign language and other non-keyboard characters from very large text files:

example:

sed 's/[^a-zA-Z0-9]//g'

The above command keeps any line that contains only alpha-numeric characters, which is close to what I want. The problem is that it also strips away any lines that contain common symbols like !@#$% and so on. I want to keep those. I tried searching for a bullion command something like !-), or something. But I could not find anything like that.

So how do I filter out Arabic an Russian and untype-able characters from my lists? (ideally, I don't want to just nuke the character, I want to nuke the whole line where its found.)

3
  • 1
    To remove lines, you should use grep -v instead of sed.
    – pLumo
    Commented Jan 28, 2021 at 7:46
  • Foreign to whom? What's your script, keyboard layout or language? English? Which one? US English, London English, Scottish English. Are $, £, "foreign" to you for instance? Commented Jan 28, 2021 at 9:25
  • Also note that your [a-z] generally matches on characters such as ɘəɚɛɜɝɞɟɠɡɢɣɤɥɦɧɨɩɪɫɬɭɮɯɰɱ (and hundreds more) which many English speakers would consider "foreign". Commented Jan 28, 2021 at 10:42

2 Answers 2

1

Apart from using classes as Kusalananda does, you might also create your own range based on unicode. Check this reference unicolde table for finding the characters you like. With PCRE a possible way of "standard" characters + TAB would be:

 grep -P '^[\x{0020}-\x{007e}\x{0008}]{1,}$' file

Note how newline \x{000A} is not included as control character, due to grep's per-line functionality (in standard mode). Please consider that MS-style newlines would be affected and use \x{000d}\x{000a} for newlines!

6
  • Both CR and TAB are already in the \x{0000}-\x{007e} range. Why are you excluding \x7F and not the other ASCII control characters? Also note that it removes empty lines. Commented Jan 28, 2021 at 9:40
  • You may want to clarify what you mean by "standard" here. Unicode is a "standard" and covers all characters in existence. A "standard" charset limited to US English and many western Europe countries is ISO8859-15 (would include é already included by the OP's [a-z] for instance). ASCII is an old (from the 60s possibly earlier, initially 6bits, now 7bits) US-only charset but misses several of the characters now commonly used even in the US. Commented Jan 28, 2021 at 9:45
  • @StéphaneChazelas I specifically did not include this range, as e.g. AKN or BEL are not printable, a TAB or newline however clearly visible in a text, despite being control characters. "standard" was meant to be as obscure as OP's (lack of) definition - hence the quotes and the referece to the table with a mere example range in the code. I expected OP to adapt.
    – FelixJN
    Commented Jan 28, 2021 at 9:55
  • @StéphaneChazelas Now I saw my error: The range was meant to start at \x{0020} not \x{0000}- corrected.
    – FelixJN
    Commented Jan 28, 2021 at 9:57
  • 1
    FF, VT, CR, BS for instance are about as "visible" as LF/TAB in text. When sent to a terminal, they move the cursor position as well. And like for TAB, the behaviour varies depending on the output device. Commented Jan 28, 2021 at 10:24
1

To remove non-ASCII characters from a text, consider using tr like so:

LC_ALL=C tr -d -c '[:print:][:cntrl:]' <file.in >file.out

The two POSIX character classes [:print:] and [:cntrl:] together span all characters in the ASCII range, and with -c we ask tr to consider the complement set to this, i.e. all non-ASCII characters. With -d we ask tr to delete characters in that complement set.

We set LC_ALL to C (or POSIX) to have [:print:] character class match only the characters in the ASCII range 32 to 126. Otherwise it may match printable characters in the local locale, such as ä. The class [:cntrl:] matches characters in the range 0 to 31, and 127. With LC_ALL=C, these two classes together cover 0 through to 127, which are the ASCII characters.

To remove whole lines that contain any non-ASCII characters:

LC_ALL=C grep -v '[^[:print:][:cntrl:]]' <file.in >file.out

The expression [^[:print:][:cntrl:]] would match a single non-ASCII character. With -v we ask grep to extract all lines not matching that expression, i.e. to extract lines that don't contain any non-ASCII characters.

These two commands could also be done with sed:

Remove non-ASCII characters:

LC_ALL=C sed 's/[^[:print:][:cntrl:]]//g' <file.in >file.out

Remove lines with non-ASCII characters:

LC_ALL=C sed '/[^[:print:][:cntrl:]]/d' <file.in >file.out

Note that, as Stéphane points out in comments, the above commands would give you a text back that contains only ASCII characters, or at least characters encoded as ASCII (depending on the encoding of the file).


A totally different approach would be to use iconv:

iconv -c -t ascii file.in >file.out

This converts the file to ASCII encoding while silently dropping any characters (not lines) that couldn't be converted.

2
  • That assumes file.in is not encoded in charsets like BIG5, BIGHKSCS, GB18030, GBK which have characters whose encoding contains byte values in the 32-127 range. Commented Jan 28, 2021 at 9:19
  • Note that strictly speaking, LC_ALL=C doesn't imply "ASCII" (though in practice it does in the immense majority of ASCII-based systems (themselves being immensely majoritary), so it's good enough an approximation. Commented Jan 28, 2021 at 9:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .