1

I have a text file that has a word on every line. I am trying to remove the lines that don't at least two different letters. For example, the file looks something like this:

words
books
aaa
letters 
zzzz

And I want the output file to look like this:

words
books
letters

I tried breaking every word in separate letters than grouping them together by using uniq -c and then wc -l but got stuck on the if statement. Also believe there must be an easier way of doing it, I just couldn't think of any other way of approaching this problem.

  • No, what I meant is that only words that contain the same letter should be removed, such as aaa but words such as dodo should remain in the file. – Cloud Nov 3 '17 at 10:18
  • 1
    Should a be removed (only one letter — but only one character long); should ßßß be removed (only one letter, but not ASCII); should !!! be removed (only one character, but not a letter)? What about aaA? (And I dare not ask about ßßs). – derobert Nov 3 '17 at 10:27
6

Assuming you mean character instead of letter (as in, you also want to delete lines that contain ... or 11 even though . or 1 are not letters):

grep -vx -e '' -e '\(.\)\1*'

or:

grep -vx '\(\(.\)\2*\)\{0,1\}'

That is remove (-v) empty lines or lines that start with one character (.) followed by that same character (\1 being a back-reference to what's captured by \(...\)) repeated 0 or more times (*) till the end of the line (-x anchors the pattern at the start and end of the line).

Portably, you can't use egrep or grep -E here as standard EREs don't have back references (only BREs do).

For lines that contain at least two different letters, ignoring the other types of characters (we'll use [[:alpha:]] here for letter, that is any character that is considered alphabetical in your locale):

grep -vx '[^[:alpha:]]*
[^[:alpha:]]*\([[:alpha:]]\)\([^[:alpha:]]*\1\)*[^[:alpha:]]*'

(on two lines, which is another way to pass two different patterns). Or:

grep -vx '[^[:alpha:]]*\([^[:alpha:]]*\([[:alpha:]]\)\([^[:alpha:]]*\2\)*[^[:alpha:]]*\)\{0,1\}'

That one would remove lines like 12345aaa (only one letter) or -+-+-+- (no letter).

If you want to delete Aaaa lines as well (that is ignore case when comparing letters), add the -i option.

Note that it works at the character level, so it may not do what you expect if there are graphemes expressed with more than one character. For instance, it would delete a line like that output by:

 $ printf 'e\u0300e\u0301\n'
 èé

(assuming GNU printf or compatible), but not one like:

 $ printf '\ue8\ue9\n'
 èé

(where e\u300 is the decomposed form and \ue8 the precomposed form of the è grapheme; e (U+0065) and è (U+00E8) are alphabetical, but not the U+0300 or U+0301 combining grave/acute accents).

To work with graphemes, you can use pcregrep or GNU grep with it's -P option:

For the first case (at least two different grapheme clusters):

grep -vxP '(?:(\X)\1*)?'

For the second case (at least two different letter grapheme clusters):

grep -vxP '(?:(?=\PL)\X)*(?:((?=\pL)\X)(?:(?:(?=\PL)\X)*\1(?!\pM))*(?:(?=\PL)\X)*)?'

Where (?=\PL)\X is a non-letter grapheme cluster (a grapheme cluster (\X) provided (?=...) it starts with a non-letter (\PL) and (?=\pL)\X a letter grapheme cluster.

\pL matches on the Letter unicode properly. Contrary to the [:alpha:] POSIX character class, it also includes letters from non-alphabetical scripts.

Note that it would consider e\u300\u301, e\u301\u300, \ue9\u300, \ue8\u301 as four different clusters even though they would all be a e with both an acute and grave accent.

Also beware of characters like (U+FB03) that contain several letters in one character.


With PCRE, you can also take positive approaches:

  • at least 2 different characters:

    grep -P '(.).*(?!\1).'
    
  • at least 2 different letter characters:

    grep -P '(\pL).*(?!\1)\pL'
    
  • at least 2 different grapheme clusters:

    grep -P '^\X*(\X)\X*(?!\1(?!\pM))\X'
    

    that one would not work properly with Koran Hangul in decomposed form (at least). PCRE (contrary to perl's RE with \b{g}) doesn't have grapheme boundary operator (AFAIK) and has limited support for unicode properties. We're using (?!\pM) (which means in that context: "provided it's not followed by a combining mark character") as an approximation, but that doesn't work for multi-part Hangul letters/syllable characters where the parts don't have that property. It would delete 려련련 for instance. Now one may also argue that each part is a distinct letter...

    With perl 5.22 or above, you could write it:

    perl -Mopen=locale -lne 'print if /\b{g}(\X).*\b{g}(?!\1\b{g})\X/'
    
  • at least 2 different letter grapheme clusters:

    grep -P '^\X*((?=\pL)\X)\X*(?!\1(?!\pM))(?=\pL)\X'
    

    Again, doesn't work on 려련련. With perl:

    perl -Mopen=locale -lne 'print if /\b{g}(?=\pL)(\X).*\b{g}(?!\1\b{g})(?=\pL)\X/'
    

With perl, we can use more straightforward approaches like:

  • at least 2 different characters:

    perl -Mopen=locale -MList::MoreUtils=uniq -lne '
      print if uniq(/./g) >= 2'
    
  • at least 2 different letter characters:

    perl -Mopen=locale -MList::MoreUtils=uniq -lne '
      print if uniq(/\pL/g) >= 2'
    
  • at least 2 different grapheme clusters:

    perl -Mopen=locale -MList::MoreUtils=uniq -lne '
      print if uniq(/\X/g) >= 2'
    
  • at least 2 different letter grapheme clusters:

    perl -Mopen=locale -MList::MoreUtils=uniq -lne '
      print if uniq(grep /^\pL/, /\X/g) >= 2'
    
1

According to your expected output - you want to skip words that have more than 2 identical characters:

grep approach:

grep -vE '(.)(\1){2,}' file

The utput:

words
books
letters 

To modify the file inplace you may apply the following sed approach:

sed -Ei '/(.)(\1){2}/d' file
  • That removes lines that contain sequences of 3 times the same character which is not what is being asked AFAICT. – Stéphane Chazelas Nov 3 '17 at 10:33
  • @StéphaneChazelas, how can you know? cause it has being asked initially wrong. It's obvious that the OP has confused in its description. AFAICT - is the crucial keyword in your comment – RomanPerekhrest Nov 3 '17 at 10:38
0

Positive search of the query:

while read -r line; do 
    n=$(echo "$line" | egrep -o . | sort -u);
    [[ ${#n} -gt 1 ]] && echo "$line"; 
done < file
  • That removes lines like " x " or with some echo implementation: -en. ITYM < <(cat < <(cat < <(cat < file))) or maybe < file instead of < <(cat file). See also Why is using a shell loop to process text considered bad practice? – Stéphane Chazelas Nov 3 '17 at 10:56
  • Thanks for the interesting post. I knew it was inefficient but i didn't know a better solution. Nevertheless, I see you already took care of that :) – gip Nov 4 '17 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.