awk filter rows that only consist a single letter

Question

I have a file (file1) that looks like this:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 2 AA 234 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ROW 3 AA 122 XXXXXXXXXXXXXXXXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
...

there are multiple rows that contain different numbers of X. however, the result should not contains the rows which only consist of X, it should be:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
...

Thank you for the help!

thanasisp · Accepted Answer · 2022-05-13 19:39:07Z

7

With awk, print the lines where last field has at least one character which is not X:

awk '$NF ~ /[^X]/' file

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Or with grep:

grep -v '[[:space:]]XX*$' file

edited May 13 at 19:39

answered May 13 at 17:32

thanasisp

7,2852 gold badges23 silver badges37 bronze badges

In case the input file is tab delimited, you have to set the separator, awk -F'\t' '$NF ~ /[^X]/' file.
– thanasisp
May 14 at 1:37
Note that for lines containing only whitespace, $NF, becomes $0, the full record, and awk will filter out empty lines, but not blank lines.
– Stéphane Chazelas
May 16 at 8:33
True, then we can use the existing grep or a negative expression. I feel that there are more cases/versions prior to this. For example, a line where the 5th field is missing and the 4th is X. Or the 5th field is empty and the 4th is a number, then a negative expression would filter it. Some small modification would be needed probably for any command for cases not included into the example.
– thanasisp
May 16 at 8:53

Add a comment |

Kusalananda · Accepted Answer · 2022-05-14 07:36:29Z

6

Using standard sed to delete all lines that contain a space followed by only X at the end:

sed '/ X\{1,\}$/d' file

or,

sed '/ XX*$/d' file

With a sed that understands extended regular expressions if given the -E option:

sed -E '/ X+$/d' file

If the delimiter in the file is not space but tab, then use [[:blank:]] in place of the initial space in the expression.

For each of the sed '/RE/d' variations above (and below), the equivalent grep command would be grep -v 'RE', e.g.

grep -v ' X\{1,\}$' file

is equivalent to

sed '/ X\{1,\}$/d' file

To remove lines that have only copies of any single character at the end:

sed '/ \(.\)\1*$/d' file

This matches a space followed by some character and then zero or more of that specific character to the end of the line. Matched lines are deleted.

Just playing with the same idea, the following would remove lines that contains only a repeated sequence (such as ABCABC or ABABABAB) at the end:

sed '/ \(..*\)\1\1*$/d' file

edited May 14 at 7:36

answered May 13 at 18:28

Kusalananda♦

282k32 gold badges566 silver badges854 bronze badges

1

Note that sed /pattern/d is the same as grep -v pattern
– Stéphane Chazelas
May 14 at 7:29
1

@StéphaneChazelas I was in sed mode. You're correct. I'll add that.
– Kusalananda ♦
May 14 at 7:31
@StéphaneChazelas: Do you mean "the same" in the sense of just comparing general behavior and response ? or did you mean that in some deeper algorithmic way ?
– Cbhihe
May 14 at 21:12

Add a comment |

αғsнιη · Accepted Answer · 2022-05-13 17:54:45Z

3

To check every field rather than only the last field, you can do:

awk -F'(^|[\t ]+)X+([\t ]+|$)' 'NF==1' infile

or explicitly check every fields using loop:

awk '{ for(i=1; i<=NF; i++) if($i ~/^X+$/)next };1' infile

Or using the sub() function and with the help of word-boundaries (GNU awk?):

awk '{ bkp=$0; if(! sub("\\<X+\\>", "")) print bkp }' infile

edited May 13 at 17:54

answered May 13 at 17:32

αғsнιη

38.9k15 gold badges66 silver badges107 bronze badges

Add a comment |

HatLess · Accepted Answer · 2022-05-13 18:29:59Z

2

Using sed

$ sed -n '/\([^ ]* \)\{4\}X\+$/!p' input_file
ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

answered May 13 at 18:29

HatLess

2,1871 gold badge5 silver badges19 bronze badges

Add a comment |

DanieleGrassini · Accepted Answer · 2022-05-13 19:01:30Z

2

awk -F ' X+$' 'NF == 1' file

Split line on any space, followed by just X till the end. If there are just one record, then the last field is made by only X.

perl -alne 'print if $F[-1] =~ y/X/-/ != length $F[-1]' test

If the number of the transliteration on the last field is equal to length of the field then the last record is made, for whole, by X.

edited May 13 at 19:01

answered May 13 at 17:45

DanieleGrassini

2,7095 silver badges17 bronze badges

All fine, but wouldn't the 2nd solution although perfectly legit, be particularly costlier ?
– Cbhihe
May 14 at 18:26
Surprise ! Your awk one liner costs 7.01ms and the perl one 4.60ms, while @kusalananda's sed based solution costs: 5.57ms, averaged over 1000 replicated executions (using date +"%s.%N", i.e. wall clock time). Why is Perl so darn fast, even with a transliteration operation and a conditional test that I still think should be rather costly ?
– Cbhihe
May 14 at 21:11
1

@Cbhihe, perl doesn't decode its input as text by default. You'll probably get different different results if you run them under LC_ALL=C.
– Stéphane Chazelas
May 16 at 8:21

Add a comment |

jubilatious1 · Accepted Answer · 2022-05-13 19:50:08Z

Using Raku (formerly known as Perl_6)

raku -ne '.put unless .words[4] ~~ /^ X+ $/ ;'

Sample Input:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 2 AA 234 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ROW 3 AA 122 XXXXXXXXXXXXXXXXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Sample Output:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Above uses a conditional to output the line unless the 5th column (whitespace-separated words) matches entirely X from beginning to end of the string, i.e. /^ … $/.

Note: 5th column is zero-index .words[4]. Also, because the match consists of a single character, there's no need to create a custom character class and the regex can simply be written as /^ X+ $/. However, if you want to remove lines where the 5th column contains a mixture of X and x, write the regex like so (create a custom character class with <[ … ]>):

raku -ne '.put unless .words[4] ~~ /^ <[Xx]>+ $/ ;'

https://raku.org

Praveen Kumar BS · Accepted Answer · 2022-05-16 08:25:03Z

2

awk '$NF !~ /^X+$/' filename

output

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

edited May 16 at 8:25

answered May 16 at 8:13

Praveen Kumar BS

4,9042 gold badges8 silver badges13 bronze badges

Add a comment |

guest_7 · Accepted Answer · 2022-05-17 06:22:20Z

2

grep -P '(?:(?!X)\S)\S*$' file

GNU grep compiled with PCRE mode
This will select all those lines which have atleast one non-X, nonwhitespace character in the end of line.
The (?!X)\S tempers the \S to be all nonwhitespaces minus 'X'
After we've seen atleast one tempered \S which is not X,

answered May 17 at 6:22

guest_7

5,6061 gold badge5 silver badges13 bronze badges

Add a comment |

Stack Exchange Network

awk filter rows that only consist a single letter

8 Answers 8

Your Answer

Not the answer you're looking for? Browse other questions tagged
bash
shell
awk
sed
or ask your own question.

Hot Network Questions

awk filter rows that only consist a single letter

8 Answers 8

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged bashshellawksed or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
bash
shell
awk
sed
or ask your own question.