2

I have a file (file1) that looks like this:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 2 AA 234 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ROW 3 AA 122 XXXXXXXXXXXXXXXXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
...

there are multiple rows that contain different numbers of X. however, the result should not contains the rows which only consist of X, it should be:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
...

Thank you for the help!

0

8 Answers 8

8

With awk, print the lines where last field has at least one character which is not X:

awk '$NF ~ /[^X]/' file

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Or with grep:

grep -v '[[:space:]]XX*$' file
3
  • In case the input file is tab delimited, you have to set the separator, awk -F'\t' '$NF ~ /[^X]/' file.
    – thanasisp
    May 14 at 1:37
  • Note that for lines containing only whitespace, $NF, becomes $0, the full record, and awk will filter out empty lines, but not blank lines. May 16 at 8:33
  • True, then we can use the existing grep or a negative expression. I feel that there are more cases/versions prior to this. For example, a line where the 5th field is missing and the 4th is X. Or the 5th field is empty and the 4th is a number, then a negative expression would filter it. Some small modification would be needed probably for any command for cases not included into the example.
    – thanasisp
    May 16 at 8:53
5

Using standard sed to delete all lines that contain a space followed by only X at the end:

sed '/ X\{1,\}$/d' file

or,

sed '/ XX*$/d' file

With a sed that understands extended regular expressions if given the -E option:

sed -E '/ X+$/d' file

If the delimiter in the file is not space but tab, then use [[:blank:]] in place of the initial space in the expression.

For each of the sed '/RE/d' variations above (and below), the equivalent grep command would be grep -v 'RE', e.g.

grep -v ' X\{1,\}$' file

is equivalent to

sed '/ X\{1,\}$/d' file

To remove lines that have only copies of any single character at the end:

sed '/ \(.\)\1*$/d' file

This matches a space followed by some character and then zero or more of that specific character to the end of the line. Matched lines are deleted.

Just playing with the same idea, the following would remove lines that contains only a repeated sequence (such as ABCABC or ABABABAB) at the end:

sed '/ \(..*\)\1\1*$/d' file
3
  • 1
    Note that sed /pattern/d is the same as grep -v pattern May 14 at 7:29
  • 1
    @StéphaneChazelas I was in sed mode. You're correct. I'll add that.
    – Kusalananda
    May 14 at 7:31
  • @StéphaneChazelas: Do you mean "the same" in the sense of just comparing general behavior and response ? or did you mean that in some deeper algorithmic way ?
    – Cbhihe
    May 14 at 21:12
3

To check every field rather than only the last field, you can do:

awk -F'(^|[\t ]+)X+([\t ]+|$)' 'NF==1' infile

or explicitly check every fields using loop:

awk '{ for(i=1; i<=NF; i++) if($i ~/^X+$/)next };1' infile

Or using the sub() function and with the help of word-boundaries (GNU awk?):

awk '{ bkp=$0; if(! sub("\\<X+\\>", "")) print bkp }' infile
2

Using sed

$ sed -n '/\([^ ]* \)\{4\}X\+$/!p' input_file
ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
2
awk -F ' X+$' 'NF == 1' file

Split line on any space, followed by just X till the end. If there are just one record, then the last field is made by only X.


perl -alne 'print if $F[-1] =~ y/X/-/ != length $F[-1]' test

If the number of the transliteration on the last field is equal to length of the field then the last record is made, for whole, by X.

3
  • All fine, but wouldn't the 2nd solution although perfectly legit, be particularly costlier ?
    – Cbhihe
    May 14 at 18:26
  • Surprise ! Your awk one liner costs 7.01ms and the perl one 4.60ms, while @kusalananda's sed based solution costs: 5.57ms, averaged over 1000 replicated executions (using date +"%s.%N", i.e. wall clock time). Why is Perl so darn fast, even with a transliteration operation and a conditional test that I still think should be rather costly ?
    – Cbhihe
    May 14 at 21:11
  • 1
    @Cbhihe, perl doesn't decode its input as text by default. You'll probably get different different results if you run them under LC_ALL=C. May 16 at 8:21
2

Using Raku (formerly known as Perl_6)

raku -ne '.put unless .words[4] ~~ /^ X+ $/ ;' 

Sample Input:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 2 AA 234 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ROW 3 AA 122 XXXXXXXXXXXXXXXXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Sample Output:

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY

Above uses a conditional to output the line unless the 5th column (whitespace-separated words) matches entirely X from beginning to end of the string, i.e. /^ … $/.

Note: 5th column is zero-index .words[4]. Also, because the match consists of a single character, there's no need to create a custom character class and the regex can simply be written as /^ X+ $/. However, if you want to remove lines where the 5th column contains a mixture of X and x, write the regex like so (create a custom character class with <[ … ]>):

raku -ne '.put unless .words[4] ~~ /^ <[Xx]>+ $/ ;' 

https://raku.org

2
awk '$NF !~ /^X+$/' filename

output

ROW 1 AA 120 APFGHKDESFNNJFHGRIHJASFGNSKDHFIXXXXXX
ROW 4 AA 89  WUAHGLIHGUNGBGDSYUXXXXXXXXXXXXXXFGOAYGIGWEIWIGFUEGFHUIWGEFU
ROW 5 AA 186 XXWANFJHOUNGRIGNO
ROW 6 AA 156 WANLHRIOGRNINGIJOHONJPHHYGKHDY
0
2
grep -P '(?:(?!X)\S)\S*$' file
  • GNU grep compiled with PCRE mode
  • This will select all those lines which have atleast one non-X, nonwhitespace character in the end of line.
  • The (?!X)\S tempers the \S to be all nonwhitespaces minus 'X'
  • After we've seen atleast one tempered \S which is not X,

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.