Detect odd characters in big text files

Question

I need to check if my entire files contained only 4 characters; "A", "T", "G" and "C". I used to split the characters using sed and then grep -o and -v to exclude the targeted characters for checking.

Is there any simple and straight forward way to do this in linux? Using sed / awk / grep?

(There seemed to be suggestion on this related questions but they were including the whole texts in the command. My file size is too big for this.)

For example, there are four lines in the input file, with possibility of other characters existing in the line (other than ATGC). I would like to detect the odd characters and show the odd characters together with the number of line they are in, if possible.

Input:

ATTGTAAGGTAAGTGGATTYTCCGGGRETC
TTVGGATCGTTGACCAGTK
GCCCGGGCCGGTCCTTTGGTGCGTGGGG
CTCTCCCAACCCCCCCACCCTCGACCTGAGCTCAGGCXC

Desired Output:

1:Y
1:R
1:E
2:V
2:K
4:X

Answer 1

-n Prefix each line of output with the 1-based line number.
-o Print only the matched parts.
[^ATGC] exclude characters.

grep -no '[^ATGC]' file

Answer 2

If you have many files, and most of those are valid, there is an efficient way to make a preliminary check. Just count the invalid characters: if there are none, there is no point making a more precise test of the file. We use tr to remove the valid ones, and wc -c to count the others.

More precise reporting is needed for the case where the count is non-zero.

I would suggest using awk, and defining the FS (field separator) as 'FS=[^ATGC]+', which means "any sequence of characters that are not A, T, G or C". If there are no incorrect characters on a line, then there will be only one field.

If more than one field exists, we can use the GNU/awk extension to split(), which provides the exact text of each field separator.

#! /bin/bash

Awk='
BEGIN { FS = "[^ATGC]+"; }

function Show (tx, Local, f, c, fTxt, fSep) {
    split (tx, fTxt, FS, fSep)
    for (f = 1; f in fSep; ++f) {
        c += length (fTxt[f]);
        printf ("File %s Line %d Column %d Has :%s:\n",
            FILENAME, FNR, 1 + c, fSep[f]);
        c += length (fSep[f]);
    }
}
NF > 1 { Show( $0); }
'
    for fn in q??; do
        cc="$( tr -d 'ATGC\n' < "${fn}" | wc -c )"
        (( cc == 0 )) && { echo "$fn is OK"; continue; }
        awk "${Awk}" "${fn}"
    done

and to test:

Paul--) head q??
==> q01 <==
TTGTAAGGTAAGTGGATTYTCCGGGRETC
TTVGGATCGTTGACCAGTK
GCCCGGGCCGGTCCTTTGGTGCGTGGGG
CTCTCCCAACCCCCCCACCCTCGACCTGAGCTCAGGCXC
BAACCCCZ

==> q02 <==
GCCCGGGCCGGTCCTTTGGTGCGTGGGG

==> q03 <==
TTGTAAGGTAAGTGGATTYTCCGGGRETC
Paul--) 
Paul--) ./qFix q01 q02 q03
File q01 Line 1 Column 19 Has :Y:
File q01 Line 1 Column 26 Has :RE:
File q01 Line 2 Column 3 Has :V:
File q01 Line 2 Column 19 Has :K:
File q01 Line 4 Column 38 Has :X:
File q01 Line 5 Column 1 Has :B:
File q01 Line 5 Column 8 Has :Z:
q02 is OK
File q03 Line 1 Column 19 Has :Y:
File q03 Line 1 Column 26 Has :RE:
Paul--)