I have a huge file that contains 80 genomes and five chromosomes. I was wondering is there is an easy way to select only the lines from ONE chromosome (for example 1) but only at the position showing SNP VARIATIONS.

1   180754  GGGGGGGCC   
1   180755  CCCCCCCCC

1   180756  CCTCCCCTC   
1   180757  AAAAAAAAA   
1   180759  TTTTTTTTT   
3   7874113 TTTTTTTTT   
3   7874114 GGGGGGGGG   
3   7874115 GGGGGGGGG   
3   7874116 GGGGGGGGG

This is what I want to get in the end:

1   180754  GGGGGGGC        
1   180756  CCTCCCCT    
  • 1
    Can you explain what "SNP VARIATIONS" means please? Apr 28, 2014 at 15:03
  • Snp means (Single nucleotide polymorphism) if on one line you have AAAAAAAAAAA (there is no snp); But if you have AAAAATTTTTT there is a SNP variation; and I am interested in those lines
    – Mike
    Apr 28, 2014 at 15:08
  • 4
    I don't know who downvoted this question but I don't think it's good form to do so just because the post makes reference to something you don't understand. The proper response is to do what steeldriver did and use the comments to ask for clarification. The question itself is legitimate, even if you don't immediately understand what it's asking.
    – Bratchley
    Apr 28, 2014 at 15:35
  • @JoelDavis, good point. Some people misuse down-voting just to discourage new users :) I mean using downvote for legit questions is not at all acceptable. I have seen it quiet frequently and it is not a good sign.
    – Ramesh
    Apr 28, 2014 at 20:55

3 Answers 3


For your example:

egrep '1 [[:digit:]]+ (GGGGGGGC|CCTCCCCT)' file

--- edit: so try this:

egrep -v '([A-Z])\1{8}' data
  • Thanks for the quick answer! But I have a huge file 20GB and want the get all lines where the snps vary. I cannot specify the sequence.
    – Mike
    Apr 28, 2014 at 14:49
  • Sorry what does the 8 refers to? the number of lines?
    – Mike
    Apr 28, 2014 at 15:12
  • Number of characters to repeat. In your example, all lines has 9 characters of this AAA... so I have to put 8. Apr 28, 2014 at 15:14

Based on the sample data you've provided you can do this using grep. Given the only valid values for nucleotides is A,C,G,T we can create a set of characters with just those included.

$ grep -Ev '([ACGT])\1{8}|^$' sample.txt 
1   180754  GGGGGGGCC
1   180756  CCTCCCCTC

This works by ignoring all the results that are mono-typed. The regular expression [ACGT]\1{8} matches all the strings that are of the form AAAAAAAA, CCCCCCCC, etc. The reversal of the matching is accomplished through the -v switch.

   -v, --invert-match
        Invert the sense of matching, to select non-matching lines.  (-v 
        is specified by POSIX.)

The use of -E makes our argument to grep a regular expression that matches the above, [ACGT]\1{8} or empty lines (^$).

   -E, --extended-regexp
        Interpret PATTERN as an extended regular expression (ERE, see 
        below).  (-E is specified by POSIX.)

The vertical bar (aka. pipe (|)) between the 2 arguments makes this an either this or that type of matching situation.

For example grep -Ev "arg1|arg2" <file>. arg1 would be [ACGT]\1{8} while arg2 would be ^$.


I'm not sure exactly what you're asking (it would be helpful to explain your requirement lexically rather than semantically) but

sed -r sed -r '/^[0-9]+[[:space:]]+[0-9]+[[:space:]]+(.)\1+\>/d'

will delete (i.e. not print) any lines in which the third space-delimited field consists entirely of repeats of the same initial character e.g.

$ sed -r '/^[0-9]+[[:space:]]+[0-9]+[[:space:]]+(.)\1+\>/d' genomes.txt
1   180754  GGGGGGGCC   

1   180756  CCTCCCCTC   

If you know that only the genome field can contain uppercase letters, it might be safe to do a much simpler expression

sed -r '/\<([A-Z])\1+\>/d' genomes.txt
  • (it would be helpful to explain your requirement lexically rather than semantically) YOU ARE RIGHT! SORRY! This is the first time I am asking something in a forum. Thank you for your answer!
    – Mike
    Apr 28, 2014 at 15:22
  • You're welcome, actually I think @sim's grep solutions are nicer - although I'd probably have written egrep -vw '([A-Z])\1+' Apr 28, 2014 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.