2

I've tried to find the existing topics about this theme and I found something but it's not the 100% what I'm looking for and my internet connection is bad last few days so I needed to quit searching and post a new thread...

So my problem is I have a .txt file with many lines (over 50000), every line has 5 letter string like this:

KKIUB
SDCVG
KJUTT
NGTHH
WWLEE
XGHTP
NJFRT
PPSFF
ZZZLP
XDRFX
JJJJJ
KIEYW
...

I want all lines in a file that contain two (or more) same letters to be deleted. The order of duplicates isn't important, so all lines that contain a letter two times in a line must be deleted. Please note that sometimes there can be all 5 same letters, sometimes 3 same letters, sometimes only 2 same letters + not close to each other, e.g "GOHIG". 

KKIUB ---> delete
SDCVG ---> stays
KJUTT ---> delete
NGTHH ---> delete
WWLEE ---> delete
XGHTP ---> stays
NJFRT ---> stays
PPSFF ---> delete
ZZZLP ---> delete
XDRFX ---> delete
JJJJJ ---> delete
KIEYW ---> stays

I'm trying with sed function but was not able to have good results. Also I would like to export it into another .txt file. Any help?

Improve this question

1 Answer 1

Reset to default
4 
sed -e '/\(.\).*\1/d' yourfile > youroutputfile
Improve this answer
6
  • thank you, it works. Is possible to make output in another file?
    – user219258
    Mar 21, 2017 at 20:02
  • 2
    done , you may want to try that.
    – user218374
    Mar 21, 2017 at 20:45
  • 8
    Or just grep -v '\(.\).*\1'
    – Stéphane Chazelas
    Mar 21, 2017 at 20:47
  • 1
    You need to change nothing.
    – user218374
    Mar 21, 2017 at 21:14
  • 1
    @Vcore9: "Is possible to make output in another file?" Yes, then it will be sed -e '/\(.\).*\1/d' yourfile > another (sorry, you asked for it ;-)
    – hschou
    Mar 21, 2017 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy