2

I've tried to find the existing topics about this theme and I found something but it's not the 100% what I'm looking for and my internet connection is bad last few days so I needed to quit searching and post a new thread...

So my problem is I have a .txt file with many lines (over 50000), every line has 5 letter string like this:

KKIUB
SDCVG
KJUTT
NGTHH
WWLEE
XGHTP
NJFRT
PPSFF
ZZZLP
XDRFX
JJJJJ
KIEYW
...

I want all lines in a file that contain two (or more) same letters to be deleted. The order of duplicates isn't important, so all lines that contain a letter two times in a line must be deleted. Please note that sometimes there can be all 5 same letters, sometimes 3 same letters, sometimes only 2 same letters + not close to each other, e.g "GOHIG". 

KKIUB ---> delete
SDCVG ---> stays
KJUTT ---> delete
NGTHH ---> delete
WWLEE ---> delete
XGHTP ---> stays
NJFRT ---> stays
PPSFF ---> delete
ZZZLP ---> delete
XDRFX ---> delete
JJJJJ ---> delete
KIEYW ---> stays

I'm trying with sed function but was not able to have good results. Also I would like to export it into another .txt file. Any help?

| improve this question | |
4
sed -e '/\(.\).*\1/d' yourfile > youroutputfile
| improve this answer | |
  • thank you, it works. Is possible to make output in another file? – user219258 Mar 21 '17 at 20:02
  • 2
    done , you may want to try that. – user218374 Mar 21 '17 at 20:45
  • 8
    Or just grep -v '\(.\).*\1' – Stéphane Chazelas Mar 21 '17 at 20:47
  • 1
    You need to change nothing. – user218374 Mar 21 '17 at 21:14
  • 1
    @Vcore9: "Is possible to make output in another file?" Yes, then it will be sed -e '/\(.\).*\1/d' yourfile > another (sorry, you asked for it ;-) – hschou Mar 21 '17 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy