3

This question already has an answer here:

For example, a file file1.txt contains

Hi how are you  
hello  
today is monday  
hello  
I am fine  
Hi how are you 

After the processing of file1.txt it should write to file2.txt and contents should be like this without repeating the same lines.

Hi how are you  
hello  
today is monday  
I am fine  

What command can I use to do that in linux terminal?

marked as duplicate by dr01, Jeff Schaller, don_crissti, Jakuje, user34720 Jan 25 '16 at 13:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

5

This is an easy job for sort, use the unique (-u) option of sort:

% sort -u file1.txt
hello
Hi how are you
I am fine
today is monday

To save it in file2.txt:

sort -u file1.txt >file2.txt

If you want to preserve the initial order:

% nl file1.txt | sort -uk2,2 | sort -k1,1n | cut -f2
Hi how are you
hello
today is monday
I am fine
  • Not sure how this is marked as correct as the output shown in the answer does not match the OPs request. – Ian W Jan 25 '16 at 9:15
  • 1
    @IanW Fixed.... – heemayl Jan 25 '16 at 10:06
6
start cmd:> awk 'lines[$0]++ == 0' input
Hi how are you
hello  
today is monday  
I am fine
  • @Rmano Not my achievement, though; seen here before... ;-) (easier to reproduce than to find the other question). – Hauke Laging Jan 25 '16 at 12:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.