0

How do I efficiently combine multiple text files and remove duplicate lines in the final file in Ubuntu?

I have these files:

file1.txt contains

alpha
beta
gamma
delta

file2.txt contains

beta
gamma
delta
epsilon

file3.txt contains

delta
epsilon
zeta
eta

I would like the final.txt file to contain:

alpha
beta
gamma
delta
epsilon
zeta
eta

I would appreciate the help.

  • 1
    Does the order of the lines in the final file matter? Otherwise, sort -u all the input files > output would do it. – Jeff Schaller Jul 20 '18 at 1:27
  • The order of lines doesn't matter. The result of sort -u file1.txt file2.txt file3.txt > final.txt contains 2 of delta and 2 of epsilon. I was looking for something that matches the final.txt – AvidLearner Jul 20 '18 at 1:35
0

If you want to print only the first instance of each line without sorting:

$ awk '!seen[$0]++' file1.txt file2.txt file3.txt
alpha
beta
gamma
delta
epsilon
zeta
eta
  • The output for awk '!seen[$0]++' file1.txt file2.txt file3.txt contains 2 lines of delta and 2 lines of epsilon. I am looking to remove any additional duplicates. – AvidLearner Jul 20 '18 at 2:04
  • @AvidLearner I tested it with the exact input you posted - if you are seeing something different, then your files are not the same (i.e. some apparently duplicate lines are actually distinct - for example, they have trailing whitespace) – steeldriver Jul 20 '18 at 2:08
  • Thank you. The trailing white spaces were the issue. I should have added my output of commands I tried in the original post for clarity. – AvidLearner Jul 20 '18 at 2:15
  • @AvidLearner if the inputs consist of single words per line, then you can avoid the trailing whitespace issue by keying on $1 rather than $0 – steeldriver Jul 20 '18 at 2:17
2

Very Simple

sort -u file[123].txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.