What is the fastest command line way to merge the different lines of files? For example, I have two files:

a.txt:

foo  
bar
foobar

b.txt

foo
foobar
line
by
bar

And I would like to get the following output:

foo
bar
foobar
line
by

Is there any fast way to merge files like the example above? (The order of the lines isn't important)

up vote 8 down vote accepted

Use awk seen if you don't want to sort the file:

$ awk '!seen[$0]++' a.txt b.txt
foo  
bar
foobar
line
by
  • Does it delete the duplicated lines in a single file? – kbenda Aug 31 at 11:26
  • @KrisztiánBenda, it just print the uniq lines in each file. – msp9011 Aug 31 at 11:33
  • @KrisztiánBenda you can redirect the out to another file, if required. – msp9011 Aug 31 at 11:34
  • Thanks @SivaPrasath for your comment. It seems to me seen is not a valid command in my environment. And apt-get install seen does not find it. – kbenda Aug 31 at 11:41
  • 2
    @KrisztiánBenda It's an associative array in awk. You may use any array name and seen is often used for these kinds of operations. awk '!p[$0]++' would work just as well. Note that this solution is appropriate for reasonably large amounts of data, but that memory consumption grows with the number of unique lines in the input. – Kusalananda Aug 31 at 12:20
$ sort -u a.txt b.txt
bar
by
foo
foobar
line

This would sort the (combined) contents of the two files, removing the duplicate lines. The downside is that if a line is duplicated in one of the files, this would also be removed.

To write the result to c.txt, use

sort -u -o c.txt a.txt b.txt

or a standard redirection

sort -u a.txt b.txt >c.txt
  • Fastest/shortest is perhaps sort -u {a,b,-oc}.txt then ? – steve Aug 31 at 10:51
  • @steve Gah! I would avoid that. That requires a shell that understands brace expansions, and a sort implementation that allows for options to come after the operands on the command line. It's far safer to just be explicit, at least with the command line options and their arguments. You may use sort -u -o c.txt [ab].txt if you want. – Kusalananda Aug 31 at 10:56

Based on your desired output, I presume that you just want both files merged with only unique strings.

In which case, cat, sort and uniq can do this for you:

cat a.txt b.txt | sort | uniq > c.txt

cat opens the contents of both files

sort sorts the output alphabetically

uniq lists only unique strings

> c.txt puts all of the output in a new file c.txt

  • sort command is more enough to do.. – shas Aug 31 at 12:10
  • 3
    so I have learned.. that being said the answer is still an alternative, albeit uneccessarily long-winded (relatively anyway) working answer. – RobotJohnny Aug 31 at 12:15
  • this is also a fast method. – prosti Aug 31 at 22:54

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.