Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
What is the fastest command line way to merge the different lines of files? For example, I have two files:
a.txt:
foo
bar
foobar
b.txt
foo
foobar
line
by
bar
And I would like to get the following output:
foo
bar
foobar
line
by
Is there any fast way to merge files like the example above? (The order of the lines isn't important)
$ sort -u a.txt b.txt
bar
by
foo
foobar
line
This would sort the (combined) contents of the two files, removing the duplicate lines. The downside is that if a line is duplicated in one of the files, this would also be removed.
To write the result to c.txt, use
sort -u -o c.txt a.txt b.txt
or a standard redirection
sort -u a.txt b.txt >c.txt
sort implementation that allows for options to come after the operands on the command line. It's far safer to just be explicit, at least with the command line options and their arguments. You may use sort -u -o c.txt [ab].txt if you want.
Use awk seen if you don't want to sort the file:
$ awk '!seen[$0]++' a.txt b.txt
foo
bar
foobar
line
by
apt-get install seen does not find it.
awk. You may use any array name and seen is often used for these kinds of operations. awk '!p[$0]++' would work just as well. Note that this solution is appropriate for reasonably large amounts of data, but that memory consumption grows with the number of unique lines in the input.
Based on your desired output, I presume that you just want both files merged with only unique strings.
In which case, cat, sort and uniq can do this for you:
cat a.txt b.txt | sort | uniq > c.txt
cat opens the contents of both files
sort sorts the output alphabetically
uniq lists only unique strings
> c.txt puts all of the output in a new file c.txt
