I have an embedded linux system using Busybox (OpenWRT) - so commands are limited. I have two files that look like:

first file


second file


I need to merge these 2 lists into 1 file, and remove the duplicates. I don't have diff (space is limited) so we get to use the great awk, sed, and grep (or other tools that might be included in a standard Busybox instance). Going to a merge file like:

command1 > mylist.merge 
command2 mylist.merge > originallist

is totally ok. It doesn't have to be a single-line command.

Currently defined functions in the instance of Busybox that I am using (default OpenWRT): [, [[, arping, ash, awk, basename, brctl, bunzip2, bzcat, cat, chgrp, chmod, chown, chroot, clear, cmp, cp, crond, crontab, cut, date, dd, df, dirname, dmesg, du, echo, egrep, env, expr, false, fgrep, find, free, fsync, grep, gunzip, gzip, halt, head, hexdump, hostid, hwclock, id, ifconfig, init, insmod, kill, killall, klogd, less, ln, lock, logger, logread, ls, lsmod, md5sum, mkdir, mkfifo, mknod, mktemp, mount, mv, nc, netmsg, netstat, nice, nslookup, ntpd, passwd, pgrep, pidof, ping, ping6, pivot_root, pkill, poweroff, printf, ps, pwd, reboot, reset, rm, rmdir, rmmod, route, sed, seq, sh, sleep, sort, start-stop-daemon, strings, switch_root, sync, sysctl, syslogd, tail, tar, tee, telnet, telnetd, test, time, top, touch, tr, traceroute, true, udhcpc, umount, uname, uniq, uptime, vconfig, vi, watchdog, wc, wget, which, xargs, yes, zcat

migrated from serverfault.com Oct 6 '12 at 15:56

This question came from our site for system and network administrators.


I think

sort file1 file2 | uniq

will do what you want.

Additional Documentation: uniq sort

  • 8
    busybox sort supports the unique flag -u. – Thor Oct 2 '12 at 18:53
  • @Thor: oooh cheers that's not a switch I'm familiar with. – Iain Oct 2 '12 at 20:18

In just one command without any pipe :

sort -u FILE1 FILE2


Suppress duplicate lines

-> http://www.busybox.net/downloads/BusyBox.html

  • which one is better for very large files? sort file1 file2 file3 file4 | uniq or sort -u file1 file2 file3 file4 – 0x90 Jan 27 at 5:52

Another solution:

awk '!a[$0]++' file_1 file_2
  • I saw that it made a difference which argument came first. Otherwise great solution, thanks. – dezza Jan 8 '17 at 15:21

To sort according to some key column use following :

awk '!duplicate[$1,$2,$3]++' file_1 file_2

here consider first, second and third column as your primary key.


The files on your question are sorted.
If the source files are indeed sorted, you can uniq and merge in one step:

sort -um file1 file2 > mylist.merge

For numeric sort (not alphanumeric), use:

sort -num file1 file2 > mylist.merge

That could not be done in-place (redirected to one source file).

If the files are not sorted, sort them (this sort could be done in place, using the sort option -o. However, the whole file needs to be loaded into memory):

sort -uo file1 file1
sort -uo file2 file2
sort -um file1 file2 > mylist.merge
mv mylist.merge originallist

That would be faster than the simpler "one command line" to sort all:

cat file1 file2 | sort -u >mylist.merge

However, this line could be useful for small files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy