Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).

What I need is to remove all the repetitions while (preferably, but this can be sacrificed for a significant performance boost) maintaining the original sequence order. In the result each line is to be unique. If there were 100 equal lines (usually the duplicates are spread across the file and won't be neighbours) there is to be only one of the kind left.

I have written a program in Scala (consider it Java if you don't know about Scala) to implement this. But maybe there are faster C-written native tools able to do this faster?

share|improve this question
this will destroy your ordering but,have you tried sort -u, I have no idea how or if it can run on such a massive file –  stemartin Jan 27 '12 at 15:57
C is often not significantly faster than Java, and if you're running it (in-order) now, there's a fair chance it'll finish before you get an answer here, implement it, and it finishes running; out of order, sort -u will probably be faster. –  Kevin Jan 27 '12 at 15:59

6 Answers 6

up vote 83 down vote accepted

An awk solution seen on #bash (Freenode):

awk '!seen[$0]++' filename
share|improve this answer
This is great! It worked flawlessly on my Solaris box. I only tested it on a small file though –  rahmu Jan 27 '12 at 16:50
Just tried this on a 2G file and it took three minutes on my notebook. Not bad. I also tried uniq filename | awk '!seen[$0]++', but it wasn't any faster. –  mgjk Jan 27 '12 at 19:27
This command seems not just working but more than 100 times faster than the program of mine. Thanks. –  Ivan Jan 27 '12 at 21:40
This is surprisingly faster than a more verbose awk version using 2 array lookups (shown as an expanded explanation in Gilles answer) : 0m36.132s vs 0m49.958s.. for 50 million lines.. I thought the bottleneck would be the I/O, but the extra array lookup is... 1 million elements in the array seems to make a rather significant dent... –  Peter.O Jan 28 '12 at 5:53
By the way, @larsmans has helped me to find out what was wrong with the proggy of mine, so don't think Scala/Java is that slow. The discussion is here: –  Ivan Jan 28 '12 at 18:18

There's a simple (which is not to say obvious) method using standard utilities which doesn't require a large memory except to run sort, which in most implementations has specific optimizations for huge files (a good external sort algorithm). An advantage of this method is that it only loops over all the lines inside special-purpose utilities, never inside interpreted languages.

<input nl -b a -s : |           # number the lines
sort -t : -k 2 -u |             # sort and uniquify ignoring the line numbers
sort -t : -k 1n |               # sort according to the line numbers
cut -d : -f 2- >output          # remove the line numbers

If all lines begin with a non-whitespace character, you can dispense with some of the options (use a literal tab instead of \t if your sed doesn't support \t):

<input nl | sort -k 2 -u | sort -k 1n | cut -f 2- >output

For a large amount of duplication, a method that only requires storing a single copy of each line in memory will perform better. With some interpretation overhead, there's a very concise awk script for that (already posted by enzotib):

<input awk '!seen[$0]++'

Less concisely: !seen[$0] {print} {seen[$0] += 1}, i.e. print the current line if it hasn't been seen yet, then increment the seen counter for this line (uninitialized variables or array elements have the numerical value 0).

For long lines, you can save memory by keeping only a non-spoofable checksum (e.g. a cryptographic digest) of each line. For example, using SHA-1, you only need 20 bytes plus a constant overhead per line. But computing digests is rather slow; this method will only win if you have a fast CPU (especially one with a hardware accelerator to compute the digests) and not a lot of memory relative to the size of the file and sufficiently long lines. No basic utility lets you compute a checksum for each line; you'd have to bear the interpretation overhead of Perl/Python/Ruby/… or write a dedicated compiled program.

<input perl -MDigest::MD5 -ne '$seen{Digest::MD5::md5($_)}++ or print' >output
share|improve this answer
+1 for nice explanation –  Eve Freeman Jan 28 '12 at 3:40

Assuming you can afford to keep as much as the de-duplicated file in memory (if your data is indeed duplicated by a factor of 100, that should be about 20MiB + overhead), you can do this very easily with Perl.

$ perl -ne 'print unless $dup{$_}++;' input_file > output_file

This preserves the order too.

You could extract the number of occurrences of each line from the %dup hash if you so wished, as an added free bonus.

If you prefer awk, this should do it too (same logic as the perl version, same ordering, same data gathered in the dup variable):

$ awk '{if (++dup[$0] == 1) print $0;}' input_file > output_file
share|improve this answer
This is too good @Mat, I was about to slurp the file, lol ;-). –  Nikhil Mulley Jan 27 '12 at 16:10
Now waiting for @ManAtWork for his sed and awk magic weavery too :-) –  Nikhil Mulley Jan 27 '12 at 16:11
awesome again for the awk tip :-) –  Nikhil Mulley Jan 27 '12 at 16:18
sort big-csv-file.csv | uniq > duplicates-removed.csv

Note that the output file will be sorted.

share|improve this answer
Not as fast as the awk command in other answers, but conceptually simple! –  Johann Mar 31 at 23:11
@Johann I am doing this pretty often on files with hundreds of thousands (even million) of short newline terminated strings. I get the results pretty quick for the experiments I am doing. It can be more important if used in scripts which are run again and again, savings in time can be considerable. –  xeon Mar 31 at 23:13
Use sort -u to remove duplicates during the sort, rather than after. (And saves memory bandwidth) piping it to another program). This is only better than the awk version if you want your output sorted, too. (The OP on this question wants his original ordering preserved, so this is a good answer for a slightly different use-case.) –  Peter Cordes Sep 14 at 15:39

Python One liners :

python -c "import sys; lines = sys.stdin.readlines(); print ''.join(sorted(set(lines)))" < InputFile
share|improve this answer
this causes the entire file to be slurped into memory and may not be a good fit for the OP's problem. Also not guaranteed to retain order –  1_CR Sep 15 '13 at 14:50
Thanks for the suggestion, I've been just learning python.. just tried this for learning purpose.. :) –  Rahul Patil Sep 15 '13 at 19:52
Here's a Python 2.7 version that is not a one-liner but (succinctly) returns unique lines preserving order without either loading the entire file into memory or creating a single gigantic string to feed to print –  1_CR Sep 16 '13 at 16:37
Thanks @1_CR I have something learn today :) OrderedDict –  Rahul Patil Sep 16 '13 at 16:39

With bash 4, a pure-bash solution that takes advantage of associative arrays can be used. Here is an example

unset llist; declare -A llist;
while read -r line; do
if [[ ${llist[$line]} ]]; then
  printf '%s\n' "$line"
done < file.txt
share|improve this answer
Don't use read loops to process big text files. bash has to read one-byte-at-a-time to avoid overshooting a newline. Bash is also not very fast at text processing in general compared to awk. If you do use this, read -ra will avoid eating backslashes in your input. Also, don't forget to unset llist after the loop, if you put this in a shell function or use it interactively. –  Peter Cordes Sep 14 at 15:44
@PeterCordes, or you could have just referenced this :-) –  1_CR Sep 14 at 20:41

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.