18

My question is similar to this question but with a couple of different constraints:

  • I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB.
  • I need to remove any duplicate lines.
  • The process may sort the list during the course of removing the duplicates but not required.
  • There is enough space on the partition to hold the new unique wordlist outputted.

I have tried both of these methods but they both fail with out of memory errors.

sort -u wordlist.lst > wordlist_unique.lst
awk '!seen[$0]++' wordlist.lst > wordlist_unique.lst
awk: (FILENAME=wordlist.lst FNR=43601815) fatal: assoc_lookup: bucket-ahname_str: can't allocate 10 bytes of memory (Cannot allocate memory)

What other approaches can I try?

1

1 Answer 1

20

Try using sort with the -o/--output=FILE option instead of redirecting the output. You might also try setting the buffer-size with the -S/--buffer-size=SIZE. Also, try -s/--stable. And read the man page, it offers all of the info I gave.

The full command you can use that might work for what you're doing:

sort -us -o wordlist_unique.lst wordlist.lst

You might also want to read the following URL:

http://www.gnu.org/s/coreutils/manual/html_node/sort-invocation.html

That more thoroughly explains sort than the man page.

3
  • thanks for the suggestion, unfortunately using --output still doesn't fix the out of memory error. Running that still gives me sort: write failed: /root/tmp/sortVxscLn: No space left on device. The problem is a bit annoying in that it doesn't fail right away. It seems like you have to wait for the memory to get exhausted before it will error out.
    – greatwolf
    Commented Aug 29, 2011 at 4:20
  • 10
    @Victor T.: That is not an out of memory error, but an out of disk space error. Is /root on a different filesystem to your data? If so, use the -T/--temporary-directory option to sort to use a filesystem with more free space.
    – camh
    Commented Aug 29, 2011 at 5:04
  • @camh thanks that did the trick. Didn't realize you could specify which intermediate buffer to use.
    – greatwolf
    Commented Aug 29, 2011 at 5:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .