16

My question is similar to this question but with a couple of different constraints:

  • I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB.
  • I need to remove any duplicate lines.
  • The process may sort the list during the course of removing the duplicates but not required.
  • There is enough space on the partition to hold the new unique wordlist outputted.

I have tried both of these methods but they both fail with out of memory errors.

sort -u wordlist.lst > wordlist_unique.lst
awk '!seen[$0]++' wordlist.lst > wordlist_unique.lst
awk: (FILENAME=wordlist.lst FNR=43601815) fatal: assoc_lookup: bucket-ahname_str: can't allocate 10 bytes of memory (Cannot allocate memory)

What other approaches can I try?

18

Try using sort with the -o/--output=FILE option instead of redirecting the output. You might also try setting the buffer-size with the -S/--buffer-size=SIZE. Also, try -s/--stable. And read the man page, it offers all of the info I gave.

The full command you can use that might work for what you're doing:

sort -us -o wordlist_unique.lst wordlist.lst

You might also want to read the following URL:

http://www.gnu.org/s/coreutils/manual/html_node/sort-invocation.html

That more thoroughly explains sort than the man page.

  • thanks for the suggestion, unfortunately using --output still doesn't fix the out of memory error. Running that still gives me sort: write failed: /root/tmp/sortVxscLn: No space left on device. The problem is a bit annoying in that it doesn't fail right away. It seems like you have to wait for the memory to get exhausted before it will error out. – greatwolf Aug 29 '11 at 4:20
  • 8
    @Victor T.: That is not an out of memory error, but an out of disk space error. Is /root on a different filesystem to your data? If so, use the -T/--temporary-directory option to sort to use a filesystem with more free space. – camh Aug 29 '11 at 5:04
  • @camh thanks that did the trick. Didn't realize you could specify which intermediate buffer to use. – greatwolf Aug 29 '11 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.