Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

My question is similar to this question but with a couple of different constraints:

  • I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB.
  • I need to remove any duplicate lines.
  • The process may sort the list during the course of removing the duplicates but not required.
  • There is enough space on the partition to hold the new unique wordlist outputted.

I have tried both of these methods but they both fail with out of memory errors.

sort -u wordlist.lst > wordlist_unique.lst
awk '!seen[$0]++' wordlist.lst > wordlist_unique.lst
awk: (FILENAME=wordlist.lst FNR=43601815) fatal: assoc_lookup: bucket-ahname_str: can't allocate 10 bytes of memory (Cannot allocate memory)

What other approaches can I try?

share|improve this question

1 Answer 1

up vote 7 down vote accepted

Try using sort with the -o/--output=FILE option instead of redirecting the output. You might also try setting the buffer-size with the -S/--buffer-size=SIZE. Also, try -s/--stable. And read the man page, it offers all of the info I gave.

The full command you can use that might work for what you're doing:

sort -us -o wordlist_unique.lst wordlist.lst

You might also want to read the following URL:

http://www.gnu.org/s/coreutils/manual/html_node/sort-invocation.html

That more thoroughly explains sort than the man page.

share|improve this answer
    
thanks for the suggestion, unfortunately using --output still doesn't fix the out of memory error. Running that still gives me sort: write failed: /root/tmp/sortVxscLn: No space left on device. The problem is a bit annoying in that it doesn't fail right away. It seems like you have to wait for the memory to get exhausted before it will error out. –  greatwolf Aug 29 '11 at 4:20
3  
@Victor T.: That is not an out of memory error, but an out of disk space error. Is /root on a different filesystem to your data? If so, use the -T/--temporary-directory option to sort to use a filesystem with more free space. –  camh Aug 29 '11 at 5:04
    
@camh thanks that did the trick. Didn't realize you could specify which intermediate buffer to use. –  greatwolf Aug 29 '11 at 5:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.