Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm sorting a 25 GB file containing 200M lines on Fedora 18 using:

sort -S 10G -T /bigdisk bigfile

and I am getting process killed due to no memory. The process isn't maxing out my RAM before death (12 GB free) and disk space is ok.

Any suggestions as to the cause appreciated. I'm sure sort can manage this due to its partitioning and merge.

More info from dmesg

[87278.935572] Out of memory: Kill process 1971 (sort) score 258 or sacrifice child
[87278.935574] Killed process 1971 (sort) total-vm:4512168kB, anon-rss:4237040kB, file-rss:584kB 
share|improve this question

migrated from serverfault.com Mar 16 '13 at 16:48

This question came from our site for professional system and network administrators.

    
Does the system have any per process or per user RAM limits? (check with limit or ulimit) –  Hennes Mar 15 '13 at 15:04
    
nope, Firefox and eclipse manage to eat it all :) –  Aiden Bell Mar 15 '13 at 15:06
    
Does it really get killed? You may connect with strace shortly before the crash and have a look at what's really happening. –  Hauke Laging Mar 15 '13 at 15:23
    
Is this a 32 or 64 bit kernel? –  Zoredache Mar 15 '13 at 15:25
    
64 bit kernel. it gets killed, output is via dmesg sigalling killed for memory as opposed to a segfault or something –  Aiden Bell Mar 15 '13 at 15:53

2 Answers 2

Don't use -S 10G, it is far too much (and probably not doing what you think). OOMkiller launching does mean your system is using all of it's memory.

According to the algorithm used by sort, it will use memory according to what is available: half of the biggest number between TotalMem/8 and AvailableMem.

So, for example, if you have 4 GB of available mem (out of 8 GB), sort will use 2GB of RAM. It should also create many 2 GB files in /bigdisk and finally merge-sort them.

share|improve this answer
1  
To be fair, sort is killed off if I use -S or not. Both top and /proc/meminfo report I have free memory on death. The machine has 16G of ram and only 3 is actually in use when the sort process is launched. Manually using split, sort on the parts and them sort -m worked with no memory issues. –  Aiden Bell Mar 18 '13 at 23:25
    
@AidenBell I just thought of something: are you using a 32 bit version of sort? Check with file $(which sort). –  Totor Mar 19 '13 at 0:07
    
nope. 64-bit. I'm beginning to think it might be an actual gasp bug somewhere. –  Aiden Bell Mar 19 '13 at 18:29

Try setting vm.overcommit_memory = 1: «…

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

…»

And do use swap.

share|improve this answer
    
The OOM killer is only triggered when "it actually runs out" of memory. So this settings won't solve anything. It might have if the Out of mem message had come from sort and not from the kernel. –  Totor Mar 16 '13 at 18:12
    
@Totor What? Have you read the doc? –  poige Mar 16 '13 at 18:15
    
@poige It'd help if you actually mentioned what was wrong with his comment –  Michael Mrozek Mar 18 '13 at 13:39
    
@MichaelMrozek, Sapienti sat –  poige Mar 18 '13 at 17:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.