Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 4 down vote favorite

I'm sorting a 25 GB file containing 200M lines on Fedora 18 using:

sort -S 10G -T /bigdisk bigfile

and I am getting process killed due to no memory. The process isn't maxing out my RAM before death (12 GB free) and disk space is ok.

Any suggestions as to the cause appreciated. I'm sure sort can manage this due to its partitioning and merge.

More info from dmesg

[87278.935572] Out of memory: Kill process 1971 (sort) score 258 or sacrifice child
[87278.935574] Killed process 1971 (sort) total-vm:4512168kB, anon-rss:4237040kB, file-rss:584kB
share|improve this question
Does the system have any per process or per user RAM limits? (check with limit or ulimit) – Hennes Mar 15 at 15:04
nope, Firefox and eclipse manage to eat it all :) – Aiden Bell Mar 15 at 15:06
Does it really get killed? You may connect with strace shortly before the crash and have a look at what's really happening. – Hauke Laging Mar 15 at 15:23
Is this a 32 or 64 bit kernel? – Zoredache Mar 15 at 15:25
64 bit kernel. it gets killed, output is via dmesg sigalling killed for memory as opposed to a segfault or something – Aiden Bell Mar 15 at 15:53
show 8 more comments

migrated from serverfault.com Mar 16 at 16:48

2 Answers

up vote 2 down vote

Don't use -S 10G, it is far too much (and probably not doing what you think). OOMkiller launching does mean your system is using all of it's memory.

According to the algorithm used by sort, it will use memory according to what is available: half of the biggest number between TotalMem/8 and AvailableMem.

So, for example, if you have 4 GB of available mem (out of 8 GB), sort will use 2GB of RAM. It should also create many 2 GB files in /bigdisk and finally merge-sort them.

share|improve this answer
1  
To be fair, sort is killed off if I use -S or not. Both top and /proc/meminfo report I have free memory on death. The machine has 16G of ram and only 3 is actually in use when the sort process is launched. Manually using split, sort on the parts and them sort -m worked with no memory issues. – Aiden Bell Mar 18 at 23:25
@AidenBell I just thought of something: are you using a 32 bit version of sort? Check with file $(which sort). – Totor Mar 19 at 0:07
nope. 64-bit. I'm beginning to think it might be an actual gasp bug somewhere. – Aiden Bell Mar 19 at 18:29
up vote 0 down vote

Try setting vm.overcommit_memory = 1: «…

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

…»

And do use swap.

share|improve this answer
The OOM killer is only triggered when "it actually runs out" of memory. So this settings won't solve anything. It might have if the Out of mem message had come from sort and not from the kernel. – Totor Mar 16 at 18:12
@Totor What? Have you read the doc? – poige Mar 16 at 18:15
@poige It'd help if you actually mentioned what was wrong with his comment – Michael Mrozek♦ Mar 18 at 13:39
@MichaelMrozek, Sapienti sat – poige Mar 18 at 17:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.