7

I'm sorting a 25 GB file containing 200M lines on Fedora 18 using:

sort -S 10G -T /bigdisk bigfile

and I am getting process killed due to no memory. The process isn't maxing out my RAM before death (12 GB free) and disk space is ok.

Any suggestions as to the cause appreciated. I'm sure sort can manage this due to its partitioning and merge.

More info from dmesg

[87278.935572] Out of memory: Kill process 1971 (sort) score 258 or sacrifice child
[87278.935574] Killed process 1971 (sort) total-vm:4512168kB, anon-rss:4237040kB, file-rss:584kB
|improve this question
  • Does the system have any per process or per user RAM limits? (check with limit or ulimit) – Hennes Mar 15 '13 at 15:04
  • nope, Firefox and eclipse manage to eat it all :) – Aiden Bell Mar 15 '13 at 15:06
  • Does it really get killed? You may connect with strace shortly before the crash and have a look at what's really happening. – Hauke Laging Mar 15 '13 at 15:23
  • 1
    Are all lines the same length? Then I would propose a sort-in-place, i.e. mmap the file and quicksort it. – ott-- Mar 16 '13 at 17:19
  • 1
    I had the same problem when I used /run/shm as my /bigdisk for storing sorts temp files. It seems the kernel considers the ram disk in /run/shm as more important, so when sort was writing to it and memory ran out, the kernel killed sort. Using a sluggish VM hard disk instead solved it. – drevicko Apr 22 '15 at 2:14
3

Don't use -S 10G, it is far too much (and probably not doing what you think). OOMkiller launching does mean your system is using all of it's memory.

According to the algorithm used by sort, it will use memory according to what is available: half of the biggest number between TotalMem/8 and AvailableMem.

So, for example, if you have 4 GB of available mem (out of 8 GB), sort will use 2GB of RAM. It should also create many 2 GB files in /bigdisk and finally merge-sort them.

|improve this answer
  • 2
    To be fair, sort is killed off if I use -S or not. Both top and /proc/meminfo report I have free memory on death. The machine has 16G of ram and only 3 is actually in use when the sort process is launched. Manually using split, sort on the parts and them sort -m worked with no memory issues. – Aiden Bell Mar 18 '13 at 23:25
  • @AidenBell I just thought of something: are you using a 32 bit version of sort? Check with file $(which sort). – Totor Mar 19 '13 at 0:07
  • nope. 64-bit. I'm beginning to think it might be an actual gasp bug somewhere. – Aiden Bell Mar 19 '13 at 18:29
1

Making an answer from my comment:

I had the same problem when I used /run/shm as my /bigdisk for storing sort temp files. /run/shm is a ram disk, so when sort needed to cache partial results on disk (which it does when memory is almost full), memory ran out. The kernel killed sort, as it was the process using the most memory.

Using a location stored on a physical disk instead of the ram disk solved it.

|improve this answer
  • The OP used -T /bigdisk for this exact reason, but still ran out of memory. – roaima May 5 '15 at 10:30
  • 1
    @roaima, there's nothing in the OP's question that specifies that /bigdisk is not at least in part backed by RAM. drevicko makes it clear that his/her answer's premise is that /bigdisk is RAM-backed. It will certainly help people with the same problem even if not the OP, so doesn't deserve a downvote I would say. – Stéphane Chazelas May 5 '15 at 10:38
  • @StéphaneChazelas fair point and one that I'd not considered. Can you undo my downvote (I can't)? – roaima May 5 '15 at 10:40
  • @roaima, I edited the answer so you should be able to cancel your downvote now (note that I'm not a moderator). – Stéphane Chazelas May 5 '15 at 10:45
-1

Try setting vm.overcommit_memory = 1: «…

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

…»

And do use swap.

|improve this answer
  • The OOM killer is only triggered when "it actually runs out" of memory. So this settings won't solve anything. It might have if the Out of mem message had come from sort and not from the kernel. – Totor Mar 16 '13 at 18:12
  • RTFM: „… The OOM killer can be completely disabled with the following command. This is not recommended for production environments, because if an out-of-memory condition does present itself, there could be unexpected behavior depending on the available system resources and configuration. … anything from a kernel panic to a hang depending on the resources available to the kernel at the time of the OOM condition. sysctl vm.overcommit_memory=2 echo "vm.overcommit_memory=2" >> /etc/sysctl.conf …” ­— oracle.com/technetwork/articles/servers-storage-dev/… – poige May 5 '15 at 11:33
  • I fail to see the relevance of your comment. Disabling OOM-killer will not solve the problem, which is: sort uses/needs too much memory. With overcommit_memory=2, sort will just fail to malloc() and will very probably fail to sort, then. – Totor May 5 '15 at 23:19
  • mine answer wasn't =2. It was a cite showing that settings of this option has direct relation to OOM-killing. If you want to know the subject better, well, okay, that would suit your purpose (hopefully): lwn.net/Articles/317814 – poige May 6 '15 at 2:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.