0

Today my system crashed with:

Feb 24 20:07:34 kernel: [46929872.935456]  [<ffffffff81141073>] ? out_of_memory+0x473/0x4b0

But I was able to still login to SHH and open the top, many aplications crashed and refused to start, but in the top only 50% of my memory was marked as "used" and another 50% of the memory was marked as "cached Mem".

I always thought that cached memory is used to cache files to reduce write/reads from drive and when program needs to allocate more memory, some files are dropped from the cache and used for running programs. But apparently that is not true, because 50% of my total memory was used for cache and system still crashed while it should just drop some cache.

Can someone give me some deeper explanation of how cached memory works ? Thanks.

1

Perhaps it was write-cache data. Dropping that would require writing it to its ultimate on-disk location first. And if you have a failing disk that isn't acknowledging any write operations neither as successfully completed nor as failed, those caches cannot be dropped.

I've seen some old disks fail like that: they won't report an actual failure, but just get stuck in a loop of infinite retries. It probably should not happen, but apparently it does.

  • So basically you mean that there was a huge memory requierement so the drive did not have enough time to wite everything that was in cached memory ? – Welite Feb 25 '18 at 18:06
  • I mean that the disk stopped accepting new data for writing (or maybe a NFS server became unreachable or something) but the applications kept producing new data that needed to be stored. Check the health of your disks (with smartctl or similar): it is possible that you have a failing disk. – telcoM Feb 26 '18 at 6:09
  • I dont have problems with the disk, I have problem that my processes use only about 15 GB of memory and another 17 GB is used as "cached memory" and I ask why the cached memory is not cleared when processes demand more memory for processing so my processes crashed and syslog said that the system run out of memory. – Welite Feb 26 '18 at 12:23
  • OK, if you say so. Then you'll need to set up the kdump facility, reproduce the problem, and then analyze the crash dump and/or the logs to find out what actually went wrong. I was just saying that a disk problem may sometimes be the root cause of an out-of-memory situation. – telcoM Feb 26 '18 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.