0

Today my system crashed with:

Feb 24 20:07:34 kernel: [46929872.935456]  [<ffffffff81141073>] ? out_of_memory+0x473/0x4b0

But I was able to still login to SHH and open the top, many aplications crashed and refused to start, but in the top only 50% of my memory was marked as "used" and another 50% of the memory was marked as "cached Mem".

I always thought that cached memory is used to cache files to reduce write/reads from drive and when program needs to allocate more memory, some files are dropped from the cache and used for running programs. But apparently that is not true, because 50% of my total memory was used for cache and system still crashed while it should just drop some cache.

Can someone give me some deeper explanation of how cached memory works ? Thanks.

1 Answer 1

1

Perhaps it was write-cache data. Dropping that would require writing it to its ultimate on-disk location first. And if you have a failing disk that isn't acknowledging any write operations neither as successfully completed nor as failed, those caches cannot be dropped.

I've seen some old disks fail like that: they won't report an actual failure, but just get stuck in a loop of infinite retries. It probably should not happen, but apparently it does.

4
  • So basically you mean that there was a huge memory requierement so the drive did not have enough time to wite everything that was in cached memory ?
    – Welite
    Feb 25, 2018 at 18:06
  • I mean that the disk stopped accepting new data for writing (or maybe a NFS server became unreachable or something) but the applications kept producing new data that needed to be stored. Check the health of your disks (with smartctl or similar): it is possible that you have a failing disk.
    – telcoM
    Feb 26, 2018 at 6:09
  • I dont have problems with the disk, I have problem that my processes use only about 15 GB of memory and another 17 GB is used as "cached memory" and I ask why the cached memory is not cleared when processes demand more memory for processing so my processes crashed and syslog said that the system run out of memory.
    – Welite
    Feb 26, 2018 at 12:23
  • OK, if you say so. Then you'll need to set up the kdump facility, reproduce the problem, and then analyze the crash dump and/or the logs to find out what actually went wrong. I was just saying that a disk problem may sometimes be the root cause of an out-of-memory situation.
    – telcoM
    Feb 26, 2018 at 12:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .