4

I'm using Ubuntu 20.04.2 LTS with kernel 5.4.0-70-generic. Shared memory keeps piling up until system hangs as it doesn't have any memory left.

I'm using Prometheus + Grafana to monitor my system resources and from their graph it's visible that it takes approximately 4-7 days since restart until shared memory reached 20 GB. But it doesn't stop there, it keeps growing until I make another restart. Screenshot from Grafana

Same amount of shared memory usage can be seen from free -h command. Screenshot of free -h

The usage of tmpfs seems to be low: Screenshot of df -h | grep tmpfs

Shared memory keeps growing even if I close all the apps and not using the computer. What could be the cause of such behavior? How could I start investigating the problem? How could I find out which processes use that much shared memory?

$ ipcs -m --human

------ Shared Memory Segments --------
key        shmid      owner      perms      size       nattch     status      
0x00000000 3145729    martsraits 600          256M     2          dest         
0x00000000 2588677    martsraits 600          512K     2          dest         
0x00000000 1245191    martsraits 600          512K     2          dest         
0x00000000 5832713    martsraits 600          512K     2          dest         
0x00000000 10         martsraits 600          512K     2          dest         
0x00000000 1212427    martsraits 600          512K     2          dest         
0x00000000 12         martsraits 600          512K     2          dest         
0x00000000 1212429    martsraits 600           64M     2          dest         
0x00000000 32785      martsraits 600          512K     2          dest         
0x00000000 2064402    martsraits 600          512K     2          dest         
0x00000000 5537814    martsraits 600          512K     2          dest         
0x00000000 1114138    martsraits 600          512K     2          dest         
0x00000000 8945695    martsraits 600         23,3K     2          dest         
0x00000000 1507360    martsraits 600          512K     2          dest         
0x00000000 2916388    martsraits 600          512K     2          dest         
0x00000000 4816933    martsraits 606          8,2M     2          dest         
0x00000000 4816934    martsraits 606          8,2M     2          dest         
0x00000000 4816935    martsraits 600          128M     2          dest         
0x00000000 3670057    martsraits 600          512K     2          dest         
0x00000000 3309611    martsraits 600          512K     2          dest         
0x00000000 1441844    martsraits 600          512K     2          dest         
0x00000000 2555960    martsraits 600            8M     2          dest         
0x00000000 917564     martsraits 600          512K     2          dest         
0x00000000 3899453    martsraits 600          512K     2          dest     

The sum of size column in ipcs -m is only 500M.

9
  • looks like something's leaking a lot of shared memory. find out what it is and fix it (or replace it if it can't be fixed). you can find out what's using shmem with ipcs -m
    – cas
    Apr 13, 2021 at 10:58
  • I update my post, added ipcs -m output. What conclusions can I make from this? To me it seems that it doesn't show what is using most of the shared memory as the sum of ipcs -m is only 500M but shared memory usage is more than 20G.
    – martsraits
    Apr 19, 2021 at 6:38
  • There's nothing obvious there, maybe try running it as root (otherwise you only get shmem from processes owned by your uid). Once you've found a shmid that's using a lot of memory, you can use ipcs -p match it to a process id (e.g. if the shmid is 2555960, run ipcs -p | awk '$1 == 2555960').
    – cas
    Apr 19, 2021 at 7:11
  • 1
    Does 'ls /dev/shm/*' give any useful information?
    – C. M.
    Apr 21, 2021 at 9:14
  • 1
    Thanks to everyone for trying to help. The problem disappeared without me doing anything special. So I don't know what caused it and don't know what made it go away. It could have been some software update which fixed it but I don't know for sure.
    – martsraits
    Feb 2, 2022 at 15:25

1 Answer 1

0

I've run into this just now on an embedded system, and my issue was tmpfs - specifically, logs are stored in there (I need to change that), and they took up too much ram. And yes, in the output of free it showed up as taking up shared memory.

1
  • That's what I get for skimming the question... just now noticed you looked in tmpfs. Eh, leaving this answer for future people who also skim the question.
    – jaskij
    Jan 2, 2023 at 14:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .