1

We have a Proxmox VE (kernel 5.3) cluster running some IT services using LXC, one of which is used as a gateway for the whole intranet. It's set up with a Debian image and does forwarding & NAT using iptables. We're frequently seeing absurdly high memory usage in this container, and it just died (memory and swap both full).

htop reports full memory usage, but the processes listed (the RES column) sums up to less than 100 MiB:

htop

Host cgroup controller reports full usage for memory.usage_in_bytes:

root@proxmox:/sys/fs/cgroup/memory/lxc/100# cat memory.kmem.usage_in_bytes
47140864
root@proxmox:/sys/fs/cgroup/memory/lxc/100# cat memory.usage_in_bytes
1066528768

Writing 3 into /proc/sys/vm/drop_caches on the host doesn't have any visible effect in the container.

Where is the "extra" memory? How to remediate this?

2
  • I am not sure how you are doing your memory calculations, just the RES amounts from the top ruby processes are consuming 580MB of RAM (57.2% of your total). The RES column is represented in KiB by default. I believe your issue is that you have allowed those ruby processes to allocate more virtual memory (VIRT) then you currently have available (1.3G vs 1G). Check that there is not a memory leak in your ruby processes or run less ruby processes or get more RAM. Commented May 1, 2020 at 18:12
  • @GracefulRestart The top ruby items are one single process (htop shows threads as green and processes (main threads) as white). Also virtual memory isn't something important here.
    – iBug
    Commented May 2, 2020 at 2:50

1 Answer 1

0

I ran into a similar problem.
Every cronjob that ran in my LXD container created a new transient user slice, like this one:

● session-c48037.scope - Session c48037 of user apache
   Loaded: loaded (/run/systemd/transient/session-c48037.scope; transient)
Transient: yes
   Active: active (abandoned) since Tue 2023-05-02 16:24:02 CEST; 13min ago
    Tasks: 0
   Memory: 352.0K
   CGroup: /user.slice/user-48.slice/session-c48037.scope

May 02 16:24:02 pos34 systemd[1]: Started Session c48037 of user apache.

For reasons I don't understand yet, the sessions were never cleaned up after the cronjobs were done.
The kilobytes of memory added up over time and caused very high memory consumption by the kernel itself, reported as SUnreclaim in /proc/meminfo.
So while this doesn't answer "How to remediate this?", it might be the answer to "Where is the extra memory?".

This was observed on a Ubuntu 20.04 host system with a RockyLinux 8.7 LXD guest. LXD version 4.0.9 installed via Snap.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .