Does QEMU on Linux Ubuntu 20.04.1 x86_64 have a memory leak?

Question

We have a testbed for an OSv project that runs (5.15.0-72-generic - 20.04.1-Ubuntu - x86_64) the same instances a lot of time. The script for the execution of a single run is very simple and follows:

while [ $x -le $t ]
   do
   ./scripts/capstan_run.sh "$delay"
   now="$(date +'%d%m%Y-%H%M%S')"
   ./scripts/stats.sh > stats/"$x"_"$delay"_stats_"$now".txt & PID=$!
   sleep "$delay" #sleep delay mills for the execution
   kill $PID ; wait $PID 2>/dev/null
   echo "Delay $delay before fetches"
   sleep "$delay" #sleep delay mills before fetch files
   ./scripts/fetch_files.sh "$delay"
   ./scripts/shutdown_vm.sh

   ((x++))
   done

capstun_run.sh initiates the simulation with containers executing on the QEMU virtualization layer. It then sleeps and retrieves files from the instances. The shutdown.sh script terminates QEMU:

killall qemu-system-x86_64

We observe between runs an increment of used memory. It is constant and never decreases. The server has 126G of RAM and 24 CPUs.

For example we observe that used memory starts from 8% and arrives at 12%, with an increment of 0.1%.

Date                Memory      Disk        CPU
...
07/04/2023-163242       12.03%      27%     15.00%      
07/04/2023-163247       12.03%      27%     16.00%      
07/04/2023-163252       12.03%      27%     16.00%      
07/04/2023-163257       12.03%      27%     16.00%      
07/04/2023-163303       12.03%      27%     16.00%      
07/04/2023-163308       12.04%      27%     16.00%      
07/04/2023-163313       12.03%      27%     16.00%      
07/04/2023-163318       12.04%      27%     15.00%      
07/04/2023-163323       12.04%      27%     16.00%      
07/04/2023-163328       12.04%      27%     16.00%      
07/04/2023-163334       12.04%      27%     16.00%      
07/04/2023-163339       12.04%      27%     16.00%      
07/04/2023-163344       12.06%      27%     16.00%      
07/04/2023-163349       12.08%      27%     16.00%      
07/04/2023-163354       12.09%      27%     16.00%      
07/04/2023-163359       12.09%      27%     15.00%      
07/04/2023-163405       12.09%      27%     15.00%      
07/04/2023-163410       12.09%      27%     15.00%      
07/04/2023-163415       12.09%      27%     15.00%      

Is there any memory leak in QEMU?

=== UPDATE ===

The stats.sh computes the used % mem in this manner:

free -m | awk 'NR==2{printf "%.2f%%\t\t", $3*100/$2 }

So I think there is an error, because it's the "used/total*100" e does not include the cache.

Is my evaluation correct?

Answer 1

If the QEMU process has terminated, then the kernel has freed all memory that it was using. Any memory leak within its own address space would not survive the process' own termination. It's more likely that you're seeing memory which is being used for caching. This memory is used to improve performance and can be reclaimed at any time if the need arises. Memory used for caching will not generally increase memory pressure.

It's possible, although unlikely, that QEMU is allocating memory outside of its virtual address space, such as in tmpfs (which, while it counts as cached memory, cannot be dropped). There are various ways a process can use up resources that are not automatically freed, or cause another process to leak resources, for example by failing to close file descriptors. Without knowing more about how you are using QEMU, it's not possible to determine if this is the case, but it's unlikely.

---

You should be using the "available" field from free, not the "used" field. While the latter subtracts caches, some memory that is considered cached memory cannot be reclaimed. The former field, added in 2014, is a more accurate representation of memory that is genuinely usable:

Many load balancing and workload placing programs check /proc/meminfo to estimate how much free memory is available. They generally do this by adding up "free" and "cached", which was fine ten years ago, but is pretty much guaranteed to be wrong today.

It is wrong because Cached includes memory that is not freeable as page cache, for example shared memory segments, tmpfs, and ramfs, and it does not include reclaimable slab memory, which can take up a large fraction of system memory on mostly idle systems with lots of files.

You should replace your awk line with this instead:

awk '$1 == "MemTotal:" {total=$2} $1 == "MemAvailable:" {avail=$2} END {printf "%.2f%%\t\t", (avail/total)*100}' < /proc/meminfo

In addition to using available memory for the calculation, it uses /proc/meminfo rather than free, which has a stable (predictable) format and intended to be machine-readable.