4

We have a testbed for an OSv project that runs (5.15.0-72-generic - 20.04.1-Ubuntu - x86_64) the same instances a lot of time. The script for the execution of a single run is very simple and follows:

while [ $x -le $t ]
   do
   ./scripts/capstan_run.sh "$delay"
   now="$(date +'%d%m%Y-%H%M%S')"
   ./scripts/stats.sh > stats/"$x"_"$delay"_stats_"$now".txt & PID=$!
   sleep "$delay" #sleep delay mills for the execution
   kill $PID ; wait $PID 2>/dev/null
   echo "Delay $delay before fetches"
   sleep "$delay" #sleep delay mills before fetch files
   ./scripts/fetch_files.sh "$delay"
   ./scripts/shutdown_vm.sh

   ((x++))
   done

capstun_run.sh initiates the simulation with containers executing on the QEMU virtualization layer. It then sleeps and retrieves files from the instances. The shutdown.sh script terminates QEMU:

killall qemu-system-x86_64

We observe between runs an increment of used memory. It is constant and never decreases. The server has 126G of RAM and 24 CPUs.

For example we observe that used memory starts from 8% and arrives at 12%, with an increment of 0.1%.

Date                Memory      Disk        CPU
...
07/04/2023-163242       12.03%      27%     15.00%      
07/04/2023-163247       12.03%      27%     16.00%      
07/04/2023-163252       12.03%      27%     16.00%      
07/04/2023-163257       12.03%      27%     16.00%      
07/04/2023-163303       12.03%      27%     16.00%      
07/04/2023-163308       12.04%      27%     16.00%      
07/04/2023-163313       12.03%      27%     16.00%      
07/04/2023-163318       12.04%      27%     15.00%      
07/04/2023-163323       12.04%      27%     16.00%      
07/04/2023-163328       12.04%      27%     16.00%      
07/04/2023-163334       12.04%      27%     16.00%      
07/04/2023-163339       12.04%      27%     16.00%      
07/04/2023-163344       12.06%      27%     16.00%      
07/04/2023-163349       12.08%      27%     16.00%      
07/04/2023-163354       12.09%      27%     16.00%      
07/04/2023-163359       12.09%      27%     15.00%      
07/04/2023-163405       12.09%      27%     15.00%      
07/04/2023-163410       12.09%      27%     15.00%      
07/04/2023-163415       12.09%      27%     15.00%      

Is there any memory leak in QEMU?

=== UPDATE ===

The stats.sh computes the used % mem in this manner:

free -m | awk 'NR==2{printf "%.2f%%\t\t", $3*100/$2 }

So I think there is an error, because it's the "used/total*100" e does not include the cache.

Is my evaluation correct?

1
  • See this comprehensive answer to a question of mine.
    – Vorac
    May 25, 2023 at 13:47

1 Answer 1

10

If the QEMU process has terminated, then the kernel has freed all memory that it was using. Any memory leak within its own address space would not survive the process' own termination. It's more likely that you're seeing memory which is being used for caching. This memory is used to improve performance and can be reclaimed at any time if the need arises. Memory used for caching will not generally increase memory pressure.

It's possible, although unlikely, that QEMU is allocating memory outside of its virtual address space, such as in tmpfs (which, while it counts as cached memory, cannot be dropped). There are various ways a process can use up resources that are not automatically freed, or cause another process to leak resources, for example by failing to close file descriptors. Without knowing more about how you are using QEMU, it's not possible to determine if this is the case, but it's unlikely.


You should be using the "available" field from free, not the "used" field. While the latter subtracts caches, some memory that is considered cached memory cannot be reclaimed. The former field, added in 2014, is a more accurate representation of memory that is genuinely usable:

Many load balancing and workload placing programs check /proc/meminfo to estimate how much free memory is available. They generally do this by adding up "free" and "cached", which was fine ten years ago, but is pretty much guaranteed to be wrong today.

It is wrong because Cached includes memory that is not freeable as page cache, for example shared memory segments, tmpfs, and ramfs, and it does not include reclaimable slab memory, which can take up a large fraction of system memory on mostly idle systems with lots of files.

You should replace your awk line with this instead:

awk '$1 == "MemTotal:" {total=$2} $1 == "MemAvailable:" {avail=$2} END {printf "%.2f%%\t\t", (avail/total)*100}' < /proc/meminfo

In addition to using available memory for the calculation, it uses /proc/meminfo rather than free, which has a stable (predictable) format and intended to be machine-readable.

4
  • in that sense, it'd make sense to check whether the behaviour changes when instead of just killing qemu, the monitor console is used to send a quit command. May 22, 2023 at 7:39
  • yes probably there is an errore in the formula that computes the used %mem. I've just updated the post.
    – robob
    May 22, 2023 at 7:51
  • All good advice. Note that if for whatever reason you want to reclaim the memory used for caching you can echo 3 | sudo tee /proc/sys/vm/drop_caches. You'll see the effect instantly in the numbers reported by free - but in general, the caches help you (e.g. the fs is reading data from them instead of the drives).
    – ttsiodras
    May 28, 2023 at 16:44
  • There's never any reason to drop caches unless you're testing the Linux mm subsystem, performing benchmarks that caching would interfere with, or want data to be removed from memory for obscure and highly-specific security reasons (e.g. you have a disk that uses hardware encryption and you want to prevent previously accessed files from being accessible via a cold boot attack).
    – forest
    Aug 2, 2023 at 8:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .