I have a strange problem that looks like a Memory Cache Leak (not a memory leak) leading to a slow down of a program.

Let me set the stage first. Reproducibly (using ganglia to monitor), on a cluster I have noticed that the cached memory is increasing, relatively slowly. When it becomes large, something like 2/3 of the total memory (Intel Gold with 32 cores & 192Gb) a program is running slower by about a factor of ~1.5. If I clear the cache and sync the disc (I have not tested which matter) with "sync ; echo 3 > /proc/sys/vm/drop_caches" the speed of the program increases back (~1.5 times faster).

The issue is associated with I/O -- only CPUs that are doing any I/O shows the cache leak. The program is doing a fair amount of I/O, but not massive amounts (10-40 Mb). I compile using ifort with -assume buffered_io. It appears that the I/O leaves cached files at the end, effectively a "cache memory leak".

There are multiple parts to this issue, both why this "leak" is occurring, how to avoid it (e.g. brute force cron a cache cleaning every 3 h) as well as how/why the program slows down so much (inefficiency of vector operations for non-contiguous memory as a speculation I am not sure I believe).

Any and all suggestions both for cures as well as ways to better understand it would be useful.

N.B., it is a big program, so reproducing this with a small code may not be relevant.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.