I have a strange problem that looks like a Memory Cache Leak (not a memory leak) leading to a slow down of a program.
Let me set the stage first. Reproducibly (using ganglia to monitor), on a cluster I have noticed that the cached memory is increasing, relatively slowly. When it becomes large, something like 2/3 of the total memory (Intel Gold with 32 cores & 192Gb) a program is running slower by about a factor of ~1.5. If I clear the cache and sync the disc (I have not tested which matter) with "
sync ; echo 3 > /proc/sys/vm/drop_caches" the speed of the program increases back (~1.5 times faster).
The issue is associated with I/O -- only CPUs that are doing any I/O shows the cache leak. The program is doing a fair amount of I/O, but not massive amounts (10-40 Mb). I compile using ifort with -assume buffered_io. It appears that the I/O leaves cached files at the end, effectively a "cache memory leak".
There are multiple parts to this issue, both why this "leak" is occurring, how to avoid it (e.g. brute force cron a cache cleaning every 3 h) as well as how/why the program slows down so much (inefficiency of vector operations for non-contiguous memory as a speculation I am not sure I believe).
Any and all suggestions both for cures as well as ways to better understand it would be useful.
N.B., it is a big program, so reproducing this with a small code may not be relevant.