OpenSUSE 42.3, Kernel 4.4.175-89-default
Running memory bandwidth intensive applications I realised the following behaviour: The application uses ~55% of the physical memory of a NUMA system with 2 nodes. The application is parallelized using OpenMP, but without accounting for NUMA. So it relies on page migration to achieve a somewhat decent execution speed.
At around 180 iterations, I cleared caches manually using
# echo 3 >| /proc/sys/vm/drop_caches
The result is an immediate performance improvement. What prevents the system from doing proper page migration before I cleared the caches manually?