We are setting up a Linux based benchmarking cluster. Each node is going to be a headless, diskless machine, booted via tftp, the OS copied to a local ramdrive, and the same ramdrive is used as local drive by the benchmarked application. My problem is the following:

The machines have 2 CPUs, each has its own memory banks and have 4 memory channels to those banks (so the banks are populated with multiples of 4 memory chips to get maximum memory throughput). If I can't control which memory regions are used for the ramdisk, then it may be possible that it will be created in a region that is on one single channel, and uses all memory on that chip. Which means that when my application is running then the threads running on the CPU from whose memory bank the ramdisk was taken would have 25% less memory bandwidth to their "local" memory than the threads on the other cpu. That would be BAD. Hence the desire to control which memory regions are used by the ramdisk.

Or is this a non-issue and I can just trust the memory controller to lay out contiguous memory addresses in a strided fashion among the chips on the 4 channels? That would make sense, since that would maximize the memory bandwidth when pulling in large chunks of memory into cache.

I just don't know how these things work and would appreciate some enlightment...

  • Wouldn't numa zones help you out here? – Lmwangi Jul 25 '16 at 18:28

Are you using tmpfs for the ramdisk support?

If so, you can use the parameter mpol to assign to a specific NUMA node (Please note that your kernel needs CONFIG_NUMA enabled).

mount -t tmpfs -o size=32g,mpol=prefer:0 tmpfs /mnt/ramdisk would create a 32GB ramdisk prefered on NUMA Node 0 mounted at /mnt/ramdisk

The documentation of tmpfs is quite good: https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.