6

I have a machine that is intended for general use and which I also used to run a QEMU virtual machine. Because the virtual machine should be as performant as possible, I want to back the VM memory with hugepages, ideally 1GB hugepages. The machine has 32GB of ram and I want to provide 16 to the VM. The problem is that during my normal use of the machine, I might need to use all 32GB, so allocating the 16G of hugepages at boot is not an option.

To work around this I have a hook script that allocates the 16G of hugepages when the VM boots. As you might expect, for 1GB hugepages, this fails if the host machine has been used for any amount of time (it seems to work reliably with 2M hugepages though this is not ideal).

What I don't understand is exactly why this is happening. For example, I can open several applications (browser window, code editor, etc just to force some fragmentation for testing), then close them so that only my desktop is open. My memory usages in this case is around 2.5G/32G.

Is there really no way that the kernel can find 16 1G-pages of contiguous aligned memory, out of the remaining 30G of RAM, that seems like very high fragmentation. Furthermore, I can run 

$ sudo tee /proc/sys/vm/compact_memory <<<1

to try to defrag the RAM, but even then, I have never successfully allocated 16 1G hugepages for the VM. This in particular is really shocking to me, since after defragging only 2.5G of RAM the remaining 30G still isn't contiguous or aligned.

What I'm misunderstanding about this process? Does this seem like expected behavior? Additionally, is there any way to check if compact_memory actually did anything? I don't see any output in dmesg or similar after running that command.

Improve this question
5
  • 1
    Do you mean to say 16 contiguous 1G pages. This is half your memory, with no fragmentation. Do you not just need 16 1G pages. Where each page is contiguous?
    – ctrl-alt-delor
    Jun 22 '18 at 17:01
  • Yes that is what I mean: each 1GB page needs to be contiguous for it to be allocated but the entire group of 16 pages does not need to be contiguous (that is how I think huge pages work)
    – Max Ehrlich
    Jun 23 '18 at 0:50
  • 1
    may or may-not be relevant: lwn.net/Articles/211505
    – ctrl-alt-delor
    Jun 23 '18 at 8:31
  • 1
    Are you sure your processor supports 1G pages?
    – crass
    Jun 23 '18 at 19:51
  • Yes 1G huge pages are supported and I've successfully allocated them at boot
    – Max Ehrlich
    Jun 24 '18 at 13:39
0

You can pre-allocate the Huge Pages at the host boot time because on-line allocation may not work (due to issues you mention).

Something like this might do the trick, but you'll have to reboot your host:

GRUB_CMDLINE_LINUX_DEFAULT="hugepagesz=1GB default_hugepagesz=1G hugepages=16"

Check out this great FOSDEM 2019 talk by Fernando Laudares at around minute 20 for a description of both on-line and boot-time methods for allocating huge pages: https://www.youtube.com/watch?v=jTJ_X3fJ1Ik

Slides here: https://fosdem.org/2019/schedule/event/hugepages_databases/attachments/slides/3038/export/events/attachments/hugepages_databases/slides/3038/Huge_pages_and_databases___FOSDEM19.pdf

Improve this answer
1
  • 1
    Yeah I know and as I stated in the question, I might want to use all 32GB during general use so I can't commit to losing any memory at boot time, I need to allocate the hugepages only when I need the VM
    – Max Ehrlich
    Jul 18 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.