8

On a server, the linux kernel at boot informs about the RAM setup. It indicates that of the physical 512GiB RAM (536409480kiB), only roughly 503GiB RAM (527942676kiB) are available.

root@ada:~# dmesg | grep Memory:
[    5.891484] Memory: 527942676K/536409480K available (10252K kernel code, 1241K rwdata, 3320K rodata, 1592K init, 2272K bss, 8466804K reserved, 0K cma-reserved)

Some memory being unavailable is expected given the IO regions reserved by the bios.

root@ada:~# dmesg | grep reserved
[    0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000004f66f000-0x0000000057677fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000006cdcf000-0x000000006efcefff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000507ff80000-0x000000507fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000607ff80000-0x000000607fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000707ff80000-0x000000707fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000807ff80000-0x000000807fffffff] reserved

which however do not amount to more than 100MiB.

I wonder how to investigate what causes this memory to be reserved? What is the cause?

Without understanding the purpose for the reserved memory it seems simply like a ~9GiB loss of memory. Given that the system acts as a virtualization host, this "loss" is excarbated as each virtualized guest system in turn has a similar fraction of its dedicated RAM also "reserved".

Since other question have suggested that such memory could be reserved for "shared memory" of a graphic card, I looked this up, yet the adapter present only seems to use ~50MiB at most.

root@ada:~# lspci  | grep -i vga
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
root@ada:~# lspci -s 03:00.0 -vvv
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04) (prog-if 00 [VGA controller])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0 (4000ns min, 8000ns max)
        Interrupt: pin A routed to IRQ 243
        NUMA node: 0
        Region 0: Memory at eb000000 (32-bit, prefetchable) [size=16M]
        Region 1: Memory at f9808000 (32-bit, non-prefetchable) [size=16K]
        Region 2: Memory at f9000000 (32-bit, non-prefetchable) [size=8M]
        [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: mgag200
        Kernel modules: mgag200

Update

Here is the complete dmesg output

Update It is a Dell Poweredge Server, that boots in BIOS mode (not UEFI)

Update

Here is the output of /proc/iomem

Update

Here the evaluated output of /proc/iomem

root@ada:~# cat /proc/iomem | tr [a-z] [A-Z] | while IFS='-: ' read AD1 AD2 REST;
> do echo "$(( $(echo "obase=10; ibase=16; ( $AD2 - $AD1 ) " | bc) >> 20))MB for  $REST" ; 
> done | sort -h  

[...]
14MB for  ACPI NON-VOLATILE STORAGE                                                                                                        
15MB for  0000:03:00.0
15MB for  MGADRMFB_VRAM
15MB for  PCI BUS 0000:02
15MB for  PCI BUS 0000:03
33MB for  RESERVED
128MB for  RESERVED
207MB for  PCI BUS 0000:20
207MB for  PCI BUS 0000:40
207MB for  PCI BUS 0000:60
207MB for  PCI BUS 0000:80
207MB for  PCI BUS 0000:A0
207MB for  PCI BUS 0000:C0
207MB for  PCI BUS 0000:E0
255MB for  PCI MMCONFIG 0000 [BUS 00-FF]
255MB for  PNP 00:00
315MB for  PCI BUS 0000:00
343MB for  SYSTEM RAM
511MB for  RESERVED
543MB for  RESERVED
1269MB for  SYSTEM RAM
63475MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
65535MB for  SYSTEM RAM
915967MB for  PCI BUS 0000:00
915967MB for  PCI BUS 0000:20
915967MB for  PCI BUS 0000:40
915967MB for  PCI BUS 0000:60
915967MB for  PCI BUS 0000:80
915967MB for  PCI BUS 0000:A0
915967MB for  PCI BUS 0000:C0
915967MB for  PCI BUS 0000:E0

The output of dmidecode on the system is (since I expect that to be fitting in the context of RAM availability):

Getting SMBIOS data from sysfs.
SMBIOS 3.2 present.
81 structures occupying 6778 bytes.
Table at 0x6E8AD000.

Handle 0xDA00, DMI type 218, 11 bytes
OEM-specific Type
        Header and Data:
                DA 0B 00 DA B2 00 17 20 0E 10 03

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: Dell Inc.
        Version: 1.14.3
        Release Date: 07/17/2020
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 0 MB
        Characteristics:
                ISA is supported
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
                UEFI is supported
            BIOS Revision: 1.14
    
    Handle 0x0100, DMI type 1, 27 bytes
    System Information
            Manufacturer: Dell Inc.
            Product Name: PowerEdge R7425
            Version: Not Specified
            Serial Number: XXXXXX
            UUID: XXXXXXXX-XXXX-4a10-8048-c3c04f593533
            Wake-up Type: Power Switch
            SKU Number: SKU=NotProvided;ModelName=PowerEdge R7425
        Family: PowerEdge

The complete dmidecode with possible information regarding the mysterious unavailability of roughly 9GiB of RAM can be seen here https://pastebin.com/nHYyuH7h

enter image description here

27
  • Can you post your dmesg? Commented Oct 30, 2020 at 9:42
  • @ArtemS.Tashkinov, sure I can , it is however very long, would that be appropriate to put it into the question (as it would become instantly very very long) ?
    – fraleone
    Commented Oct 30, 2020 at 9:54
  • You could use pastebin.com. :-) And please set expiration just in case. Maybe you have some data in it you don't want to remain public forever. Commented Oct 30, 2020 at 9:55
  • 1
    Please upload /proc/iomem and the output of https://github.com/madrisan/linux-iomem/blob/master/iomem.sh Commented Oct 30, 2020 at 13:40
  • 1
    I wonder if it could be GPU mapped memory. Commented Nov 14, 2020 at 10:42

1 Answer 1

5

I think I know the answer (not 100% sure though).

Linux has struct page structure associated with memory pages. Size of this structure varies between architectures, on 32-bit x86 Linux it's 40 bytes, on 64-bit, I beleive, due to bigger pointer sizes, it's 64 bytes.

You can find it here: https://elixir.bootlin.com/linux/v6.1/source/include/linux/mm_types.h

Those structures could be organized in different ways (see https://lwn.net/Articles/789304/), but ultimately, every memory page on your system gets one such structure.

That means that kernel needs to reserve 64 bytes for each 4096 bytes (page size) And that means, for 512G it will reserve ((512×1024³)/4096)×64 = 8589934592 bytes, or 8388608K, or 8G exactly.

Add this to the code, ro/rwdata, init, bss sections, and you get 8407285K, really close to 8466804K we are trying to explain here.

You can also boot your server with, say, mem=256G kernel parameter. This will limit the available memory size to 256G, and you will find out that the reserved size shrunk nearly by the factor of 2. That also speaks for the theory above, and, unfortunately, given how well the information is packed within this structure and how essential it is, I don't think there is a way to make this memory available for other uses.

1
  • Using hugepages (there are 2M and 1G variants) might be able to reduce your page table overhead. QEMU (and likely the other major virtualization systems on linux) can take advantage of hugepages--on the host side. On the guest side, you can try to get the guest apps to use hugepages as well. Note that when running a VM, you pay the page table overhead twice, so even using it only on the host side will be a benefit. See -mem-path for QEMU Commented Apr 17, 2023 at 22:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .