4

I have two x86_64 kernels compiled on the same machine against the same code (4.15.0 in Linus' source tree).

The config files were produced by running make localmodconfig against that source, using different, larger original config files coming from different distros: Arch and Slackware respectively. I'll nickname them

arch config

and

slk config

for that reason.

The issue: running cat /proc/meminfo consistently reports about 55-60 MB more in the MemTotal field for arch than for slk:

MemTotal: 32600808 kB for arch

vs

MemTotal: 32544992 kB for slk

I say 'consistently' because I've tried the experiment with the same config files against earlier versions of the source (a bunch of the 4.15-rc kernels, 4.14 before that, etc., rolling over from one source to the next with make oldconfig).

This is reflected in the figures reported by htop, with slk reporting ~60MB less usage on bootup than arch. This is consistent with the htop dev's explanation of how htop's used memory figures are based on MemTotal.

My question is: any suggestions for which config options I should look at that would make the difference?

I of course don't mind the 60MB (the machine the kernels run on has 32 GB..), but it's an interesting puzzle to me and I'd like to use it as a learning opportunity.

Memory reporting on Linux is discussed heavily on these forums and outside in general, but searching for this specific type of issue (different kernels / same machine => different outcome in memory reporting) has not produced anything I found relevant.


Edit

As per the suggestions in the post linked by @ErikF, I had a look at the output of

journalctl --boot=#

where # stands for 0 or -1, for the current and previous boots respectively (corresponding to the two kernels). These lines do seem to reflect the difference, so it is now a little clearer to me where it stems from:

arch (the one reporting larger MemTotal):

Memory: 32587752K/33472072K available (10252K kernel code, 1157K rwdata, 2760K rodata, 1364K init, 988K bss, 884320K reserved, 0K cma-reserved)

slk (the one reporting smaller MemTotal):

Memory: 32533996K/33472072K available (14348K kernel code, 1674K rwdata, 3976K rodata, 1616K init, 784K bss, 938076K reserved, 0K cma-reserved)

That's a difference of ~55 MB, as expected!

I know the slk kernel is larger, as verified by comparing the sizes of the two vmlinuz files in my /boot/ folder, but the brunt of the difference seems to come from how much memory the two respective kernels reserve.

I'd like to better understand what in the config files affects that to the extent that it does, but this certainly sheds some light.


Second edit

Answering the questions in the comment by @Tim Kennedy.

Do you have a dedicated GPU, or use shared video memory

No dedicated GPU; it's a laptop with on-board Intel graphics.

and do both kernels load the same graphics driver?

Yes, i915.

Also, compare the output of dmesg | grep BIOS-e820 | grep reserved

As you expected, does not change. In all cases it's 12 lines, identical in every respect (memory addresses and all).


(Final?) edit

I believe it may just be as simple as this: the kernel reporting less MemTotal has much more of the driver suite built in; I just hadn't realized it would make such a noticeable difference..

I compared:

du -sh /lib/modules/<smaller-kernel>/modules.builtin

returns 4K, while

du -sh /lib/modules/<larger-kernel>/modules.builtin

returns 16K.

So in the end, I believe I was barking up the wrong tree: it won't be a single config option (or a handful), but rather the accumulated effect of many more built-in drivers.

  • Possibly related answer: unix.stackexchange.com/questions/84695/proc-meminfo-memtotal – ErikF Feb 3 '18 at 23:00
  • That helps, thanks! Edited the post to relay new information. – grobber Feb 3 '18 at 23:59
  • A couple of things to validate would be which kernel modules are built in, and which modules are loaded dynamically, and which are actually loaded. Do you have a dedicated GPU, or use shared video memory, and do both kernels load the same graphics driver? Also, compare the output of dmesg | grep BIOS-e820 | grep reserved (ought not to change on the same maching, but you never know). – Tim Kennedy Feb 4 '18 at 0:04
  • Thank you! Added that as Second edit above. Incidentally, the post might grow unwieldy, what with all of the edits; is it common practice to always just edit the original when new info becomes available? – grobber Feb 4 '18 at 0:46
0

I believe the picture is as in my last edit above; I just did not know enough about the process of reserving memory to realize when I asked the question: it all boiled down to the larger kernel having more of the drivers built in rather than modularized.

Incidentally, I did want to put this to some (remotely) practical use: I wanted to have as slim a kernel as I could, but still boot it without an initrd. I knew that

  • the smaller kernel (moniker arch above) is very lightweight and fast to boot, but not without an initramfs;

  • the larger kernel (slk) will boot without an initramfs, but takes longer.

So I made a compromise that I think I'll stick with for now. I took the two config files, call them large and small, and made sure that every unset config option in small is reflected in large.

First,

awk '/^# CO/ {print} small > staging'

grabs all lines in the small config file that are of the form

# CONFIG_BLAH is not set

and dumps them in a staging text file. Then,

for i in $(awk '{print $2}' staging) ; do sed -i "s/^$i=./# $i is not set/" large ; done

takes all lines from the config file large that set (via =y or =m) an option contained in staging and unsets them.

I then ran make oldconfig on the resulting file large in the kernel source directory and compiled. It all went through all right:

  • the new kernel boots some 3 seconds faster

  • without an initramfs

  • is much smaller in the sense that its /lib/modules/<kernel>/modules.builtin got cut down in half, from 16K to 8K.


Those are not much to write home about, but as I said this was puzzling me.. I think I'm all set on the issue now.

Presumably the more straightforward thing to do would have been to figure out once and for all precisely which drivers I need on this machine in order to boot without an initramfs, but I'll leave that for some other day. Besides, playing off one config against the other was a fun exercise in its own right.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.