While browsing through the Kernel Makefiles, I found these terms. So I would like to know what is the difference between
This is the Linux kernel in an statically linked executable file format. Generally, you don't have to worry about this file, it's just a intermediate step in the boot procedure.
The raw vmlinux file may be useful for debugging purposes.
The same as vmlinux, but in a bootable raw binary file format. All symbols and relocation information is discarded. Generated from
objcopy -O binary vmlinux vmlinux.bin.
The vmlinux file usually gets compressed with
zlib. Since 2.6.30
bzip2 are also available. By adding further boot and decompression capabilities to vmlinuz, the image can be used to boot a system with the vmlinux kernel. The compression of vmlinux can occur with zImage or bzImage.
decompress_kernel() handles the decompression of vmlinuz at bootup, a message indicates this:
Decompressing Linux... done Booting the kernel.
This is the old format for small kernels (compressed, below 512KB). At boot, this image gets loaded low in memory (the first 640KB of the RAM).
The big zImage (this has nothing to do with
bzip2), was created while the kernel grew and handles bigger images (compressed, over 512KB). The image gets loaded high in memory (above 1MB RAM). As today's kernels are way over 512KB, this is usually the preferred way.
An inspection on Ubuntu 10.10 shows:
ls -lh /boot/vmlinuz-$(uname -r) -rw-r--r-- 1 root root 4.1M 2010-11-24 12:21 /boot/vmlinuz-2.6.35-23-generic file /boot/vmlinuz-$(uname -r) /boot/vmlinuz-2.6.35-23-generic: Linux kernel x86 boot executable bzImage, version 2.6.35-23-generic (buildd@rosea, RO-rootFS, root_dev 0x6801, swap_dev 0x4, Normal VGA
Do a verbose kernel build and search for the files
This approach can give some insight, will never get out of date, and will help you to easily find which part of the build system is doing what.
Once you have a build configuration that generates one of the files, build with:
make V=1 |& tee f.log
Modify a comment on some C file to force a re-link (e.g.
init/main.c is a good one) if you have already built previously.
f.log and search for the images of interest.
For example, on v4.19 we will conclude that:
init/main.c | | gcc -c | v init/.tmp_main.o | | CONFIG_MODVERSIONS stuff | v init/main.o | | ar T (thin archive) | v init/built-in.a | | ar T (thin archive) | v built-in.a | | ld | v vmlinux (regular ELF file) | | objcopy | v arch/x86/boot/compressed/vmlinux.bin | | GZIP | v
arch/x86/boot/compressed/vmlinux.bin.gz | | .incbin | v arch/x86/boot/compressed/piggy.S | | gcc -c | v arch/x86/boot/compressed/piggy.o | | ld | v arch/x86/boot/compressed/vmlinux (regular ELF file with gzipped code) | | objcopy | v arch/x86/boot/vmlinux.bin | | arch/x86/boot/tools/build.c | v arch/x86/boot/bzImage
Thin archives are mentioned at: https://stackoverflow.com/questions/2157629/linking-static-libraries-to-other-static-libraries/27676016#27676016 They are archives that just point other archives / objects instead of copying them.
The kernel moved from incremental linking to thin archives in v4.9 as described at: https://stackoverflow.com/questions/29391965/what-is-partial-linking-in-gnu-linker/53959624#53959624
Full log interpretation
When we start reading the verbose build logs from the back up, first we see:
ln -fsn ../../x86/boot/bzImage ./arch/x86_64/boot/bzImage
so those two are just symlinked.
Then we search a bit further for
x86/boot/bzImage and find:
arch/x86/boot/tools/build \ arch/x86/boot/setup.bin \ arch/x86/boot/vmlinux.bin \ arch/x86/boot/zoffset.h \ arch/x86/boot/bzImage
arch/x86/boot/tools/build is an executable, so we run it, see the help message:
Usage: build setup system zoffset.h image
and grep to find the source:
So this tool must be generating
arch/x86/boot/vmlinux.bin and other files TODO what is the point of
If we follow
arch/x86/boot/vmlinux.bin we see that it is just an
objcopy \ -O binary \ -R .note \ -R .comment \ -S arch/x86/boot/compressed/vmlinux \ arch/x86/boot/vmlinux.bin
arch/x86/boot/compressed/vmlinux is just a regular ELF file:
ld \ -m elf_x86_64 \ -z noreloc-overflow \ -pie \ --no-dynamic-linker \ -T arch/x86/boot/compressed/vmlinux.lds \ arch/x86/boot/compressed/head_64.o \ arch/x86/boot/compressed/misc.o \ arch/x86/boot/compressed/string.o \ arch/x86/boot/compressed/cmdline.o \ arch/x86/boot/compressed/error.o \ arch/x86/boot/compressed/piggy.o \ arch/x86/boot/compressed/cpuflags.o \ arch/x86/boot/compressed/early_serial_console.o \ arch/x86/boot/compressed/kaslr.o \ arch/x86/boot/compressed/kaslr_64.o \ arch/x86/boot/compressed/mem_encrypt.o \ arch/x86/boot/compressed/pgtable_64.o \ -o arch/x86/boot/compressed/vmlinux
ls -hlSr says that
piggy.o is by far the largest file, so we search for it, and it must come from:
gcc \ -Wp,-MD,arch/x86/boot/compressed/.piggy.o.d \ -nostdinc \ -Ilinux/arch/x86/include \ -I./arch/x86/include/generated \ -Ilinux/include \ -I./include \ -Ilinux/arch/x86/include/uapi \ -I./arch/x86/include/generated/uapi \ -Ilinux/include/uapi \ -I./include/generated/uapi \ -include linux/include/linux/kconfig.h \ -D__KERNEL__ \ -m64 \ -O2 \ -fno-strict-aliasing \ -fPIE \ -DDISABLE_BRANCH_PROFILING \ -mcmodel=small \ -mno-mmx \ -mno-sse \ -ffreestanding \ -fno-stack-protector \ -Wno-pointer-sign \ -D__ASSEMBLY__ \ -c \ -o arch/x86/boot/compressed/.tmp_piggy.o \ arch/x86/boot/compressed/piggy.S
.tmp_ prefix explained below.
arch/x86/boot/compressed/vmlinux.bin.gz comes from:
cat arch/x86/boot/compressed/vmlinux.bin arch/x86/boot/compressed/vmlinux.relocs | \ gzip -n -f -9 > arch/x86/boot/compressed/vmlinux.bin.gz
which comes from:
objcopy -R .comment -S vmlinux arch/x86/boot/compressed/vmlinux.bin
which comes from:
ld \ -m elf_x86_64 \ -z max-page-size=0x200000 \ --emit-relocs \ --build-id \ -o vmlinux \ -T ./arch/x86/kernel/vmlinux.lds \ --whole-archive \ built-in.a \ --no-whole-archive \ --start-group \ lib/lib.a \ arch/x86/lib/lib.a \ --end-group \ .tmp_kallsyms2.o
vmlinux is huge, but all shown objects are tiny according to
ls -l, so I researched and learned about a new
ar feature I didn't know about: thin archives.
the build does:
ar \ rcsTPD \ built-in.a \ arch/x86/kernel/head_64.o \ arch/x86/kernel/head64.o \ arch/x86/kernel/ebda.o \ arch/x86/kernel/platform-quirks.o \ init/built-in.a \ usr/built-in.a \ arch/x86/built-in.a \ kernel/built-in.a \ certs/built-in.a \ mm/built-in.a \ fs/built-in.a \ ipc/built-in.a \ security/built-in.a \ crypto/built-in.a \ block/built-in.a \ lib/built-in.a \ arch/x86/lib/built-in.a \ drivers/built-in.a \ sound/built-in.a \ firmware/built-in.a \ arch/x86/pci/built-in.a \ arch/x86/power/built-in.a \ arch/x86/video/built-in.a \ net/built-in.a \ virt/built-in.a
T specifies the thin archive.
We can then see that all sub archives are also thin, e.g., since I modified
init/main.c, we have:
ar \ rcSTPD \ init/built-in.a \ init/main.o \ init/version.o \ init/do_mounts.o \ init/do_mounts_initrd.o \ init/initramfs.o \ init/calibrate.o \ init/init_task.o
which finally comes from the C file through a command like:
gcc \ -Wp,-MD,init/.main.o.d \ -c \ -o \ init/.tmp_main.o \ /work/linux-kernel-module-cheat/submodules/linux/init/main.c
I can't find the
init/main.o step on the logs which is a shame... with:
git grep '\.tmp_'
we see that likely comes from
scripts Makefile.build and is linked to
CONFIG_MODVERSIONS which I had enabled:
ifndef CONFIG_MODVERSIONS cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< else # When module versioning is enabled the following steps are executed: # o compile a .tmp_<file>.o from <file>.c # o if .tmp_<file>.o doesn't contain a __ksymtab version, i.e. does # not export symbols, we just rename .tmp_<file>.o to <file>.o and # are done. # o otherwise, we calculate symbol versions using the good old # genksyms on the preprocessed source and postprocess them in a way # that they are usable as a linker script # o generate <file>.o from .tmp_<file>.o using the linker to # replace the unresolved symbols __crc_exported_symbol with # the actual value of the checksum generated by genksyms cmd_cc_o_c = $(CC) $(c_flags) -c -o $(@D)/.tmp_$(@F) $< cmd_modversions_c = \ if $(OBJDUMP) -h $(@D)/.tmp_$(@F) | grep -q __ksymtab; then \ $(call cmd_gensymtypes_c,$(KBUILD_SYMTYPES),$(@:.o=.symtypes)) \ > $(@D)/.tmp_$(@F:.o=.ver); \ \ $(LD) $(KBUILD_LDFLAGS) -r -o $@ $(@D)/.tmp_$(@F) \ -T $(@D)/.tmp_$(@F:.o=.ver); \ rm -f $(@D)/.tmp_$(@F) $(@D)/.tmp_$(@F:.o=.ver); \ else \ mv -f $(@D)/.tmp_$(@F) $@; \ fi; endif
Analysis done with this config which contains
Just an uncompressed
objcopy -O binary -R .note -R .note.gnu.build-id -R .comment -S vmlinux arch/arm64/boot/Image
vmlinux is obtained in basically the exact same way as for x86 though the thin archives.
Very similar to X86 with a zipped
vmlinux, but no magic
build.c step. Call chain summary:
objcopy -O binary -R .comment -S arch/arm/boot/compressed/vmlinux arch/arm/boot/zImage ld \ -EL \ --defsym _kernel_bss_size=469592 \ -p \ --no-undefined \ -X \ -T arch/arm/boot/compressed/vmlinux.lds \ arch/arm/boot/compressed/head.o \ arch/arm/boot/compressed/piggy.o \ arch/arm/boot/compressed/misc.o \ arch/arm/boot/compressed/decompress.o \ arch/arm/boot/compressed/string.o \ arch/arm/boot/compressed/hyp-stub.o \ arch/arm/boot/compressed/lib1funcs.o \ arch/arm/boot/compressed/ashldi3.o \ arch/arm/boot/compressed/bswapsdi2.o \ -o arch/arm/boot/compressed/vmlinux gcc \ -c \ -o arch/arm/boot/compressed/piggy.o \ linux/arch/arm/boot/compressed/piggy.S .incbin "arch/arm/boot/compressed/piggy_data" cat arch/arm/boot/compressed/../Image | gzip -n -f -9 > arch/arm/boot/compressed/piggy_data objcopy -O binary -R .comment -S vmlinux arch/arm/boot/Image
QEMU v4.0.0 can boot from bzImage but not vmlinux
This is another important practical difference: https://superuser.com/questions/1451568/booting-an-uncompressed-kernel-in-qemu
A non-compressed and non-bootable Linux kernel file format, just an intermediate step to producing
A compressed and bootable Linux kernel file. It is actually
For old kernels, just fit
640k ram size.
Big zImage, no
640k ram size limit, can much larger.
Please refer this document: vmlinuz Definition.
I don't usually use Wikipedia to understand things like this but it's surprisingly useful for vmlinux.
As the Linux kernel matured, the size of the kernels generated by users grew beyond the limits imposed by some architectures, where the space available to store the compressed kernel code is limited. The bzImage (big zImage) format was developed to overcome this limitation by splitting the kernel over non-contiguous memory regions.
The bzImage was compressed using gzip until Linux 2.6.30
The bzImage file is in a specific format. It contains concatenated bootsect.o + setup.o + misc.o + piggy.o. piggy.o contains the gzipped vmlinux file in its data section.
I know I've only partially answered the question - I'm still trying to understand it myself.