*tldr; I would like to generally understand how the world of linux / embedded linux works. What do I need to do to take the Linux mainline and compile/deploy it on a board with different processors and peripherals, from scratch.

How I currently See It Working:

Steps to get Linux running on arbitrary board:

  • Get sources for uBoot (for embedded) or GRUB (desktop/x86 SOM)
  • Modify uBoot or GRUB for specific system, write code to init specific chip and get required interfaces for memory and console up and running
  • Modify uBoot/GRUB config.txt to configure code written above
  • compile these and deploy to board, verify that bootloader console comes up and can interact with it
  • Get kernel mainline sources
  • "make config" to select drivers and modules that will be available (At this point these selections will change the source - wherever these settings are stored will no longer match a git clone from the mainline) (Track this .config file in source control for future reference)
  • Get tools such as Busybox or desktop alternative? Install in source directories
  • Get ucLibc or other libraries and install in source directories
  • Compile kernel source using cross compiler toolchain for specific chip
  • Create device tree files .dtb for board (both embedded/desktop? or desktop does not use?) This connects drivers to physical pins
  • Use Uboot/GRUB and TFTP/serial console or memory card etc to load compiled kernel image.
  • Boot up and verify shell access through serial/SSH etc depending on drivers and device tree config
  • Modify uEnv.txt (embedded) or mysteryfile.txt (desktop) for board specific configurations? This is essentially a script that blocks or adds kernel startup steps? What is the desktop equivalent?
  • apt-get desired packages and drivers
  • write drivers and application code and test (manually loading drivers)
  • Add device tree files to account for hardware and drivers implemented above (these are separate from the intial BSP one created)
  • To include these in kernel image build the kernel and create the file structure with all of these sources and config file mods in the folder strcuture (additions/mods to Linux mainline)
  • Could have a separate folder for Linux mainline and mods, copy mods directly overwritting/adding files to mainline in a third staging folder. This will allow all additions and non-mainline mods to be source controlled separate.

If you can get a base system that you can SSH into, and at this point you have drivers for all the common components (Video, USB, mouse etc) then you can pretty much do anything at this point (install X11 server, LXDE, networking etc)? Which drivers need to be handled by the bootloader/bios and which ones are purely in the kernel domain?

There are Kconfig files for configuring the kernel build. This makes sense and the kernel module development docs that I have seen seem to describe this well.

There are also files like uEnv.txt and config.txt that handle the run time configuration and which devices should be loaded. There are also device tree blobs which also determine which devices should be loaded?

How do the magic strings in these files tie into the kernel, are these modifications done to the mainline for a specific board? Something has to read these to determine if HDMI should be enabled or not, and this can't be the exact same code as what is on the desktop version of Linux.

Once drivers are in the mainline are they still developed independently from the mainline? For example I have been using a couple of drivers but there are notes they are now included in the mainline, does this mean that it is no longer possible to download directly on its own? The steps I have followed downloaded the headers for my board, the source and then compiled it and installed it. If it is in the mainline do I need to pull it from there now instead?

Background and Specific Thoughts

I am an EE and have experience with Microcontrollers and Windows development, but do not have much Linux experience. The framing of my question is "If I started off with this arbitrary (with linux compiler available) processor, and these peripherals how do I (and what are my options) for building a linux release"


I have been able to find RPI2 and BBB (Beaglebone Black) specific documentation and how-to's but when you get into more advanced topics like the bootloader there are only a few crumbs to vaguely describe what is going on. For example the RPI2 has a 3 stage bootloader (of which from reading it does not sound like it is totally uBoot based) and the BBB has a more "traditional" uBoot based bootloader. Now the new BBx15 has jumpers where you can select where you want to boot from.

The desktop systems use GRUB (IIRC) and embedded systems typically use uBoot. I have read that the RPI uses the GPU during boot and reads the first stage bootloader off of a separate ROM. And that is all the information available. If you wanted to spin your own version of the board (for discussions sake, this is not really practical) then in addition to uBoot what is going on? Doesn't uBoot for the BBx15 have extra modifications to allow for the jumper boot selection?

Does Linux know anything about the staging of booting or is it oblivious to this once it is running? The BBB uses uBoot to load the image off the eMMC into RAM, the RPI2 uses the 3 stage bootloader. I am guessing that the BBB uses the ARM processor to do this but the RPI2 uses the GPU. I thought on power up that the ARM processor starts executing, what would they have to modify to stage these load procedure? Does the GPU hold the ARM in reset until it has completed its ROM code? Since the GPU is part of the boot procedure does that mean the code it executes is taken out of the uBoot code, that other systems without this GPU would have to then run in the uBoot code? This whole procedure implies to me that if you modify the second or third stage bootloader that you could run Linux entirely off the GPU alone (if the kernel was compiled with the GPU toolchain)?

Is the third stage bootloader and config.txt actually just uBoot?

Regarding the headers for the board in use. Are these just the headers from the mainline with the drivers that have been overlayed included or is there something more to this. The "headers" are just the mainline headers if that is what you have started running with?

For embedded microcontroller development I am used to having a HAL layer. The HAL has function stubs where you setup the peripherals and then point the drivers to those resources. The board support package typically have these HAL stubs already coded for the board in question. I am sure there has to be some parallels here to Linux development but I can't quite see where these divisions are.

There are packages such as Buildroot and Yocto. Are these just the Linux mainline with an interface to automate selecting the ARM processor and drivers to include?

  • A lot of questions. Haven't read it completely but I think that you should take a look at linuxfromscratch.org. There you find information on how to install Linux by compiling all the software. – Marco Nov 11 '15 at 15:32
  • @GPIB BBB has also 3-Stage Bootloader (ROM-Load (CPU), MLO (SPL), u-boot). i would recommand you to use elbe to build your own distro based on debian. – arash javan Nov 11 '15 at 15:42
  • @Marco checked this out and it looks to provide quite a bit of information on this, thanks – GPIB Nov 12 '15 at 0:54
  • @arash I did not realize this, and now reading into it more it looks like quite a few of these ARM applications processors do this (first stage) then from what I have read the MLO and u-boot come in the same package. Anyways this was a good lead, thanks – GPIB Nov 12 '15 at 0:55

From my small experience with router hardware I played with, I can say that this is a dumb small hardware picked up just to do one thing.

At hardware level, it's simple:

U-Boot there is not only bootloader, but a BIOS in PC terms. So it's not only a bootloader, but also it initializes all hardware. At start, CPU executes it directly (from FLASH for example), and it decides what to do next, but usually it relocates itself into memory. Then it does what it needs to: reads configuration from flash piece then loads image at specified address and transfers control there. Nothing special, but it's important to know.

U-Boot (embedded on router hardware) does not access root filesystem at all. Instead, there is a dedicated space for whole kernel image (usually compressed). So, at least on routers - there is no /boot/vmlinuz file.

RPI indeed uses it's own, proprietary boot sequence. They have closed source binaries which user puts on SD flash. The first stage init code is hardcoded into CPU or somewhere there on board. And they start ARM core after GPU, and whole code is done for GPU. More about that maybe you're already found, but if not: https://raspberrypi.stackexchange.com/questions/10489/how-does-raspberry-pi-boot

So, because I did some fun with routers and had rebuilt them into my small servers completely from source code, I can list my own building sequence:

  • Obtain and build u-boot for platform
  • Build Linux kernel
  • Build userspace (kernel and userspace usually divided, even on flash)
  • Flash that u-boot into flash on programmer
  • Solder flash on to board
  • Connect to board via UART
  • Boot it, verify u-boot inits all hardware well
  • tftp kernel, write to flash inside board
  • tftp rootfs, write to flash inside board
  • reset, verify all works ok
  • fine-tune rootfs: set permissions, preload default config via tftp
  • dump whole image, flash it on many devices

Linux kernel then can, or cannot, support your board. Please verify that. You will not able to just take the latest kernel and build it, for example, for your router. The same with RPi: they have their own kernel tree. That happens often in embedded world, only few (and usually generic) platforms are supported by Linux kernel directly. Be prepared for that.

As for userspace, you can select whatever you need, balance your needs between what you need and how many space is left. Usually in embedded, you either compress anything, or strip unneeded things, or both.

I hope this will shed some light on. If you have further questions - welcome to comments! :-)

| improve this answer | |
  • Thank you this explains it well. "u-boot inits all hardware well" means that both u-boot and the linux kernel needs to know about all the same hardware? Need to find more about how this works. For something like SPI or the display uboot inits these to a known state? Then when the kernel takes over it only needs to know about the connection to the ARM chip (pins or memory location) since everything is already initialized? There is no init code for hardware in linux then? – GPIB Nov 12 '15 at 0:59
  • u-boot definitely needs to know about hardware it bootstraps at start. For display devices and such, take RPi as reference: they start all hardware in their own bootloader, so it is required to initialize all hardware before kernel takes ready environment to work with. At least, CPU and memory must be present to continue booting process. – user140866 Nov 12 '15 at 3:40
  • And you've probably seen the situation when OS cannot boot because it does not know about how to read and write to storage media it was booted from. There is same: u-boot inits flash media well, takes image from it and boots it in a friendly working environment, then image is on it's own, and if it does not know how to control hardware, it cannot work. The result - kernel panic as an example. So both u-boot and kernel mush be aware of hardware they reside on. – user140866 Nov 12 '15 at 3:43
  • Let me see if I have this right. There is an interface here that both u-boot and linux must agree on, uboot does not know anything about what linux is going to do. Linux knows about the hardware somehow, so is this a combination of the make menuconfig, device tree files/overlays, and the old way of including kernel drivers? For example, the display might be a framebuffer driver for a specific TFT, and this is included (using dtb or compiled in driver). The driver specifies the pin connections and how to talk to it, but there is no init code in the linux driver since it is initialized? – GPIB Nov 12 '15 at 12:32
  • So the main part of getting linux running on a new board after you have the toolchain is getting/writing drivers for all the peripherals and hardware you plan on using? Uboot needs to have this and Linux does as well, the init is uboot and the main part of the driver is in Linux. Somehow you need to have something that defines the raw chip pin connections (and common drivers like SPI), then the next level is the connection of the pins to the driver software "ports" (Framebuffer I/O) – GPIB Nov 12 '15 at 12:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.