Partitionless Btrfs suddenly booting to GRUB rescue

Question

The Problem

I have a machine which boots into Arch Linux from an eight-disk Btrfs RAID10 volume. Recently, I rebooted and landed at this GRUB rescue prompt:

I'm not sure what exactly triggered this. Since the last boot, I installed updates (pacman -Syu) at least twice and may have made other changes to the system including reinstalling GRUB. I am able to start the system if I add a dedicated boot disk.

The theory

While troubleshooting with a helpful member of the #archlinux IRC channel, I ran into this note on the Arch wiki:

Partition offset

The offset problem may happen when you try to embed core.img into a partitioned disk. It means that it is OK to embed grub's core.img into a Btrfs pool on a partitionless disk (e.g. /dev/sdX) directly. GRUB can boot Btrfs partitions, however the module may be larger than other file systems. And the core.img file made by grub-install may not fit in the first 63 sectors (31.5KiB) of the drive between the MBR and the first partition. Up-to-date partitioning tools such as fdisk and gdisk avoid this issue by offsetting the first partition by roughly 1MiB or 2MiB.

This seems to say that there can be problems with installing GRUB onto partitioned and partition-less disks which newer partitioning tools mitigate (for now) by leaving extra room at the beginning of the disk.

The GRUB wiki page is more direct:

Install to partition or partitionless disk

Warning: GRUB strongly discourages installation to a partition boot sector or a partitionless disk as GRUB Legacy or Syslinux does. This setup is prone to breakage, especially during updates, and is not supported by the Arch developers.

Well, shucks.

For what it's worth, grub-install -v includes this line in its output:

grub-install: info: the total module size is 0xa07c.

That's ~41K, solidly over the 31.5KiB limit mentioned above, but I don't know GRUB well enough to be sure that this is the cause of my woes.

The Questions

1. If this is the problem, how can I prove it — and why doesn't grub-install fail loudly if so?

2. What's the right way to format bootable Btrfs disks going forward? MBR? GPT? — a dedicated boot disk is tempting, but I like having a redundant bootloader on every device in the volume.

3. Is there a nicer way to migrate each disk to a partition table than erasing and running btrfs replace for each disk?

Answer 1

Back in the day creating an MBR partition table with say, fdisk, would by default leave the first 63 sectors untouched. That's where GRUB and other bootloaders could be installed.

Fast forward to modern days and the same tool (fdisk) leaves more unused space; enough for GRUB2 to install it's stage1 with BTRFS support. I believe the offset is about 1MB, by default; whatever that amounts to in sectors.

I don't know why grub-install doesn't fail, but I suppose it doesn't check the bootsector size; and without partitions how could it.

I don't see a problem with having redundant bootloaders. You've have to manage that manually, but GRUB2's stage1 doesn't change often. But it means you'd need partitions.

I don't know of a tool which can migrate the disk to add a partition table. The problem is the filesystem is tied to sectors on the disk, and adding a partition table changes those sectors. If your BTRFS filesystems were on LVM, then yes you'd be able to move things around because the filesystem would be tied to "virtual sectors". Not saying you should do that, just illustrating what the issue is.