1

I was using GRUB for UEFI PXE boot or UEFI HTTP boot of a Linux Live OS (made by Ubuntu Focal or Bionic or Jammy). I used the same grub.cfg for both methods. It worked well under UEFI PXE boot, but failed during UEFI HTTP boot.

Then I found something interesting about in which phase who did http access.

  • in UEFI PXE boot, the HTTP access log is like this:
"GET /liveos/focal/kernel HTTP/1.1" 200 11780639 "-" "GRUB 2.06-2ubuntu14.1"
"GET /liveos/focal/initrd HTTP/1.1" 200 89168508 "-" "GRUB 2.06-2ubuntu14.1"
"GET /liveos/focal/squashfs HTTP/1.1" 200 388669747 "-" "Wget"

So GRUB loaded kernel and initrd, then initrd used wget to load squashfs. OK, it makes sense.

  • however, when in UEFI HTTP boot, the HTTP access log is like this:
"GET /bootx64.efi HTTP/1.1" 200 955941 "-" "UefiHttpBoot/1.0"
"GET /grubx64.efi HTTP/1.1" 200 1493150 "-" "UefiHttpBoot/1.0"

"GET /grub/grub.cfg HTTP/1.1" 200 1823 "-" "UefiHttpBoot/1.0"

"GET /liveos/focal/kernel HTTP/1.1" 200 11780639 "-" "UefiHttpBoot/1.0"
"GET /liveos/focal/initrd HTTP/1.1" 200 449680 "-" "UefiHttpBoot/1.0"

So GRUB use the http agent "UefiHttpBoot/1.0" to load kernel and initrd. Unfortunately the access log shows that server did not respond full size of data to the UEFI HTTP boot client. Full size response should be like 89168508, but now are just 449680.

Here comes the questions:

  • Where is the http agent "UefiHttpBoot/1.0" implemented? in the firmware?
  • Why does GRUB2 not use same logic as in UEFI PXE boot, i.e., using its own http agent to get initrd?

EDIT: it looks like that, in UEFI HTTP boot, from the beginning, it is the http agent "UefiHttpBoot/1.0" that loading bootx64.efi, so "UefiHttpBoot/1.0" should comes from the firmware, however, it should then pass execution control to the bootx64.efi(GRUB2), then GRUB2 should use its own http tool to read initrd, but actually it is not, why? Seems like the UEFI HTTP boot cooperates with Linux boot protocol in some degree.

EDIT: more info:

  [    1.947449] Unpacking initramfs...
  [    1.950954] Initramfs unpacking failed: junk in compressed archive

  or 
  [    1.943484] Unpacking initramfs...
  [    1.946991] Initramfs unpacking failed: broken padding

The grub.cfg is like this: (please remember that it works well in UEFI PXE boot):

  set default=0
  set timeout=1
  menuentry boot_liveos {
    linux (http,100.0.0.101)/liveos/focal/kernel nomodeset ro root=squash:http://100.0.0.101/liveos/focal/squashfs ip=::::hostname:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 ds=nocloud console=tty0 console=ttyS1,115200 BOOTIF=01-${net_default_mac}
    initrd (http,100.0.0.101)/liveos/focal/initrd
  }

PS, not only focal, but also bionic, jammy, all same issue. I have also tried different GRUB2 versions, from 20200320.0 to 20230222.0, all same issue.

The problem becomes that something wrong in the "UefiHttpBoot/1.0", it can not accept large data.

EDIT: if this is an firmware issue, then is there anyway to tell grub to load http files by itself?

EDIT: continue to investigate the issue.

$ strings bootx64.efi |grep UefiHttpBoot
UefiHttpBoot/1.0
$ strings grubx64.efi |grep UefiHttpBoot
UefiHttpBoot/1.0

The GRUB2 binaries embeds the http agent string "UefiHttpBoot/1.0", so the execution control should have been passed to GRUB2, it is just that GRUB2 continues to use same agent string to do http access. I have checked source code at http://archive.ubuntu.com/ubuntu/pool/main/g/grub2-unsigned/grub2-unsigned_2.06.orig.tar.xz (parent page is https://packages.ubuntu.com/focal-updates/grub-efi-amd64) but have not found the agent string, but found a constant var PACKAGE_STRING, probably which is defined when building the binaries.

http_establish (struct grub_file *file, grub_off_t offset, int initial)
{
  http_data_t data = file->data;
  grub_uint8_t *ptr;
  int i;
  struct grub_net_buff *nb;
  grub_err_t err;

  nb = grub_netbuff_alloc (GRUB_NET_TCP_RESERVE_SIZE
               + sizeof ("GET ") - 1
               + grub_strlen (data->filename)
               + sizeof (" HTTP/1.1\r\nHost: ") - 1
               + grub_strlen (file->device->net->server)
               + sizeof ("\r\nUser-Agent: " PACKAGE_STRING
                     "\r\n") - 1
               + sizeof ("Range: bytes=XXXXXXXXXXXXXXXXXXXX"
                     "-\r\n\r\n"));
  if (!nb)
    return grub_errno;

Found the http agent string in GRUB2's debian patched source: http://archive.ubuntu.com/ubuntu/pool/main/g/grub2-unsigned/grub2-unsigned_2.06-2ubuntu14.1.debian.tar.xz, it was a patch file suse-add-support-for-UEFI-network-protocols.patch, which replaced some http related functions to use efi firmware functions.

+static grub_err_t
+efihttp_request (grub_efi_http_t *http, char *server, char *name, int use_https, int headeronly, grub_off_t *file_size)
+{
+  grub_efi_http_request_data_t request_data;
+  grub_efi_http_message_t request_message;
+  grub_efi_http_token_t request_token;
+  grub_efi_http_response_data_t response_data;
+  grub_efi_http_message_t response_message;
+  grub_efi_http_token_t response_token;
+  grub_efi_http_header_t request_headers[3];
+
+  grub_efi_status_t status;
+  grub_efi_boot_services_t *b = grub_efi_system_table->boot_services;
+  char *url = NULL;
+
+  request_headers[0].field_name = (grub_efi_char8_t *)"Host";
+  request_headers[0].field_value = (grub_efi_char8_t *)server;
+  request_headers[1].field_name = (grub_efi_char8_t *)"Accept";
+  request_headers[1].field_value = (grub_efi_char8_t *)"*/*";
+  request_headers[2].field_name = (grub_efi_char8_t *)"User-Agent";
+  request_headers[2].field_value = (grub_efi_char8_t *)"UefiHttpBoot/1.0";
+
...

+  /* request token */
+  request_token.event = NULL;
+  request_token.status = GRUB_EFI_NOT_READY;
+  request_token.message = &request_message;
+
+  request_callback_done = 0;
+  status = efi_call_5 (b->create_event,
+                       GRUB_EFI_EVT_NOTIFY_SIGNAL,
+                       GRUB_EFI_TPL_CALLBACK,
+                       grub_efi_http_request_callback,
+                       NULL,
+                       &request_token.event);
+
+ 
...

EDIT: almost got there. The GRUB2 I used is patched with an optimization that use underlying efi firmware functionality to access http file. Sounds great, however it has several drawbacks:

  • it seems does support url querystring well.
  • it does not handle large data well. I will report this to GRUB2 developers. Turns out that the there are a lot of TCP retransmission and windows size adjustment.
4
  • sorry, that is just a typo, all are consistent. I have tried focal, bionic, jammy, neither works. And tried different GRUB2 versions, from 20200320.0 to 20230222.0, all same issue.
    – osexp2000
    Jun 9, 2023 at 6:31
  • The most important thing is that, without change, all works in UEFI PXE boot. Just once switch to UEFI HTTP boot, it fails due to the initrd loading issue.
    – osexp2000
    Jun 9, 2023 at 6:31
  • See my edit, the kernel clearly reported the initramfs content is wrong. The size is wrong actually, it should be 58M+.
    – osexp2000
    Jun 9, 2023 at 6:33
  • Thanks for the clarification; since that was just a typo, I deleted my earlier comment.
    – telcoM
    Jun 9, 2023 at 7:14

2 Answers 2

1

The "UefiHttpBoot/1.0" HTTP client is indeed implemented in the UEFI firmware (UEFI specification version 2.5 or newer).

When UEFI PXE boot is used, the firmware will load the initial bootx64.efi using TFTP, because that might be the only network file transfer protocol the firmware can do on its own. So GRUB must carry its own HTTP client implementation if you want to use HTTP in this case.

But with a quick grep through the GRUB source tree, I could not find any reference to UEFI HTTP protocol. The UEFI network driver for GRUB, grub-core/net/drivers/efi/efinet.c only relies on the firmware-provided Simple Network Protocol of the earlier UEFI spec versions.

So it seems to me that when UEFI HTTP boot is used, GRUB apparently is not even aware it's using network at all. The UEFI spec says that with HTTP boot, the firmware provides a UEFI device handle that supports the UEFI LoadFile protocol, so GRUB can use it to load files, just like loading files from a disk. Of course GRUB could investigate the device handle further to see that it's actually a network device - but it doesn't have to do that in order to do its main job.

That also makes it pretty clear that the UEFI boot failure to load the initrd is definitely a firmware issue.

9
  • thank you so much! it makes sense: GRUB apparently is not even aware it's using network at all. The UEFI spec says that with HTTP boot, the firmware provides a UEFI device handle that supports the UEFI LoadFile protocol, so GRUB can use it to load files, just like loading files from a disk
    – osexp2000
    Jun 9, 2023 at 7:11
  • I saw discussion in groups.google.com/g/linux.debian.bugs.dist/c/CqfwbhAd-Xg shows that grub borrows efi firmware functionalities to do file access.
    – osexp2000
    Jun 9, 2023 at 7:13
  • Is there anyway to force grub to use its own http agent to access remote files?
    – osexp2000
    Jun 9, 2023 at 7:13
  • I don't know of one; you might ask from the GRUB developers, and perhaps present your issue as a case where such a feature would be helpful.
    – telcoM
    Jun 9, 2023 at 7:17
  • 1
    Okay, that might be a distribution-specific patch then, as it does not exist on the upstream Git repository code.
    – telcoM
    Jun 9, 2023 at 8:34
0

Let me answer it by myself.

The solution is finding an older grubx64.efi which does not contain the word UefiHttpBoot, I found a 20200320.0 version in local history fortunately, it works well. (It seems no longer exist in Ubuntu's repo or achieves on internet.) The corresponding bootx64.efi still contains word UefiHttpBoot, but it does not matter because that is just a shim and it just load grubx64.efi then pass control to that.

The http access is like this:

"GET /bootx64.efi HTTP/1.1" 200 1335102 "-" "UefiHttpBoot/1.0"
"GET /grubx64.efi HTTP/1.1" 200 1164950 "-" "UefiHttpBoot/1.0"
...
"GET /grub/grub.cfg HTTP/1.1" 200 737 "-" "GRUB 2.02-2ubuntu8.15"
...
"GET /liveos/focal/kernel HTTP/1.1" 200 11780639 "-" "GRUB 2.02-2ubuntu8.15"
"GET /liveos/focal/initrd HTTP/1.1" 200 89168508 "-" "GRUB 2.02-2ubuntu8.15"
"GET /liveos/focal/squashfs HTTP/1.1" 200 388669747 "-" "Wget"
...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .