Getting Started - GSoC '23

June 2, 2023 · 2064 words · 10 min · gsoc

After a week of setting up the iPXE server, my Raspberry Pi 3B+ can now boot Linux kernel from it with TLS mutual authentication enabled. In this post I’ll introduce how I built the SDcard image for my RPi and set up the iPXE server. Both of them are currently only for testing purposes so the process could be kind of messy. Anyway, let’s just get it running and leave the optimization for future. Also, like before I’ll append the TODO list to the end of this blog. Take a look if this post interests you. :D

The SDcard Image

We use iPXE as the netboot firmware. To be able to boot iPXE modernly, a UEFI compatible bootloader is needed. For RPi, there’s two bootloaders as our candidates, EDK2 and U-Boot. As the former seems way too heavy for our use and the latter has better support among ARM SBCs, I chose U-Boot, hoping that could make it easier for us to port the image to SBCs other than RPi.

Here is a demo script I wrote to build the SDcard image. Feel free to have a look if there’s any ambiguity in my description below.

RPi3 Boot Flow

When powered on, the VideoCore (VC) runs the BootROM stored in SoC, which loads bootcode.bin from SDcard to L2 cache and runs it, while the ARM core is resetted.
bootcode.bin initializes SDRAM and then reads fixup.dat and start.elf from SDcard.
- fixup.dat is a linker file that helps spliting the SDRAM between VC and ARM.
- start.elf is the VC firmware, which then reads config.txt from SDcard and carries out rest of the boot process.
start.elf is able to load kernel, device tree blob (DTB) and apply configurations from config.txt. We’ll configure it to load U-Boot.
Finally, the reset signal for ARM core is cleared. U-Boot executes iPXE, which fetches kernel and initrd from remote server. Boom, our RPi boots!

RPi firmwares
fixup.dat use
RPi boot process

Now, let’s build up the SDcard image step by step.

Firmware

bootcode.bin, fixup.dat and start.elf are all firmwares provided by the vendor, which can be downloaded from RPi firmware repository. The repo also contains DTB for the RPi kernel. For RPi3B+, the file name is bcm2710-rpi-3-b-plus.dtb. If you use RPi kernel instead of mainline kernel, it’s convenient to download DTB from here directly.

Now it time to grab a SDcard and copy the firmwares to it. RPi requires the first partition on the SDcard to be FAT16 or FAT32. That’s the where bootcode.bin, fixup.dat, start.elf shoud be placed.

However, FAT32 seems to not work well with my RPi. I still don’t know why but switching to FAT16 did indeed solve my problem. I wrote it down here in case someone could have a similar issue.

Here’s how I created the partition.

# suppose your sdcard is /dev/sda
parted --align optimal /dev/sda mklabel msdos mkpart primary fat16 0M 32M

# now create the filesystem
mkfs.fat -F 16 /dev/sda1

RPi partitioning explained

Then just copy the firmwares to root directory of the partition we’ve just created.

Config.txt

Apart from those blobs, we also need to provide a config.txt. In following example, some of the parameters are necessary for boot process, some are set for the convenience of debug.

# we want to start u-boot in 64-bit mode
arm_64bit=1

# this is the device tree blob start.elf should load
device_tree=bcm2710-rpi-3-b-plus.dtb

# we'll build u-boot and place the binay here later
kernel=u-boot.bin

# miniuart is disabled by default, enable it
enable_uart=1

# set frequency of VC core to a fixed value so that miniuart has a stable baud rate
core_freq=250

# enable logging for bootcode.bin and start.elf
uart_2ndstage=1

Among configurations above, the trickiest part is to get the UART working. RPi3B+ has 2 UART, PL011 and mini UART. By default, the former is used by bluetooth and the latter is mapped to GPIO. However, the clock of mini UART is linked to clock of VC core. If frequency of VC core is not fixed, we’ll never get valid output from mini UART due to the changing baud rate.

documentation of config.txt

Now, simply plug the SDcard into the RPi, connect the serial, power it up and you should see some logs printed out like

Raspberry Pi Bootcode
Read File: config.txt, 209
Read File: start.elf, 2975744 (bytes)
Read File: fixup.dat, 7266 (bytes)
MESS:00:00:01.212454:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.216673:0: brfs: File read: 209 bytes
...

U-Boot

There’s RPi support in mainline U-Boot. So it’s pretty straightforward to boot it on our board.

First, let’s build for RPi3B+ target.

git clone https://github.com/u-boot/u-boot.git

cd u-boot
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_3_b_plus_defconfig

# this should generate u-boot.bin
make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- -j`nproc`

Then, we need to create a boot script (boot.cmd) which executes the iPXE binay (efi/tools/snp.efi).

boot.cmd

load mmc 0:1 $kernel_addr_r efi/tools/snp.efi
bootefi $kernel_addr_r

# this should generate boot.scr
./tools/mkimage -A arm -T script -d boot.cmd boot.scr

Copy u-boot.bin and boot.scr to root directory of the boot partition and power the RPi up. There should be U-Boot logs printed out.

U-Boot 2023.04-rc4-00053-g8be7b4629e (Jun 03 2023 - 22:19:18 +0800)

DRAM:  948 MiB
RPI 3 Model B+ (0xa020d3)
Core:  66 devices, 14 uclasses, devicetree: embed
MMC:   mmc@7e202000: 0, mmc@7e300000: 1
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1...
In:    serial
Out:   vidconsole
Err:   vidconsole
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0

iPXE

Before building iPXE, enable a few features we’ll need later firstly.

git clone https://github.com/ipxe/ipxe.git
cd ipxe/src

config/local/general.h

/* general.h */
#define NSLOOKUP_CMD            /* Name resolution command */
#define PING_CMD                /* Ping command */
#define NTP_CMD                 /* NTP commands */
#define VLAN_CMD                /* VLAN commands */
#define IMAGE_EFI               /* EFI image support */
#define DOWNLOAD_PROTO_HTTPS    /* Secure Hypertext Transfer Protocol */
#define DOWNLOAD_PROTO_FTP      /* File Transfer Protocol */
#define DOWNLOAD_PROTO_NFS      /* Network File System Protocol */
#define DOWNLOAD_PROTO_FILE     /* Local file system access */

Like U-Boot, we need to create a boot script (embed.ipxe).

embed.ipxe

#!ipxe

:retry
dhcp || goto retry
ntp pool.ntp.org || goto retry

# we'll replace the url with our ipxe server later
chain -a https://example.com || goto retry

# unreachable
goto retry

According to iPXE documentation, we should select snp target for ARM devices. I didn’t notice that and tried several times before finding the right one. :P

# this should generate bin-arm64-efi/snp.efi
make CROSS_COMPILE=aarch64-linux-gnu- EMBED=embed.ipxe bin-arm64-efi/snp.efi -j`nproc`

Next, copy snp.efi to /efi/tools/snp.efi and we should get a working iPXE.

iPXE 1.21.1+ (gb00935) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP VLAN AoE EFI Menu
lan78xx_eth Waiting for PHY auto negotiation to complete....... done
Configuring (net0 b8:27:eb:ab:cb:dc)....... ok

The iPXE Server

In Valve Infra, a gateway fetches its kernel and initrd through public network. To protect our netboot from man-in-the-middle attack and unauthorized access, we need a HTTPS iPXE server with client certificate enabled. Basically, we need to get the following things done.

Get a SSL certificate for the domain.
Set up a HTTPS server with Nginx.
Create a CA and enable client verification in Nginx.
Issue client certificates using the CA we’ve just generated. Embed the ceritficate in our iPXE binary.

Here are some references.

mupuf’s blog
valve infra - ipxe boot server
ipxe - crypto
nginx - http_ssl_module

HTTPS Server

Thanks to ACME protocol and service providers like Let’s Encrypt, ZeroSSL, etc, we can get a SSL ceritificate in a few minutes. I used acme.sh to automate the process.

Remember to get a RSA certificate as iPXE does not support ECC certificate currently.

There used to be size limit for the server certificate because iPXE didn’t support fragmented TLS handshake. But now the issue has already been fixed. So feel free to get a SSL certificate of any length. :D

Now simply install and configure Nginx. Create a new configuration at /etc/nginx/sites-available and link it to /etc/nginx/sites-enabled. Add a server block with a normal SSL configuration. Set ssl_certificate to the SSL certificate and ssl_certificate_key to your private key. This should give you a working HTTPS server.

iPXE doesn’t support all cipher suites though. So take a look at their documentation and check your Nginx settings twice. I also found Mozilla SSL configuration generator useful. It generates Nginx configuration that works out of the box with iPXE when set to Intermediate mode.

You can use nmap to detect cipher suites your server allows.

nmap -script ssl-enum-ciphers -p 443 example.com

If all above are set correctly, you can replace the url in the iPXE boot script with your server address and host a simple iPXE script with Nginx, like

#!ipxe

echo hello from server

The RPi should be able to output hello from server.

Client Certificate

About how to create CA and issue client certificates, explanation from Valve Infra and iPXE are clear enough. It’s basically some OpenSSL thing. Let’s simply skip it.

I suppose you’ve already got a CA, a client certificate and corresponding private key.

Server

For server side, adding two lines to the server block of the Nginx configuration created above is enough.

ssl_client_certificate /path/to/ca
ssl_verify_client on;

Now, the resources of the server shouldn’t be accessible from client without certificates, like your browser.

Client

For client side, we need to rebuild iPXE to embed the client certificate and private key.

make bin-arm64-efi/snp.efi -j`nproc` \
    CROSS_COMPILE=aarch64-linux-gnu- \
    EMBED=embed.ipxe \
    PRIVKEY=client.key CERT=client.crt

Be sure to keep the generated snp.efi in a safe place. It contains unencryped private key of your client certificate.

Now, only your RPi could download scripts/binaries from your iPXE server!

An iPXE Issue

When tring to embed the private key in the iPXE binary, I encountered a weird issue. When doing a SSL handshake, iPXE complained that no private key was found corresponding to the client certificate, which I had definitely passed to make by PRIVKEY=....

I enabled more debug output and found iPXE failed parsing the private key in rsa_parse_mod_exp(). The private key is stored in ASN.1 structure. So I used openssl asn1parse to decode my private key. The output didn’t match the structure expected by the parsing function at all. At that point I realized something could be wrong with my private key.

It didn’t take me long to find that the private key I generated was in PKCS#8 format and the parsing function assumed it’s PKCS#1. I went to check OpenSSL documentation. Guess what? OpenSSL 3.0 and above outputs private key in PKCS#8 by default instead of PKCS#1, which explained exactly the problem I was having.

Adding a -traditional parameter should make OpenSSL output private key in the right format. After converting format of my private key to PKCS#1, I tried again. But it still didn’t work.

At last, I found iPXE also called OpenSSL in its Makefile to convert the private key to DER binary format, which, of course, converted my PKCS#1 input back to PKCS#8 again!

So the solution is simple. When OpenSSL version >= 3, append a -traditional to that command in Makefile. I made a small patch and created a PR. The maintainer of iPXE responded pretty fast and chose to add PKCS#8 formatted private key support instead. That’s way more better than my workaround!

What’s Next

After learning all those above and setting up the test environment, now it’s time to do some real work.

I’ll create a repository that generates SDcard image with UEFI bootloader for RPi (hope there’ll be more SBCs supported in the future). That image will serve as a base image so that we could ship different iPXE with it for different use, like a HTTP-only version for test machines and a HTTPS with client verification version for gateways. Actually, generating SDcard image for gateways from base image is also on my TODO list, which involves some changes in ipxe-boot-server.

There’s also a lot of questions left, such as

The way we load device tree
- Currently by RPi firmware or U-Boot.
- Should we load it by iPXE so that we could switch kernel and device tree together remotely?
The specification of the base image for both MBR and GPT
- So that we could put the iPXE binary in the right place when building the SDcard image.

Hope in my next blog there’ll be answers to all of them. :D

That’s all. Thank you for reading until the end! Hope to see you again soon.

← Run Boot2container on RPi - GSoC '23 The Project - GSoC '23 →