Building OpenWrt from Scratch for ARM64 UEFI ACPI VM

Posted on Last updated on

OpenWrt doesn't provide a combined disk image for ARM virtual machines, unlike what they did for x86 VMs. Meanwhile, their official ARM64 kernel release can't boot in UEFI environment. But we can still make it work by compiling it from source and building a disk image manually.

Since Arm community has various opinions on how to boot an Arm machine, such as UEFI + ACPI (widely used by commercial Arm servers as well as modern x86 systems), U-Boot + Device Tree (mostly used by embedded devices with limited resouces), and even UEFI + Device Tree (like Huawei L420 notebook I owned), I would suggest that don't expect OpenWrt will provide official support for UEFI + ACPI systems in recent days as it is designed to run on tiny routers.

Compile Kernel and Rootfs from Source

Don't be scared. With the help of buildroot, which could automatically prepare the cross-compilation toolchain we need, this step is much simple nowadays.

Note: My test environment is Ubuntu 21.10 ARM64 on Apple M1 Pro. It doesn't matter if you use a machine with a different system or architecture like AMD64, but you may need to take a few extra steps if so.

Install Dependencies

For Debian / Ubuntu users,

1
2
3
4
5
6
sudo apt update
sudo apt install build-essential ccache ecj fastjar file g++ gawk \
gettext git java-propose-classpath libelf-dev libncurses5-dev \
libncursesw5-dev libssl-dev python python2.7-dev python3 unzip wget \
python3-distutils python3-setuptools python3-dev rsync subversion \
swig time xsltproc zlib1g-dev

Note: The content of this sub-section is copied from the official guide. Take a look at it if this command is not applicable for your system.

Download the Source Code

1
2
3
4
5
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git tag
git checkout v21.02.2
./scripts/feeds update -a

Configure the Project

  1. Import the official configuration.

To save our effort, it is a good idea to modify an existing configuration instead of creating a new one.

1
2
wget https://downloads.openwrt.org/releases/21.02.2/targets/armvirt/64/config.buildinfo
cp config.buildinfo .config
  1. Add UEFI ACPI support.

Open file target/linux/armvirt/config-5.4, and append the following lines to the end of file.

1
2
3
4
5
CONFIG_EFI_STUB=y
CONFIG_EFI=y
CONFIG_EFI_VARS=y
CONFIG_ARCH_SUPPORTS_ACPI=y
CONFIG_ACPI=y
  1. Launch Memuconfig.
1
2
make menuconfig
make kernel_menuconfig

Tweak the configuration as you like, but you should clearly understand the consequence before you turn on and off something. Keeping default options is also fine.

Note: These commands will build the whole toolchain from source for the first time they are executed. The compilation process is very slow.

Build the Kernel and Rootfs

1
make -j $(nproc) defconfig download clean world

It will compile the kernel and all of the selected pre-installed utilities, then generate an EFI binary of Linux Kernel and an Ext4 / SquashFS partition image of Rootfs.

Verify the Firmware Image

The exciting moment comes. Let's test the kernel and rootfs we just built.

  1. Install QEMU.

For Ubuntu users, I would suggest to install virt-manager instead, which offers a helpful GUI wizard for QEMU.

1
sudo apt install virt-manager
  1. Launch a virtual machine.

The magical QEMU allows virtual machines to boot a kernel without a bootloader. That is a great feature enables us to test the kernel's functionality at the early stage.

1
qemu-system-aarch64 -m 512 -nographic -cpu cortex-a72 -smp 1 -M virt -kernel ~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-Image-initramfs -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd

Note: Image-initramfs is the kernel binary while it integrates the OpenWrt's Rootfs as initramfs, so this virtual machine will lose data each time it reboots.

Note: If you encounter this issue,

1
2
3
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: Failed to relocate kernel
EFI stub: ERROR: Failed to relocate kernel

The solution is to increase the memory capacity of your virtual machine. Empirically, it should be at least 256 MB.

Build the Disk Image

Considered that data loss is not acceptable, while not every hypervisor is capable of launching a kernel directly, we should put everything we built into a disk, or virtual machine's disk image.

To keep things simple, let's start from building a raw disk image, which is one of the virtual disk formats supported by QEMU.

Create an Empty Disk Image

1
2
3
4
tonny@vm:~$ dd if=/dev/zero of=disk.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.958229 s, 1.1 GB/s

This command will create an empty disk image. Feel free to replace the value of count to change the size of the disk. (size = 1 MB * 1024 = 1 GB)

Partition, Mount, and Format the Disk Image

  1. Partition the disk.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
tonny@vm:~$ fdisk disk.img

Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xb89701e3.

Command (m for help): g
Created a new GPT disklabel (GUID: 43B50BB3-20FD-3D4B-BFE1-50B5016F8059).

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-2097118, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118): +100M

Created a new partition 1 of type 'Linux filesystem' and of size 100 MiB.

Command (m for help): t
Selected partition 1
Partition type or alias (type L to list all): uefi
Changed type of partition 'Linux filesystem' to 'EFI System'.

Command (m for help): n
Partition number (2-128, default 2):
First sector (206848-2097118, default 206848):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (206848-2097118, default 2097118):

Created a new partition 2 of type 'Linux filesystem' and of size 923 MiB.

Command (m for help): p
Disk disk.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 43B50BB3-20FD-3D4B-BFE1-50B5016F8059

Device      Start     End Sectors  Size Type
disk.img1    2048  206847  204800  100M EFI System
disk.img2  206848 2097118 1890271  923M Linux filesystem

Command (m for help): w
The partition table has been altered.
Syncing disks.

A new GPT partition table with two partitions is written to the disk image.

  1. Mount the disk image as a logical disk.
1
2
3
4
5
6
tonny@vm:~$ sudo losetup -Pf disk.img
tonny@vm:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop5                       7:5    0    1G  0 loop
├─loop5p1                 259:4    0  100M  0 part
└─loop5p2                 259:5    0  923M  0 part

OS has recognized the two partitions, loop5p1 and loop5p2.

  1. Format the partitions.
1
2
tonny@vm:~/mnt$ sudo mkfs.vfat /dev/loop5p1
mkfs.fat 4.2 (2021-01-31)

We don't need to format the second partition (Rootfs) for now, because we can directly restore the partition image of Rootfs instead, which is already formatted with Ext4 File System.

  1. Mount ESP partition.

ESP partition contains the EFI executables of bootloaders (e.g., GRUB), as well as its configuration files. We can also put the kernel binary here.

Note: Some Linux distributions, like Ubuntu, will put their kernel in a third partition.

Unlike Rootfs, OpenWrt Build System won't generate an ESP partition image for ARM64 platform. That means we have to build ESP partition manually.

1
2
tonny@vm:~/mnt$ mkdir -p ~/mnt/esp
tonny@vm:~/mnt$ sudo mount /dev/loop5p1 ~/mnt/esp

Restore Rootfs Partition Image

1
2
3
4
5
6
7
8
tonny@vm:~$ sudo dd if=~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-rootfs-ext4.img of=/dev/loop5p2 bs=1M
104+0 records in
104+0 records out
109051904 bytes (109 MB, 104 MiB) copied, 0.613318 s, 178 MB/s
tonny@vm:~$ sudo resize2fs /dev/loop5p2
resize2fs 1.46.3 (27-Jul-2021)
Resizing the filesystem on /dev/loop5p2 to 236283 (4k) blocks.
The filesystem on /dev/loop5p2 is now 236283 (4k) blocks long.

The size of Rootfs image is about 128 MB, which implies that the file system inside will assume the partition size is about 128 MB. The size of our Rootfs partition is likely larger than this number, so we should notify the filesystem there is a change on the partition size.

Install GRUB to ESP Partition

Install ARM64 GRUB to Host

For Ubuntu users,

1
sudo apt install grub-efi-arm64-bin

Note: If your Host's architecture isn't ARM64, Apt may fail to find this package. Fortunately, thanks to Multiarch feature, we can easily install a package for other architectures. Take Ubuntu AMD64 as an example.

  1. Request for ARM64 architecture's packages.
1
sudo dpkg --add-architecture arm64
  1. Add an Apt Repository for ARM64.

Modify the file /etc/apt/source.list and add a ARM64 repository. Pay attention that ARM64 and AMD64 don't share the same repository, so we also need to add a filter for each repository. Here is an example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish main restricted universe multiverse
# deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish main restricted universe multiverse

deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-security main restricted universe multiverse
# deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-security main restricted universe multiverse

deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-updates main restricted universe multiverse
# deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-updates main restricted universe multiverse

deb [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-backports main restricted universe multiverse
# deb-src [ arch=amd64 ] https://mirrors.ustc.edu.cn/ubuntu/ impish-backports main restricted universe multiverse

deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish main restricted universe multiverse
# deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish main restricted universe multiverse

deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-security main restricted universe multiverse
# deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-security main restricted universe multiverse

deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-updates main restricted universe multiverse
# deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-updates main restricted universe multiverse

deb [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-backports main restricted universe multiverse
# deb-src [ arch=arm64 ] https://mirrors.ustc.edu.cn/ubuntu-ports/ impish-backports main restricted universe multiverse
  1. Install ARM64 GRUB
1
2
sudo apt update
sudo apt install grub-efi-arm64-bin

Generate EFI Executable

  1. Check Partition's UUIDs.
1
2
3
4
5
tonny@vm:~$ lsblk -o PATH,UUID,PARTUUID /dev/loop5
PATH         UUID                                 PARTUUID
/dev/loop5
/dev/loop5p1 CF95-2044                            3754ccb7-1920-2b41-9962-af81ac6a04b2
/dev/loop5p2 ff313567-e9f1-5a5d-9895-3ba130b4a864 e09a20c3-0ea7-0c48-b653-0482facd93db

Those UUIDs will be referred by the GRUB configurations.

  1. Write Early-stage GRUB Configuration.

Create a new file ~/grub-early.cfg, and write the following lines. This configuration will be hardcoded into GRUB's EFI binary.

1
2
3
search.fs_uuid CF95-2044 root
set prefix=($root)'/boot'
configfile $prefix/grub.cfg

Replace the UUID with your loop5p1's.

  1. Make GRUB EFI Executable.
1
2
3
4
# tonny@vm:~$ sudo mount /dev/loop5p1 ~/mnt/esp
tonny@vm:~$ sudo mkdir -p ~/mnt/esp/EFI/BOOT/
tonny@vm:~$ cd ~/mnt/esp/EFI/BOOT/
tonny@vm:~/mnt/esp/EFI/BOOT$ sudo grub-mkimage -c ~/grub-early.cfg -p /boot -o BOOTAA64.EFI -O arm64-efi boot chain configfile fat linux ls part_gpt reboot serial efi_gop search_fs_uuid

Note: It is not recommended to use grub-install here. One of its typical usages is,

1
sudo grub-install --target=arm64-efi --efi-directory ~/mnt/esp --bootloader-id=GRUB --boot-directory ~/mnt/esp/boot/

The hidden disgusting thing is, if you use GRUB provided by Ubuntu, this command will hardcode an important GRUB variable prefix='/EFI/ubuntu' to the EFI binary, and there is no way to change it.

Write Second-stage GRUB Configuration

1
2
3
4
tonny@vm:~/mnt/esp/EFI/BOOT$ cd ../..
tonny@vm:~/mnt/esp$ sudo mkdir boot
tonny@vm:~/mnt/esp$ cd boot/
tonny@vm:~/mnt/esp/boot$ sudo nano grub.cfg # or other text editor you feel comfortable with

The content of grub.cfg is,

1
2
3
4
5
6
7
8
9
10
11
12
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
terminal_input console serial; terminal_output console serial

set default="0"
set timeout="5"

menuentry "OpenWrt" {
	linux /boot/vmlinuz root=PARTUUID=e09a20c3-0ea7-0c48-b653-0482facd93db rootwait   console=tty0 console=ttyS0,115200n8 noinitrd
}
menuentry "OpenWrt (failsafe)" {
	linux /boot/vmlinuz failsafe=true root=PARTUUID=e09a20c3-0ea7-0c48-b653-0482facd93db rootwait   console=tty0 console=ttyS0,115200n8 noinitrd
}

Replace PARTUUIDs (not UUIDs) with your loop5p2's.

Copy Linux Kernel

1
tonny@vm:~/mnt/esp/boot$ sudo cp ~/openwrt/bin/targets/armvirt/64/openwrt-21.02.2-armvirt-64-Image vmlinuz

Verify the Disk Image

1
2
tonny@vm:~/mnt/esp/boot$ cd ~
tonny@vm:~$ qemu-system-aarch64 -m 512 -nographic -cpu cortex-a72 -smp 1 -M virt -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -drive format=raw,file=disk.img

If everything goes well, you could see your kernel is running happily. Enjoy it!

Note: You don't have to unmount the disk before launching the virtual machine. But you should sync the disk to make sure all the data cached in memory is written back.

1
tonny@vm:~/mnt/esp/boot$ sync

Clean up

1
2
tonny@vm:~/mnt$ sudo umount ~/mnt/esp
tonny@vm:~/mnt$ sudo losetup -d /dev/loop5

Launch VM with Virt-Manager

1
tonny@vm:~/mnt$ virt-manager

Note: Sometimes vert-manager requires permissions to run.

The recommended configuration:

  • Step 1:
    • Architecture: aarch64
    • Machine Type: virt
    • Import existing disk image
  • Step 2:
    • Browse ➡️ Add pool Home ➡️ Choose Volume disk.img
    • Choose OS: Generic Linux / OS
  • Step 3:
    • Memory: >= 256 MB
    • CPU: Any
  • Step 4:
    • Customize configuration before install
    • Network (LAN Port): Bridge / Macvtap Bridge
  • Configuration
    • Overview/Firmware: UEFI aarch64

Note: You can't change the firmware type after pre-install configuration.

References