ARM64 Bootstrap for VyOS

Updated 2023-12-20

This repo can assist in bootstrapping a VyOS image for arm64, without depending on the artifacts already generated by Jenkins.

The generated image is an ISO that will boot on ARM Embedded and Server Base Boot systems (e.g those that use EFI)

There are also a few minor hacks/patches applied which should be upstreamed shortly.

If you are unfamiliar with the VyOS build process, see the Build VyOS page in the documentation.

Current hardware targets:

• QEMU ARM64 Virtual Machine with EDKII
• Traverse Technologies Ten64

Note: only basic routing and NAT functionality has been tested at this time.

Build requirements:

• ARM64 host (no cross compiling)

• It is best to use a Debian machine as the build host. Issues have been encountered with AppArmor on Ubuntu

• Working docker or podman-docker installation

• sudo access from the current user

  Due to the need to do some privileged operations, the build process can't be done in a 'rootless' container. You must use sudo to execute docker/podman.

Other efforts:

Graham Hayes built his own version here: https://github.com/grahamhayes/vyos-build/. The main difference is that this one doesn't use a container to do builds. See Ten64 forum discussion

Usage:

My vyos-build branch is setup as a submodule of this repository, so you should do a recursive clone first:

git clone --recursive https://github.com/mcbridematt/vyos-arm64-builder.git

Run ./build.sh (as root or sudo) to build the 'vyos-build' container and then packages and image.

sudo ./build.sh

An ISO image will be generated under vyos-build/build/vyos-{VYOS_VERSION}.{YYYYMMDDHHMM}-arm64.iso

Package commit pinning

The source versions of each component are "pinned" to a commit or branch in repos.txt to ensure this environment will produce a working image. To use the latest available current/development versions, copy repos.development.txt in its place.

Testing inside a VM

The ./testimg.sh script can be used to boot the generated ISO image as a QEMU/KVM virtual machine.

You need to run the script on a genuine arm64 host with virtualization capabilities.

(It is possible to run it under emulation, but YMMV).

An EDK2 "bios" binary is required, under Debian you can use the qemu-efi-aarch64 package or grab a build from retrage/edk2-nightly.

cp vyos-build/build/vyos-{VYOS_VERSION}.{YYYYMMDDHHMM}-arm64.iso vyos.iso
sudo ./testimg.sh

(Hint: Use Ctrl-X to immediately exit qemu).

Running on real hardware (Ten64)

On a Ten64 a suitable block medium (NVMe SSD, USB drive or SD card) is needed. It might be possible to fit VyOS into the onboard NAND/ubifs in the future.

Write the VyOS ISO to a USB drive using dd or a similar tool:

dd if=vyos.iso of=/dev/sda

Then boot into the 'live' VyOS system and do an install:

vyos@vyos:~$ install image

See the VyOS Installation doc for more information.

Known issues:

• There are some hacks to bypass or remove certain aspects that can cause trouble:
  • udev interface renaming rules and biosdevname handling - need to debug issues with this not working properly.
  • The GRUB configuration has been modified so console= is never specified. On most Arm hosts that specify the system console in the device tree (chosen node) or ACPI this will mean the correct serial console is chosen out of the box.
• Previously this repository generated a qcow2 image, but this is currently not possible due to recent changes in vyos-build. Hopefully it can be bought back soon.