Preparing and running QEMU ARM for Cubieboard (Part 2)

Preparing and running QEMU ARM for Cubieboard (Part 2)

This is part 2 of the QEMU Board Emulation post series.

In the previous post the basic steps for obtaining and compiling QEMU, U-Boot and Linux were presented. The only part that was missing for the complete system setup was the root filesystem. Also, all of the images were injected directly into emulated RAM memory using the QEMU’s loader mechanism.

In this post we will cover the following things

Obtaining Ubuntu root filesystem

There are several ways to obtain root filesystem for an embedded system

In this post we will use the prebuilt Ubuntu root filesystem. In some of the future posts the Buildroot and Yocto approaches will be covered.

Download Ubuntu root filesystem

The archive with the Ubuntu minimal 22.04 root filesystem can be obtained using the following step

# Prepare Ubuntu
wget -c https://cdimage.ubuntu.com/ubuntu-base/releases/22.04.3/release/ubuntu-base-22.04.3-base-armhf.tar.gz

The base roof filesystem has only the minimal set of configuration and applications, so some steps need to be taken before we can use it.

Prepare Ubuntu root filesystem

Ubuntu rootfs needs to be unpacked and copied to the rootfs partition

# Unpack base Ubuntu so changes can be made
sudo mkdir -p /run/tmp/rootfs
sudo tar xzvf ./ubuntu-base-22.04.3-base-armhf.tar.gz -C /run/tmp/rootfs/

Once rootfs has been unpacked the essential applications should be installed. That can be done using chroot.

Before running chroot, following packages need to be installed on the host machine

sudo apt install qemu-user-static binfmt-support

Once packages are installed, the following set of commands can be used to enter chroot environment

# Configure minimal Ubuntu rootfs

# Prepare chroot
sudo tee -a /run/tmp/rootfs/etc/resolv.conf << "EOF"
nameserver 8.8.8.8
EOF
sudo mount --bind /dev /run/tmp/rootfs/dev
sudo mount --bind /dev/pts /run/tmp/rootfs/dev/pts
sudo mount --bind /proc /run/tmp/rootfs/proc

# Enter chroot
sudo chroot /run/tmp/rootfs

After the last command the chroot environment will be entered. Following commands should be executed in the chroot environment.

# Prepare image
apt update
apt install -y --no-install-recommends \
    systemd \
    dbus \
    init \
    kmod \
    udev \
    iproute2 \
    iputils-ping \
    vim \
    sudo
# Add user ubuntu with password temppwd
useradd -G sudo -m -s /bin/bash ubuntu
echo ubuntu:temppwd | chpasswd
# Setup hostname and hosts file
echo mistra > /etc/hostname
echo 127.0.0.1 localhost > /etc/hosts
echo 127.0.1.1 mistra >> /etc/hosts
# Cleanup
apt clean
rm /var/lib/apt/lists/ports.ubuntu.com*
exit

After the work in chroot environment is done, the mounted directories from host should be unmounted.

# Unmount
sudo umount /run/tmp/rootfs/proc
sudo umount /run/tmp/rootfs/dev/pts
sudo umount /run/tmp/rootfs/dev

Finally, we can archive the rootfs directory, so it can be easily deployed later.

sudo tar cJvf ubuntu-minimal-22.04.tar.xz -C /run/tmp/rootfs .

Now that we have created the Ubuntu root filesystem, it needs to be supplied to the Linux running under QEMU. Two options are to create a SD card image or to access it over network. In this post we will focus on SD card approach.

Booting and running from SD card

QEMU supports emulation of the SD card interface. Depending on the board that is emulated, different SD card interfaces are available.

Preparing SD card image

QEMU provides tool for creating the emulated SD card, qemu-img. Before the tool can be used, the environment script created in the previous post needs to be sourced.

Creating an empty SD card image

The SD card image can be created using the following command:

# Create empty SD card
cd $PROJ_DIR
qemu-img create sd.img 1G
Formatting 'sd.img', fmt=raw size=1073741824

The last parameter that is passed is the size and for this work size of 1GB is selected. After executing the previous command, file sd.img will be created.

Before the SD card can be used to copy data, it has to be partitioned and formatted.

In order to simplify further work, a new line can be added to the $PROJ_DIR/env.sh with the path to the SD card

# env.sh update
export SD_IMG=$PROJ_DIR/sd.img

Partitioning SD card image

The SD card will be partitioned into two partitions.

The first one will be used for the kernel image and device tree files. The size will be 64MB and if will be later formatted as FAT32.

The second partition will take up the rest of the SD card and it will later be formatted as ext4.

We can use fdisk to check the status before and after partitioning:

fdisk -l ./sd.img
Disk ./sd.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

For formatting the SD card image we will use the sfdisk application.

Partitioning is done using the following command

# Partitioning the SD card
sfdisk ./sd.img << EOF
,64M,c,*
,,L,
EOF
Checking that no-one is using this disk right now ... OK

Disk ./sd.img: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Created a new DOS disklabel with disk identifier 0xc7b8b02e.
./sd.img1: Created a new partition 1 of type 'W95 FAT32 (LBA)' and of size 64 MiB.
./sd.img2: Created a new partition 2 of type 'Linux' and of size 959 MiB.
./sd.img3: Done.

New situation:
Disklabel type: dos
Disk identifier: 0xc7b8b02e

Device     Boot  Start     End Sectors  Size Id Type
./sd.img1  *      2048  133119  131072   64M  c W95 FAT32 (LBA)
./sd.img2       133120 2097151 1964032  959M 83 Linux

The partition table has been altered.
Syncing disks.

The format of the sfdisk partitioning is start,size,type,bootable, meaning we are creating

  • first partition, from the beggining of the card (actually will start from 1st MB), size 64MB, type c (FAT32) and bootable
  • second partition, that starts after the first partition, which will fill the available space, type L (ext4).

Formatting partitions

After the SD card has been partitioned, and before the partitions can be formated using the mkfs application, the SD card must be “plugged in”, i.e. the partitions must be recognized by the host operating system. This is done using the kpartx tool

# 'Inserting' the SD card
sudo kpartx -av ./sd.img
add map loop0p1 (252:0): 0 131072 linear 7:0 2048
add map loop0p2 (252:1): 0 1964032 linear 7:0 133120

The value 0 in the output loop0p1 can differ from system to system and that is why we are using the -v switch, so the value is printed. After this command, the partitions are visible in the system under /dev/mapper/loop0p1 and /dev/mapper/loop0p2.

In order to format partitions, following commands will be used

# Formatting SD card partitions
sudo mkfs.vfat -F 32 -n "boot" /dev/mapper/loop0p1
sudo mkfs.ext4 -L rootfs /dev/mapper/loop0p2

Copying data to SD card image

After the partitions have been formatted, the data can be copied. In order to copy data, the partitions need to be mounted.

The /run/mount/ will be used as base for the mount points, where boot and rootfs directories will be created.

Boot partition

The boot partition is mounted in the following way

# Mounting boot partition
sudo mkdir -p /run/mount/boot
sudo mount /dev/mapper/loop0p1 /run/mount/boot

Linux kernel zImage file and Device tree file need to be copied to the boot partition

# Copying files to boot partition
sudo cp $ZIMAGE /run/mount/boot
sudo cp $DTB /run/mount/boot

After data has been copied, the boot partition can be unmounted using

# Umount
sudo umount /run/mount/boot

Root filesystem partition

The rootfs partition is mounted in the following way

# Mounting rootfs partition
sudo mkdir -p /run/mount/rootfs
sudo mount /dev/mapper/loop0p2 /run/mount/rootfs

Ubuntu rootfs needs to be unpacked and copied to the rootfs partition

# Copying Ubuntu files to boot partition
sudo tar xJvf ./ubuntu-minimal-22.04.tar.xz -C /run/mount/rootfs/

Before the rootfs partition is unomounted, the kernel modules must be installed

# Copying kernel modules and setting permissions
cd $PROJ_DIR/linux/build_cubieboard
sudo make ARCH=arm INSTALL_MOD_PATH=/run/mount/rootfs modules_install
sync

After data has been copied, the rootfs partition can be unmounted using

# Umount
sudo umount /run/mount/rootfs

Now the SD card can be “unplugged” from the system using

# Unplug SD card
sudo kpartx -d $PROJ_DIR/sd.img

Running QEMU with SD card image

After the SD card is ready, the QEMU can be started using the following command

# Run QEMU with SD card
cd $PROJ_DIR
qemu-system-arm -M cubieboard -m 1G -kernel $UBOOT -nographic \
                -drive file=sd.img,format=raw,if=sd

Once U-Boot starts, it can be used to copy kernel image and device tree file into RAM memory, as well as to set up the linux kernel command line

# Load items into memory and start kernel
fatload mmc 0:1 42000000 zImage
fatload mmc 0:1 43000000 sun4i-a10-cubieboard.dtb
setenv bootargs "console=ttyS0 root=/dev/mmcblk0p2 rw"
bootz 0x42000000 - 0x43000000

After kernel boots the Ubuntu banner appears and user can login with the information created in chroot.

# Logged in Ubuntu
Ubuntu 22.04.3 LTS mistra ttyS0

mistra login: ubuntu
Password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.8.1 armv7l)
ubuntu@mistra:~$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Ubuntu 22.04.3 LTS"

Making complete SD card image and running in QEMU

So far we used the -kernel option to pass the U-Boot binary to QEMU. However, the Cubieboard QEMU model supports booting from a complete SD card image.

In order to make a complete SD card image, we need to copy the SPL (secondary program loader) and u-boot.bin files to appropriate locations. This is where the information about boot flow comes in handy.

Once the Allwinner-A10 chip is powered on, the BROM starts execution. That is hardcoded in the chip itself (hence ROM). It will perform minimal set up, and then try to load SPL from available memory interfaces, one of which is the SD card interface. If it finds the correct signature at the correct location, it will load SPL into on-chip RAM and start executing it.

SPL continues with initialization, configures DRAM and then searches for u-boot binary to load.

For Allwinner-A10 build, the output binary u-boot-sunxi-with-spl.bin holds both SPL and u-boot. The image should be copied to the 8kB offset of the SD card (for details consult this page).

The dd command can be used to copy that file

# Copy U-Boot to SD card
cd $PROJ_DIR
dd if=u-boot/build_cubieboard/u-boot-sunxi-with-spl.bin of=$SD_IMG bs=1024 seek=8 conv=notrunc

After the U-Boot image has been copied, the QEMU can now be started without the -kernel parameter, as

# Run QEMU with SD card
qemu-system-arm -M cubieboard -m 1G -nographic \
                -drive file=sd.img,format=raw,if=sd

In order to streamline the experience, a u-boot script can be made which executes the commands for loading the kernel and device tree files, so it will be executed automatically every time.

The script boot.cmd should contain the same commands we typed in manually before

fatload mmc 0:1 42000000 zImage
fatload mmc 0:1 43000000 sun4i-a10-cubieboard.dtb
setenv bootargs "console=ttyS0 root=/dev/mmcblk0p2 rw"
bootz 0x42000000 - 0x43000000

In order to be able to use it, it has to be converted to boot.scr using the following command

mkimage -C none -A arm -T script -d boot.cmd boot.scr

and copied to the boot partition of the SD card, following the steps from Boot partition section.

Enable network in QEMU

The system should have internet access for many interesting use-cases.

QEMU, by default, creates a network connection to host machine. However, that network has limitations where emulated system can access outside network, but it is not accessible from the host system.

Besides the default network connection, QEMU supports different methods for enabling network access where emulated system is accessible from host system:

  • using tap interface,
  • using Bridged adapter network.

Both methods require manual setup before QEMU is started. The tap interface method is simpler, but the Bridged adapter network can make the emulated system accessible from rest of the network also, not only the host system.

In this example, we will use the tap interface to enable network connection.

More details about QEMU networking support can be found here.

The tap interface can be configured in the following way

# Create tap interface
sudo tunctl -u $(whoami) -t qemu-tap0
sudo ifconfig qemu-tap0 192.168.123.1
sudo route add -net 192.168.123.0 netmask 255.255.255.0 dev qemu-tap0
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

With commands above the QEMU instance will be able to ping and access host computer, and other way around, but it will not be able to access internet. In order to enable internet access, following commands are required on the host (set <interface> to network interface that is used on host machine for accessing internet)

# Enable guest internet access
sudo iptables -t nat -A POSTROUTING -o <interface> -j MASQUERADE
sudo iptables -I FORWARD 1 -i qemu-tap0 -j ACCEPT
sudo iptables -I FORWARD 1 -o qemu-tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT

NOTE: These commands need to be executed every time the host system is rebooted.

The commands for NAT networking were adapted from here. The approach with the bridged networking is a bit more complex and can be found here.

After the tap interface has been created the QEMU can be started.

Run QEMU with network

The QEMU with networking can be started in the following way:

# Start QEMU with networking
qemu-system-arm -M cubieboard -m 1G -nographic \
                -drive file=sd.img,format=raw,if=sd \
                -net nic -net tap,ifname=qemu-tap0,script=no

Since we are not using any DHCP server, the IP address should be set manually. Following commands can be used to configure static IP and default google DNS

sudo ip addr add 192.168.123.101/24 dev eth0
sudo ip route add default via 192.168.123.1 dev eth0
sudo ip link set eth0 up
sudo resolvectl dns 3 8.8.8.8

After these commands, the system should have internet connection, and can also be ping’ed from host

# Ping from host
ping 192.168.123.101 -c 4
PING 192.168.123.101 (192.168.123.101) 56(84) bytes of data.
64 bytes from 192.168.123.101: icmp_seq=1 ttl=64 time=0.887 ms
64 bytes from 192.168.123.101: icmp_seq=2 ttl=64 time=0.726 ms
64 bytes from 192.168.123.101: icmp_seq=3 ttl=64 time=0.735 ms
64 bytes from 192.168.123.101: icmp_seq=4 ttl=64 time=0.743 ms

--- 192.168.123.101 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.726/0.772/0.887/0.066 ms

From the gues, the external network access can be simply verified with ping

# Ping www.google.com
ping www.google.com -c 4
PING www.google.com (142.250.74.36) 56(84) bytes of data.
64 bytes from arn09s22-in-f4.1e100.net (142.250.74.36): icmp_seq=1 ttl=53 time=9.57 ms
64 bytes from arn09s22-in-f4.1e100.net (142.250.74.36): icmp_seq=2 ttl=53 time=6.89 ms
64 bytes from arn09s22-in-f4.1e100.net (142.250.74.36): icmp_seq=3 ttl=53 time=6.95 ms
64 bytes from arn09s22-in-f4.1e100.net (142.250.74.36): icmp_seq=4 ttl=53 time=8.15 ms

--- www.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3007ms
rtt min/avg/max/mdev = 6.888/7.889/9.569/1.092 ms

In order to make the changes persist across reboots, the network file 20-wired.network can be created in /etc/systemd/network/ with the following content

[Match]
Name=eth0

[Network]
Address=192.168.123.101/24
Gateway=192.168.123.1
DNS=8.8.8.8

The systemd-networkd service should be enabled as well, using

sudo systemctl enable systemd-networkd

After rebooting the system, the internet connection will still be available.

Github helper scripts

In this repository I have added several scripts which cover most of the things presented in Parts 1 and 2.

Following scripts are present:

  • install-qemu.bash - downloads toolchain and Ubuntu root filesystem, and compiles QEMU, U-Boot and Linux
  • prepare-qemu.bash - creates an SD card image based on compiled files and Ubuntu root filesystem
  • enable-networking.bash - initializes a tap network interface so QEMU instance can have networking
  • mount-sd-card.bash - mounts rootfs partition of the SD card
  • umount-sd-card.bash - umounts rootfs partition of the SD card
  • run-qemu.bash - runs QEMU instance

Summary

In this post we have covered different methods for running the complete Linux system with bootloader in QEMU.

Subscribe
If you would like to get information as soon as new content is published, please subscribe to the "MistraSolutions newsletter".
    Built with Kit

    Similar Posts

    Yocto for Cubieboard in QEMU - Step-by-step

    Yocto for Cubieboard in QEMU - Step-by-step

    Creating Custom QEMU Memory Mapped Device

    Creating Custom QEMU Memory Mapped Device