The toy project for the weekend was to get rid of Virtualbox and migrate everything to KVM-based virtualization (QEMU, libvirt, Gnome Boxes). Here's an overview of my learnings.

§ Why dumping proprietary virtualization platforms

My virtualization needs are pretty basic, I run virtual machines to try new linux distributions or to keep a couple of Windows machines for browser testing or some quaint Windows-only software. Occasionally I need to run macOS.

Years ago I used to use VMware Player but it became impossible to even simply download the free player without signing in and checking hundreds of checkboxes with unintelligible corporate blurb - ok I got it you don't me to use your software anymore.

So I had settled on Virtualbox because of its ease of use, but again, after being bought by Oracle things went downhill. While Virtualbox is still GPL, there are horror stories of Virtualbox phoning home when installing the proprietary extension and calling the vultures on you to pay hefty fees. Well, fuck you Oracle and all the IT companies for paying mojito drinks to managers with legacy software thanks but no thanks.

§ QEMU, KVM, libvirt... some clarifications

KVM is an amazing piece of software. At a very high level, it's a kernel module that interfaces hardware resources directly to the virtual machines to give maximum performances. The Linux Hypervisor has then two hardware interfaces to the CPU: either Intel VT or AMD-V.

QEMU is a project initially wrote by Fabrice Bellard that has expanded as the virtualization solution for a myriad of architectures and CPUs (half of them unknown to me).

QEMU is a full-featured, very well-documented AND user-hostile command line tool, so there are frontends for the rest of us. But its job is to spin a single VM, it's the equivalent of launching VMware Player.

QEMU can also run without KVM but then it gets slow on VM hungry of resources.

libvirt is a VM manager (invisible to the user) akin to their closed source solutions such are Virtualbox and VMware Workstation (plus more enterprise stuff).

On top of libvirt there are many GUI and command-line frontends. Perhaps virt-manager is one of most well-known. I've also been suggested to try Gnome Boxes, a very convenient and basic solution if you just want to quickly get started. As usual, the command line in the end is more convenient to use so I'll also mention virsh.

§ Down the rabbit hole

So, let's get to the meat. Like I mentioned, what I wanted to accomplish is moving all my VMs under libvirt. Using Gnome Boxes to create Linux and Windows VM is very easy (click, click, yes, yes, done), so nothing to see here.

Moving the macOS VM away from Virtualbox was more interesting. Thanks to this repo I quickly get started with a QEMU script to run macOS Catalina. But a QEMU VM is not manageable by libvirt, it must be imported, so I need an intermediate XML export file to attach the VM to libvirt. Having done that I can now see the VM from virt-manager but not from Gnome Boxes. Uhm ... there is also an open issue on the repo, let's see what we can do.

Let's step back and first things let's familiarize with some libvirt terminology (I can see the footprint of Red Hat business jargon): a "domain" is a VM that can be executed under libvirt. The domain is described and imported ("defined") using an XML file. The domain can then be removed and destroyed ("undefined") from the VM manager.

The domain XML is created using the great virsh command line tool. My current config is:

$ virsh
virsh # version
Compiled against library: libvirt 5.4.0
Using library: libvirt 5.4.0
Using API: QEMU 5.4.0
Running hypervisor: QEMU 4.0.0

Let's generate a domain XML from the QEMU shell script. A snippet of the bash script:

qemu-system-x86_64 \
    -enable-kvm \
    -m 2G \
    -machine q35,accel=kvm \
    -smp 4,cores=2 \
    -cpu Penryn,vendor=GenuineIntel,kvm=on,+sse3,+sse4.2,....
    ...

According to the documentation I need to rewrite that script putting everything in one line and feed the new script to virsh:

$ virsh domxml-from-native qemu-argv basic.args > macos.xml

This step took me a truckload of time because QEMU parameters do not match one-to-one to libvirt domain XML format (specs here). I had to look up a lot of errors, painfully tweak the QEMU script to get a valid XML.

Also some funny WTF moments, like when I could not figure out the correct QEMU syntax to assign a drive to the correct bus. I could use the index parameter, for example, but how?

-drive id=SystemDisk,if=none,index=XXX,file=MyDisk.qcow2

well, by incrementing XXX I observed that <address> parameters are treated like a single binary number 🤦‍♂️, therefore the XML changes like this:️

"index=0" gives <address type='drive' controller='0' bus='0' target='0' unit='0'/>
"index=1" gives <address type='drive' controller='0' bus='0' target='0' unit='1'/>
"index=2" gives <address type='drive' controller='0' bus='0' target='1' unit='0'/>
"index=3" gives <address type='drive' controller='0' bus='1' target='0' unit='1'/>
"index=4" gives <address type='drive' controller='1' bus='0' target='0' unit='0'/>

Anyway, in the end I gave up and modified directly the XML file (snippet):

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>macOS-test</name>
  <uuid>bb1dee99-d97f-497f-8238-f2df783deef0</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>OVMF_CODE.fd</loader>
    <nvram>OVMF_VARS-1024x768.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
  ...
  </features>
  ...

Now let's feed this XML into libvirt. Optionally one can also validate the XML (against this schema) while creating the VM. This command will create a new VM called "macOS-test".

virsh define macos.xml [--validate]

XML validation often fails even if the XML is "good" because the parser is overly-zealous.

To remove the VM, let's use the opposite command. The --keep-nvram is needed in case the VM has shared resources with other VMs.

virsh undefine macOS-test [--keep-nvram]

Of course we can now go the other way round and export a domain XML into a QEMU script (docs):

virsh domxml-to-native --domain macOS-test --format qemu-argv > macos.sh

Again, caveat applies, the generated bash script con contain resources not available outside of libvirt 🤦‍♂️ 🤦‍♂️.

So, it's clear that this conversion between QEMU bash scripts and domain XML is problematic, they recommend using it only to migrate QEMU scripts or even manually craft the XML file for new VMs.

The XML file can also be used to move a VM around:

virsh dumpxml macOS-test > export.xml

virsh create export.xml

Finally let's spin up the VM:

virsh start bb1dee99-d97f-497f-8238-f2df783deef0

or kill it (forced shutdown, equals to rip the power cord):

virsh destroy bb1dee99-d97f-497f-8238-f2df783deef0

§ Have the VM show up in Gnome Boxes

Now the virtual machine shows up in virt-manager but we also want to use Gnome Boxes. Why does it not appear? Uhm ... some more digging.

Turns out another important piece of knowledge I was missing. A QEMU VM can run inside libvirt either as privileged user (qemu:///system) or as normal user (qemu:///session). The latter is advisable unless there are certain requirements, here is an article about that.

The reason why it didn't appear is that Gnome Boxes (when run as user) only has access to user VMs, while virt-manager shows you both, although separately.

So in the end, all I needed to do was ensuring the VM was imported in libvirt userspace 🤦‍♂️🤦‍♂️🤦‍♂️, and so:

virsh -c qemu:///session define macos.xml

This distinction only became clear when I started using virsh directly, see next paragraph.

§ Networking

When you run your KVM guest as unprivileged user, by default you cannot access the virtual machine through any network interface, fullstop. And this is pretty annoying.

Your only chance without going crazy with subnetworks or playing with iptables is to configure a bridge.

Since this a topic that kept me busy for a while, it is explained in detail in a separate article.

§ Quick reference for virsh

ref: https://libvirt.org/manpages/virsh.html

Open the virsh shell:

$ virsh

Connect to the hypervisor as unprivileged user:

# connect qemu:///session

List all VMs (should see all the VMs available to this user, also those on Gnome Boxes):

# list --all

Import a VM:

# define --file macos.xml

The new VM should now be visible on Gnome Boxes (and virt-manager)

Delete a VM

# undefine --domain macOS-test --keep-nvram

§ References

A gazillion of links on stack overflow, Red Hat documentation, man qemu-system-x86_64 and whatnot.

Warning, registering an AppleID might get yourself banned by Tim Apple, remember to change the MAC address and UUID of the machine before attempting to login/register an Apple account.

Packages needed for libvirt and company:

• gnome-boxes

dmeventd gnome-boxes libdevmapper-event1.02.1 libgtk-vnc-2.0-0 libgvnc-1.0-0 liblvm2cmd2.03 libosinfo-1.0-0 libosinfo-bin libphodav-2.0-0 libphodav-2.0-common libreadline5 libspice-client-glib-2.0-8 libspice-client-gtk-3.0-5 libusbredirhost1 libvirt-daemon libvirt-daemon-driver-storage-rbd libvirt-glib-1.0-0 libvirt0 lvm2 osinfo-db spice-client-glib-usb-acl-helper thin-provisioning-tools

• virt-viewer

libgovirt-common libgovirt2 virt-viewer

Example usage: virt-viewer -c qemu:///session --domain-name debian-11-testing-bullseye

• virt-manager

gir1.2-gtk-vnc-2.0 gir1.2-libosinfo-1.0 gir1.2-libvirt-glib-1.0 gir1.2-spiceclientglib-2.0 gir1.2-spiceclientgtk-3.0 python3-libvirt python3-libxml2 virt-manager virtinst