I was trying to port OpenStack components to FreeBSD OS, and it will leverage bhyve to spawn virtual machines. The bhyve VM (denoted as "guest VM") will be nested because the development environment is in a VM, i.e., the OpenStack itself runs inside a VM (denoted as "host VM"). However, when trying to spawn a guest VM on the host VM (which is Linux KVM), the host VM will become unresponsive.

The environment I've got:

• A bare-metal server running Harvester
• A FreeBSD-13.2-RELEASE AMD64 VM as the host VM running on Harvester

The same issue happened on a Proxmox VE server, too. Both Harvester and Proxmox VE are QEMU/KVM-based solutions.

The following steps describe how I create the bhyve guest VM with the aid of vm-bhyve tool. Most of the steps are from this web page

sudo pkg install vm-bhyve bhyve-firmware
sudo zfs create zroot/bhyve
sudo zfs set recordsize=64K zroot/bhyve
sudo zfs create zroot/bhyve/.templates

sudo sysrc vm_enable=yes
sudo sysrc vm_dir="zfs:zroot/bhyve"
echo 'vmm_load="YES"' | sudo tee -a /boot/loader.conf

In /zroot/bhyve/.templates/uefi.conf

loader="uefi"

cpu=2
cpu_sockets=1
cpu_cores=2

memory=4G

ahci_device_limit="8"

network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0.img"

sudo vm init
sudo vm switch create public
sudo vm switch add public vtnet0
sudo vm create -t uefi -s 10G freebsd-test
sudo vm install freebsd-test /zroot/bhyve/FreeBSD-13.2-RELEASE-amd64-bootonly.iso

After spinning up the guest VM, the host VM froze immediately. The only thing I can do is to forcibly reboot the host VM from the hypervisor on the bare-metal server.