# SUSE Hackweek#18
### Author: Chester Lin <<[e-mail]>>
### Project: Playing openSUSE Tumbleweed on RISCV QEMU 4.0

###### tags: 'SUSE'

[Hackweek Link](https://hackweek.suse.com/18/projects/explore-risc-v#markdown-sourceC2H2G)

### Target
* **Running openSUSE by a completed boot-flow**
* ~~UEFI(*) -> GRUB -> kernel -> Systemd~~
* U-Boot(v) -> GRUB(*) -> Kernel(v) -> Systemd(v)

### Status
* **ENV SETUP**
* qemu-system-riscv64 (Done)
* -machine virt -smp 4 -m 10G
* riscv64 gcc cross-compiler & toolchain (Done)
* **UEFI: (TBD)**
* No official riscv support in EDK II.
* Found a custom branch developed by HP: https://github.com/HewlettPackard/RiscVEdk2
But there's no update since 2016 and all its patches have not been merged in upstream.
* The codebase is quite different to UDK2018.
* Tried merging with UDK2015, which is much closer to this branch's code base.
* Experienced cross-compiling errors in UDK2015. Found that we need to take care of tool-chain dependencies [bin locations, flags, gcc options, etc] used in EDKII BaseTools' template before compiling. UDK2015 can only support GCC4~GCC5. But the cross-compiler I used in x86 is based on GCC8 and the GCC I installed by zypper in RiscV VM is GCC9, which means I have to write down all environment variables.
* It could take time to try and error so I just leave it as another side project after hackweek.

* **openSUSE**
* Can only find rootfs packages in OBS.
https://download.opensuse.org/ports/riscv/tumbleweed/images/
* Created a qcow2 disk to store rootfs and boot binaries and mounted as a virtual block device [-device virtio-blk-device].
:::info
linux-8mug:/home/chester/workspace/qemu/riscv # qemu-nbd -c /dev/nbd3 opensuse.qcow2
linux-8mug:/home/chester/workspace/qemu/riscv # fdisk -l /dev/nbd3
Disk /dev/nbd3: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A531A85B-4D43-F242-9C4E-94F41824070C

Device Start End Sectors Size Type
/dev/nbd3p1 2048 411647 409600 200M EFI System
/dev/nbd3p2 411648 2508799 2097152 1G Linux swap
/dev/nbd3p3 2508800 104857566 102348767 48.8G Linux filesystem
:::

* **UBOOT**
* Cross-compiled a u-boot binary for RISCV6 * Cross-compiled a u-boot binary for RISCV64. It has to run in RISCV supervisor mode.

* Bug: Hanging at autoboot count down.
* Could be workaround by wrapping u-boot with BBL.
* No such issue when running u-boot in machine mode **[make qemu-riscv64_defconfig]**
* Stuided how to fake EFI parameters for debug purpose. **[efidebug tool]**
* Specific EFI configs are required.
* Stuided how to load & execute grub EFI binary from virtio block. **[bootefi / distro_bootcmd]**
* Can boot to manual compiled grub.
:::success
U-Boot 2019.07-rc3-00071-g38c2a8a001-dirty (Jun 26 2019 - 15:44:25 +0800)

CPU: rv64imafdcsu
Model: riscv-virtio,qemu
DRAM: 2 GiB
In: uart@10000000
Out: uart@10000000
Err: uart@10000000
Net:
Warning: virtio-net#1 using MAC address from ROM
eth0: virtio-net#1
Hit any key to stop autoboot: 0
=>
=>
=> efi boot add 0 opensuse virtio 0:1 EFI/BOOT/BOOTRISCV64.EFI; efi boot order 0; fatload virtio 0:1 ${kernel_addr_r} EFI/BOOT/BOOTRISCV64.EFI
Scanning disk virtio-blk#0...
Found 4 disks
147456 bytes read in 6 ms (23.4 MiB/s)
=> run distro_bootcmd

Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 51200.0 MB = 50.0 GB (104857600 x 512)
... is now current device
Scanning virtio 0:1...
** Unable to read file / **
Found EFI removable media binary efi/boot/bootriscv64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Booting: opensuse
Welcome to GRUB!

* **GRUB**
* U-Boot can't run grub efi built by OBS.
::: info
\# zypper in grub2 grub2-riscv64-efi
\# grub2-mkconfig -o <your-path>/grub.cfg
\# grub2-install --target=riscv64 --removable
:::
::: warning
=> run distro_bootcmd

Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 51200.0 MB = 50.0 GB (104857600 x 512)
... is now current device
Scanning virtio 0:1...
** Unable to read file / **
Found EFI removable media binary efi/boot/bootriscv64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Booting: opensuse
exception code: 2 , Illegal instruction , epc 1008 , ra fe6dcc86
\### ERROR ### Please RESET the board ###
:::
* Manually Compiled grub2.04-rc1 and installed in my VM.
* Exprerienced random crash **[unaligned pointer ...]** when running grub 2.0.4 and solved by an excellent patch from Andreas Schwab:
https://www.mail-archive.com/[e-mail]/msg29003.html
* Can run grub 2.04 from U-Boot but fail to load kernel in grub menu.
* Traced code and found that grub has not yet supported loading initrd & kernel on riscv. [grub-core/loader/riscv/linux.c]
* Trying to port ARM's code on RISCV with few modifications but still doesn't work. (TBD)
* Checked head.S and found that there's no obvious image header structure.
* Looks like the image header for RISCV is still under reviewing and has not been merged yet
* https://lore.kernel.org/patchwork/patch/1077883/
* Currently the LINUX kernel has not support EFI Stub on RISCV [no compliant PE/COFF headers]

* **Linux kernel boot-up with openSUSE Tumbleweed**
* **Unit Test:** The compiled kernel can boot as RISCV BBL (Berkeley BootLoader) payload.
* https://github.com/riscv/riscv-pk
* No console was present when wrapping opensuse factory's kernel image with BBL.
* Can by solved by applyng the following config ion QEMU: **CONFIG_SERIAL_OF_PLATFORM=y**
* Installed devel packages by zypper and tried self-compiling & installing kernel image and modules within VM. (OK)
* **Loading kernel & initrd from GRUB is still under construction in upstream.**
:::success
localhost:~ # uname -a
Linux localhost 5.1.0-rc6-130-default+ #7 SMP Tue Jun 11 16:58:35 CST 2019 riscv64 riscv64 riscv64 GNU/Linux
localhost:~ # lscpu
Architecture: riscv64
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
localhost:~ # cat /etc/os-release
NAME="openSUSE Tumbleweed"
VERSION="20190421"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20190421"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20190421"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
LOGO="distributor-logo"
localhost:~ #
:::

### TO-DO
* U-Boot hang issue if try running without BBL
* riscv linux/initrd loader in GRUB

### Reference
* RootFS of OpenSUSE TW
https://download.opensuse.org/ports/riscv/tumbleweed/images/
OpenSUSE:Factory:RiscV
https://build.opensuse.org/project/show/openSUSE:Factory:RISCV

* QEMU
https://github.com/qemu/qemu
riscv configuration tips:
https://github.com/riscv/riscv-qemu/wiki

* Tool-cahin for cross-compiling on x86-64
https://github.com/riscv/riscv-gnu-toolchain

* RISC-V proxy kernel
https://github.com/riscv/riscv-pk

### Other Notes:
Solved vda cannot be scanned issue:
* CONFIG_VIRTIO_BLK=y (m->y)
Turn on the following configurations:
* CONFIG_VIRTIO_MMIO=y