Integrate RISCOF + RISC-V Architecture Tests for RVVM

contributed by < st10740 >

Targets :

• 研讀 RVVM 文件和原始程式碼 (不用花太多時間 Google 搜尋)，學習 RV32 相關的指令集模擬
• 試著將 https://github.com/riscv-non-isa/riscv-arch-test 在 RVVM 中執行，需要用
  RV32I 和相關的 extension (如 RV32IMAC)，過程中你可能會遇到 RISCOF 的議題，可參見:
  https://hackmd.io/@Risheng/rv32emu
• RVVM 原本是系統模擬器，若要執行個別的 ELF 程式，需要運用其 ELF
  loader，但實作可能不完整，因此你或許會需要排除相關技術問題，過程中善用 GitHub Issue 提交和討論
• 最終你應該要讓 RISC-V Architecture Tests 完整在 RVVM 中運作，輸出其相容性報告

RVVM Building

Prerequisites

OpenSBI

OpenSBI is an open-source implementation of the RISC-V Supervisor Binary Interface (SBI) specifications, which is system call style calling convention between Supervisor (S-mode OS) and Supervisor Execution Environment (SEE).

Required GUN Toolchain - riscv-gnu-toolchain

Prerequisites (Ubuntu)

$ sudo apt-get install autoconf automake autotools-dev curl python3 python3-pip libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev ninja-build git cmake libglib2.0-dev

Building

$ git clone https://github.com/riscv/riscv-gnu-toolchain
$ cd riscv-gnu-toolchain
$ ./configure --prefix=/opt/risv32-linux --with-arch=rv32imac --enable-linux
$ sudo make linux

Configure the environment variable

$ vim .bashrc

# .bashrc
export PATH=$PATH:/opt/riscv32-linux/bin
export CROSS_COMPILE=/opt/riscv32-linux/bin/riscv32-unknown-linux-gnu-
export ARCH=riscv

Source .bashrc at the first time to run the script. The script will be ran everytime starting a new terminal.

$ source .bashrc

Check riscv32-unknown-linux-gnu-gcc

$ riscv32-unknown-linux-gnu-gcc -v

Expected output

Using built-in specs.
COLLECT_GCC=riscv32-unknown-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/opt/riscv32-linux/bin/../libexec/gcc/riscv32-unknown-linux-gnu/13.2.0/lto-wrapper
Target: riscv32-unknown-linux-gnu
Configured with: /home/ivywang/riscv-gnu-toolchain/gcc/configure --target=riscv32-unknown-linux-gnu --prefix=/opt/risv32-linux --with-sysroot=/opt/risv32-linux/sysroot --with-pkgversion=gc891d8dc23e --with-system-zlib --enable-shared --enable-tls --enable-languages=c,c++,fortran --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libsanitizer --disable-nls --disable-bootstrap --src=.././gcc --disable-multilib --with-abi=ilp32 --with-arch=rv32imac --with-tune=rocket --with-isa-spec=20191213 'CFLAGS_FOR_TARGET=-O2    -mcmodel=medlow' 'CXXFLAGS_FOR_TARGET=-O2    -mcmodel=medlow'
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.2.0 (gc891d8dc23e) 

Building OpenSBI

$ cd
$ git clone https://github.com/riscv-software-src/opensbi.git
$ cd opensbi
$ make PLATFORM="generic"

The file fw_jump.bin needed to build RVVM can be found in the directory build/platform/generic/firmware.

Linux Kernel

Building

$ cd
$ git clone https://github.com/torvalds/linux.git
$ cd linux
$ make ARCH=riscv CROSS_COMPILE=riscv32-unknown-linux-gnu- defconfig Image

The linux kernel image is generated at arch/riscv/boot/Image.

...
OBJCOPY arch/riscv/boot/Image
Kernel: arch/riscv/boot/Image is ready

However, using this method to build Linux kernel will generate RV64I kernel image, leeding to booting kernel failed while running RVVM because of incompatible. So I use make menuconfig to change Platform type > Base ISA to RV32I and build it using make again.

This time, I can successfully boot the Linux kernel but still encounter the same problem as fewletter and the solution given by the developer team is to download config_tiny_6.2 file, which includes everything needed to function on RVVM. Next, change its name to .config, put it into linux directory. And don't forget to use make menuconfig to enter configuration page to change the Platform type > Base IS to RV32I. Finally, build linux kernel using make again.

RISC-V Linux Root Filesystem

$ cd
$ wget https://buildroot.org/downloads/buildroot-2023.02.8.tar.gz
$ tar -zxvf buildroot-2023.02.8.tar.gz
$ cd buildroot-2023.02.8/
$ sudo apt-get -y install build-essential libncurses-dev cpio rsync file unzip bc
$ make menuconfig

We will enter the configuration page. Go to Target options page. Setting the Target Architecture to RISCV, and Architecture Size to 32-bit.

Target Architecture (RISCV)  --->                            
Target Architecture Variant (General purpose (G))  --->      
Target Architecture Size (32-bit)  --->                     
Target ABI (ilp32d)  --->                                   
Target Binary Format (ELF)  ---> 

Setting the Filesystem images to ext2/3/4 root filesystem.

[*] ext2/3/4 root filesystem                               
ext2/3/4 variant (ext2 (rev1))  ---> 

Building

$ make -j$(nproc)

Building

Build using GNU Make

$ cd
$ git clone https://github.com/LekKit/RVVM.git
$ cd RVVM
$ make
$ cd release.linux.riscv

Running

Copy the needed file for building into release.linux.riscv directory

$ cp ~/opensbi/build/platform/generic/firmware/fw_jump.bin .
$ cp ~/linux/arch/riscv/boot/Image .
$ cp ~/buildroot-2023.02.8/output/images/rootfs.ext2 .

Run

$ ./rvvm_riscv fw_jump.bin -k Image -i rootfs.ext2 -rv32 -m 1G -smp 2 -res 1280x720

RISCOF - RISC-V Compatibility Framework

I use RISCOF framework to test RVVM with riscv-arch-test.

Introduction

RISCOF is the python based RISC-V Compatibility Framework which enables testing of a RISC-V target (hard or soft implementations) against a standard RISC-V golden reference model using a suite of RISC-V architectural assembly tests.

Execution flow for users

1. User provide a YAML specification which captures the choices made in the implementation & a python plugin which can enable compilation and simulation of a test on the implementation.
2. The input YAML is first validated using the RISCV-CONFIG tool to confirm the implementation choices adhere to the those defined by the RISC-V ISA spec. The output of this is the standardized/normalized YAML spec containing all the information of the implementation.
3. The normalized YAML is then fed to the Test Selector utility to filter and select tests from the test-pool which are applicable to the implementation of the user.
4. The normalized YAML is also fed into the reference model’s python plugin to configure the model to mimic the implementation as close as possible.
5. The test-list is next forwarded to both, the user and reference defined python plugins, to initiate compilation and execution of the tests on the respective platforms.
6. RISCOF, thus declares a test to have passed on the implementation only when the its signature matches the signature produced by the reference model executing the same test.
7. RISCOF generates an HTML report which provides details of the implementation and tests that were passed/failed by the implementation.

Learn with Quickstart

First, I follow the Quickstart page of RISCOF document to learn how to use RISCOF framework. In this case, spike is DUT and SAIL-RISCV is reference model. The goal is to perform a validation check between them and output the report.

Install Python

Since I have installed python3 before, I just check the version of it.

$ python3 -V
Python 3.10.12
$ pip3 -V
pip 23.3.2 from /home/ivywang/.local/lib/python3.10/site-packages/pip (python 3.10)

Using Virtualenv for Python

Because the recommended version of python is 3.6 in the RISCOF documentation, I use pyenv to install python3.6, which prevents other software from breaking while switching between multiple python versions.

Installing pyenv and setting environment variables.

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(pyenv init -)"' >> ~/.bashrc

$ exec "$SHELL"
$ pyenv --version
pyenv 2.3.35-7-g6e3b91a8

Installing python3.6.

$ pyenv install 3.6.0

However, I encounter Segmentation fault while installing it on Ubunbu 22.04.

Downloading Python-3.6.0.tar.xz...
-> https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz
Installing Python-3.6.0...

BUILD FAILED (Ubuntu 22.04 using python-build 2.3.35-7-g6e3b91a8)

Inspect or clean up the working tree at /tmp/python-build.20240113141719.67041
Results logged to /tmp/python-build.20240113141719.67041.log

Last 10 log lines:
if test "xupgrade" != "xno"  ; then \
	case upgrade in \
		upgrade) ensurepip="--upgrade" ;; \
		install|*) ensurepip="" ;; \
	esac; \
	LD_LIBRARY_PATH=/tmp/python-build.20240113141719.67041/Python-3.6.0 ./python -E -m ensurepip \
		$ensurepip --root=/ ; \
fi
Segmentation fault (core dumped)
make: *** [Makefile:1064: install] Error 139

I follow the issue comment to first install clang and install again.

$ sudo apt install clang -y
$ CC=clang pyenv install 3.6.0

Unfortunately, another error pops up which is The Python ssl extension was not compiled. Missing the OpenSSL lib?. It seems like I encounter some problems about OpenSSL. So, I follow Common build problems collected by pyenv team to install the missing OpenSSL and its headers, and install Python3.6 again.

$ sudo apt install libssl-dev
$ CC=clang pyenv install -v 3.6.0
...
Successfully installed pip-21.3.1 wheel-0.37.1
Installed Python-3.6.0 to /home/ivywang/.pyenv/versions/3.6.0
/tmp/python-build.20240113151610.99325 ~
~

Finally, Python3.6 is installed successfully. Checking the version of it in the same shell.

$ pip3 install --upgrade pip
$ pyenv shell 3.6.0

$ python --version
Python 3.6.0
$ pip --version
pip 21.3.1 from /home/ivywang/.pyenv/versions/3.6.0/lib/python3.6/site-packages/pip (python 3.6)

Install RISCOF

Installing

$ pip install git+https://github.com/riscv/riscof.git

Add the path to RISCOF executable file to PATH by adding the following line to .bashrc

export PATH=$PATH:/home/ivywang/.pyenv/versions/3.6.0/lib/python3.6/site-packages

You can get your path to RISCOF executable file by using command line pip show riscof. And my output is

Name: riscof
Version: 1.25.3
Summary: RISC-V Architectural Test Framework
Home-page: https://github.com/riscv/riscof
Author: InCore Semiconductors Pvt. Ltd.
Author-email: [e-mail]
License: BSD-3-Clause
Location: /home/ivywang/.pyenv/versions/3.6.0/lib/python3.6/site-packages
Requires: click, GitPython, Jinja2, pytz, riscv-config, riscv-isac
Required-by: 

The path behind Location is what you need.

Source .bashrc

$ source .bashrc

Test RISCOF

$ risof --help
Usage: riscof [OPTIONS] COMMAND [ARGS]...

Options:
  --version                       Show the version and exit.
  -v, --verbose [info|error|debug]
                                  Set verbose level
  --help                          Show this message and exit.

Commands:
  arch-test     Setup and maintenance for Architectural TestSuite.
  coverage      Run the tests on DUT and reference and compare signatures
  gendb         Generate Database for the Suite.
  run           Run the tests on DUT and reference and compare signatures
  setup         Initiate Setup for riscof.
  testlist      Generate the test list for the given DUT and suite.
  validateyaml  Validate the Input YAMLs using riscv-config.
ivywang@ivywang-virtual-machine:~$ 

Install RISCV-GNU Toolchain

I have installed it previously at here.

Install Plugin Models

Install 2 RISC-V reference models: Spike and SAIL.

Spike

Installing

$ sudo apt-get install device-tree-compiler
$ git clone https://github.com/riscv-software-src/riscv-isa-sim.git
$ cd riscv-isa-sim
$ mkdir build
$ cd build
$ ../configure --prefix=/opt/spike
$ make
$ sudo make install

Configure the environment variable

$ vim ~/.bashrc

# .bashrc
export PATH=$PATH:/opt/spike/bin

Execute spike to check it installed successfully

$ spike

Spike RISC-V ISA Simulator 1.1.1-dev

usage: spike [host options] <target program> [target options]
...

SAIL

First, I follow sail install document to install sail via opam.

We need to install OCaml with version 4.08.1 or newer according to sail install document, however, the online document of RISCOF hasn't been updated, which leeds us to install version 4.06.1, generating building problem later.

$ sudo apt-get install opam
$ opam switch create 5.1.0
$ eval $(opam config env)
$ sudo apt-get install build-essential libgmp-dev z3 pkg-config zlib1g-dev
$ opam install sail
$ which sail
$ sail --help

Sail 0.17.1 (sail @ opam-v2.1.2)
usage: sail <options> <file1.sail> ... <fileN.sail>
...

Next, I follow sail-riscv Getting started section and RISCOF document to build sail.

$ git clone https://github.com/riscv/sail-riscv.git
$ cd sail-riscv
$ ARCH=RV32 make
$ ARCH=RV64 make
$ ln -s sail-riscv/c_emulator/riscv_sim_RV64 /usr/bin/riscv_sim_RV64
$ ln -s sail-riscv/c_emulator/riscv_sim_RV32 /usr/bin/riscv_sim_RV32

Create Neccesary Env Files

$ mkdir demoRISCOF
$ cd demoRISCOF
$ riscof setup --dutname=spike

The above command will generate the following files and directories in the /demoRISCOF:

├── config.ini               # configuration file for riscof
├── sail_cSim                # reference plugin templates
│   ├── env
│   │   ├── link.ld          # Reference linker script
│   │   └── model_test.h     # Reference model specific header file
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── riscof_sail_cSim.cpython-36.pyc
│   └── riscof_sail_cSim.py  # Reference model python plugin.
└── spike                    # DUT plugin templates
    ├── env
    │   ├── link.ld          # DUT linker script
    │   └── model_test.h     # DUT specific header file
    ├── __pycache__
    │   └── riscof_model.cpython-36.pyc
    ├── riscof_spike.py      # DUT python plugin
    ├── spike_isa.yaml       # DUT ISA yaml based on riscv-config
    └── spike_platform.yaml  # DUT Platform yaml based on riscv-config

Change content of genreated files

config.ini

[RISCOF]
ReferencePlugin=sail_cSim
ReferencePluginPath=/home/ivywang/demoRISCOF/sail_cSim
DUTPlugin=spike
DUTPluginPath=/home/ivywang/demoRISCOF/spike

[spike]
pluginpath=/home/ivywang/demoRISCOF/spike
ispec=/home/ivywang/demoRISCOF/spike/spike_isa.yaml
pspec=/home/ivywang/demoRISCOF/spike/spike_platform.yaml
target_run=1

[sail_cSim]
pluginpath=/home/ivywang/demoRISCOF/sail_cSim
+ PATH=/home/ivywang/sail-riscv/c_emulator

riscof_spike.py

class spike(pluginTemplate):
    __model__ = "spike"

    #TODO: please update the below to indicate family, version, etc of your DUT.
-    __version__ = "XXX"
+    __version__ = "1.1.1"

Change GUN Toolchain from riscv32-unknown-elf-gcc to riscv32-unknown-linux-gnu-gcc, which is used in this project.

- self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+ self.compile_cmd = 'riscv{1}-unknown-linux-gnu-gcc -march={0} \
        -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -g\
        -T '+self.pluginpath+'/env/link.ld\
        -I '+self.pluginpath+'/env/\
        -I ' + archtest_env + ' {2} -o {3} {4}'

riscof_sail_cSim.py

Again, Change GUN Toolchain from riscv32-unknown-elf-gcc to riscv32-unknown-linux-gnu-gcc.

    def initialise(self, suite, work_dir, archtest_env):
        self.suite = suite
        self.work_dir = work_dir
-        self.objdump_cmd = 'riscv{1}-unknown-elf-objdump -D {0} > {2};'
+        self.objdump_cmd = 'riscv{1}-unknown-linux-gnu-objdump -D {0} > {2};'
-        self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+        self.compile_cmd = 'riscv{1}-unknown-linux-gnu-gcc -march={0} \
         -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles\
         -T '+self.pluginpath+'/env/link.ld\
         -I '+self.pluginpath+'/env/\
         -I ' + archtest_env

-        objdump = "riscv{0}-unknown-elf-objdump".format(self.xlen)
+        objdump = "riscv{0}-unknown-linux-gnu-objdump".format(self.xlen)
        if shutil.which(objdump) is None:
            logger.error(objdump+": executable not found. Please check environment setup.")
            raise SystemExit(1)
-        compiler = "riscv{0}-unknown-elf-gcc".format(self.xlen)
+        compiler = "riscv{0}-unknown-linux-gnu-gcc".format(self.xlen)
        if shutil.which(compiler) is None:
            logger.error(compiler+": executable not found. Please check environment setup.")
            raise SystemExit(1)

Cloning the Architectural Tests

$ riscof --verbose info arch-test --clone

If we follow the RISCOF document to typo riscof --verbose info arch-tests --clone, there will be an error Error: No such command 'arch-tests'. pops up. According to this issue, we need to use riscof --verbose info arch-test --clone instead.

Running RISCOF

Validate the input yaml files are configured correctly

This step internally calls the riscv-config on both the isa and platform yaml files indicated in the config.ini file.

$ riscof validateyaml --config=config.ini

The output

...
INFO | Initiating Validation
INFO | No Syntax errors in Input Platform Yaml. :)
INFO | Dumping out Normalized Checked YAML: /home/ivywang/demoRISCOF/riscof_work/spike_platform_checked.yaml

Generate the list of tests that need to be run on the models

$ riscof testlist --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env

The output

...
INFO | Selecting Tests.

The tests are listed in the file: riscof_work/test_list.yaml.

Run the tests

The last step is to run the tests on the each of the models and compare the signature values to guarantee correctness.

$ riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env

Unfortunately, when following the RISCOF document to install Python 3.6, the error AttributeError: 'Token' object has no attribute 'test' appears. Then, I opt to use pyenv to install Python 3.8 and riscof, running the command again, the program functions properly.

Finally, The report is generated and opened as below:

...
INFO | Test report generated at /home/ivywang/demoRISCOF/riscof_work/report.html.
INFO | Opening test report in web-browser

The report is about validation check between spike (DUT in this case) and SAIL-RISCV (Reference model in this case). The ISA is RV32IMCZicsr_Zifencei. It passes all the 90 tests, which include RV32IMC, Zifencei and privilege test.

Integrate RISCOF + riscv-arch-test for RVVM

After reading the Understanding RISCOF Inputs, Building your Model Plugin and Running RISCV-ARCH-TESTS sections, I have comprehensive understanding about how to execute riscv-arch-test on my own DUT using RISCOF.

In this case, I will generate report about validation check between RVVM (DUT) and SAIL-RISCV (Reference model).

Building Model Plugin

$ mkdir rvvm-riscof
$ cd rvvm-riscof
$ riscof setup --refname=sail_cSim --dutname=rvvm

There are one file config.ini and two directories rvvm and sail_cSim generated under rvvm-riscof. The rvvm-riscof/rvvm directory has the following structure:

├── env
│   ├── link.ld
│   └── model_test.h
├── __pycache__
│   └── riscof_model.cpython-38.pyc
├── riscof_rvvm.py
├── rvvm_isa.yaml
└── rvvm_platform.yaml

Update config.ini

• Verify that the path in the config.ini file generated by riscof is correct.
• Add path to the SAIL binaries and path to the RVVM binaries to the config because they are not in the $PATH.

[RISCOF]
ReferencePlugin=sail_cSim
ReferencePluginPath=/home/ivywang/rvvm-riscof/sail_cSim
DUTPlugin=rvvm
DUTPluginPath=/home/ivywang/rvvm-riscof/rvvm

[rvvm]
pluginpath=/home/ivywang/rvvm-riscof/rvvm
ispec=/home/ivywang/rvvm-riscof/rvvm/rvvm_isa.yaml
pspec=/home/ivywang/rvvm-riscof/rvvm/rvvm_platform.yaml
+ PATH=/home/ivywang/RVVM/release.linux.riscv
target_run=1

[sail_cSim]
pluginpath=/home/ivywang/rvvm-riscof/sail_cSim
+ PATH=/home/ivywang/sail-riscv/c_emulator

Update GNU Toolchain in rvvm/riscof_rvvm.py & sail_cSim/riscof_sail_cSim.py

Change the name of executable file from rvvm to the correct name rvvm_riscv. This is in the __init__ function of rvvm class.

- self.dut_exe = os.path.join(config['PATH'] if 'PATH' in config else "","rvvm")
+ self.dut_exe = os.path.join(config['PATH'] if 'PATH' in config else "","rvvm_riscv")

Change GUN Toolchain from riscv32-unknown-elf-gcc to riscv32-unknown-linux-gnu-gcc, which is required by RVVM. This is in the initialise function of rvvm class.

- self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+ self.compile_cmd = 'riscv{1}-unknown-linux-gnu-gcc -march={0} \
         -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -g\
         -T '+self.pluginpath+'/env/link.ld\
         -I '+self.pluginpath+'/env/\
         -I ' + archtest_env + ' {2} -o {3} {4}'

In riscof_sail_cSim.py, We also need to change GUN Toolchain from riscv32-unknown-elf-gcc to riscv32-unknown-linux-gnu-gcc.

The following code is in the initialise function of sail_cSim class.

- self.objdump_cmd = 'riscv{1}-unknown-elf-objdump -D {0} > {2};'
+ self.objdump_cmd = 'riscv{1}-unknown-linux-gnu-objdump -D {0} > {2};'
- self.compile_cmd = 'riscv{1}-unknown-elf-gcc -march={0} \
+ self.compile_cmd = 'riscv{1}-unknown-linux-gnu-gcc -march={0} \
         -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles\
         -T '+self.pluginpath+'/env/link.ld\
         -I '+self.pluginpath+'/env/\
         -I ' + archtest_env

The following code is in the unknown function of sail_cSim class.

- objdump = "riscv{0}-unknown-elf-objdump".format(self.xlen)
+ objdump = "riscv{0}-unknown-linux-gnu-objdump".format(self.xlen)
        if shutil.which(objdump) is None:
            logger.error(objdump+": executable not found. Please check environment setup.")
            raise SystemExit(1)
- compiler = "riscv{0}-unknown-elf-gcc".format(self.xlen)
+ compiler = "riscv{0}-unknown-linux-gnu-gcc".format(self.xlen)

Update the macros in the rvvm/env/model_test.h

Reference

• OpenSBI Deep Dive
• Linux Root filesystem
• 使用 RISCOF 測試 rv32emu