Run FreeRTOS and multitasking on VexRiscv

# Run FreeRTOS and multitasking on VexRiscv contributed by < `OscarShiang` > :::success :pencil: The implementation is in [OscarShiang/FreeRTOS-Briey](https://github.com/OscarShiang/FreeRTOS-Briey) ::: ## Briey SoC VexRiscv is a risc-v hardware implementation written by SpinalHDL. This implementation not only provides the basic extension like [I] and [M] but also other plugins like CSR plugin for previledged registers and instructions to do switching between machine mode and user mode. Briey SoC is a basic implementation based on VexRiscv. It has the following peripherals to use: - Timer - UART - VGA ![](https://raw.githubusercontent.com/SpinalHDL/VexRiscv/master/assets/brieySoc.png) It has several well-documented samples in [VexRiscvSocSoftware](https://github.com/SpinalHDL/VexRiscvSocSoftware/tree/master/projects/briey) as well. So I choose to port Briey SoC to FreeRTOS. :::info :information_source: The installation and setup for Briey simulator can be found in [Final Project:Run FreeRTOS on VexRiscv and access the peripherals such as VGA.](https://hackmd.io/@4a740UnwQE6K9pc5tNlJpg/H1olFPOCD) ::: ## Prerequiements ### OpenOCD SpinalHDL has its fork of OpenOCD. It can be accessed at [SpinalHDL / openocd_riscv](https://github.com/SpinalHDL/openocd_riscv) We can download the code and build it from source: ```shell $ ./bootstrap $ ./configure --enable-ftdi --enable-dummy $ make ``` ### RISC-V Toolchain It is unlikely to use the toolchain generated by xPack because xPack one needs us to provide `_sbrk` for newlib supportance, while FreeRTOS seems not provide these functions. I use `riscv64-unknown-elf` toolchain, which can be generated by [crosstool-ng](https://github.com/crosstool-ng/crosstool-ng) I follow the steps to config and build - Clone and generate local config ```shell $ ./configure --enable-local $ make $ ./ct-ng menuconfig ``` - Enable the folowing configs ``` CT_EXPERIMENTAL=y CT_ARCH_RISCV=y CT_ARCH_64=y CT_ARCH_ARCH=rv32ima CT_ARCH_ABI=ilp32 CT_MULTILIB=y CT_DEBUG_GDB=y ``` - Build the GNU toolchain for RISC-V. ``` $ ./ct-ng build ``` After setting up the environment, we can now build the hardware and simulator: ```shell $ cd <directory_to_VexRiscv> $ sbt "runMain vexriscv.demo.Briey" $ cd src/test/cpp/briey $ make run ``` ## FreeRTOS Config ### Config headers SiFive has defined some options in RISC-V portable header, we should disable them because we do not have these features ```cpp #define portasmHAS_SIFIVE_CLINT 0 #define portasmHAS_MTIME 0 #define portasmADDITIONAL_CONTEXT_SIZE 0 ``` ### Memory layout As the linker script in VexRiscvSocSoftwares, the init assembly and stack should be placed on on-chip-ram at `0x80000000`, while other text and data sections should be placed on SDRAM starting from `0x40000000`. ## Run FreeRTOS After changing the above settings, we can now build our demo and try to run ```sh # open GDB console $ cd <demo_directory> $ make DEBUG=1 # for debug symbol $ riscv64-unknown-elf-gdb build/RTOSdemo.axf ``` Meanwile, we should run the simulator from VexRiscv directory and OpenOCD for accepting gdb connection: ```sh # open simulator $ cd <directory_to_vexriscv>/src/test/cpp/briey $ make run # turn on OpenOCD $ cd <directory_to_openocd_riscv> $ src/openocd -f tcl/interface/jtag_tcp.cfg -c 'set BRIEY_CPU0_YAML ../VexRiscv/cpu0.yaml' -f tcl/target/briey.cfg # open GDB and load ELF file $ cd <directory_to_free_rtos_briey_demo> $ riscv64-unknown-elf.gdb build/RTOSdemo.axf (gdb) target remote :3333 (gdb) monitor reset halt (gdb) load (gdb) continue ``` ## Debugging ### Task switching To perform task switching, we should ensure that the order of initialze / save / restore registers must be same. we can look into the comment for context to save: ```cpp /* <...> * The RISC-V context is saved to FreeRTOS tasks in the following stack frame, * where the global and thread pointers are currently assumed to be constant so * are not saved: * * mstatus * x31 * x30 * x29 * x28 * x27 * x26 * x25 * x24 * x23 * x22 * x21 * x20 * x19 * x18 * x17 * x16 * x15 * x14 * x13 * x12 * x11 * pvParameters * x9 * x8 * x7 * x6 * x5 * portTASK_RETURN_ADDRESS * [chip specific registers go here] * pxCode */ ``` To make sure that we can switch between threads without failures, we must always save / restore all the registers in order. The code for initialize thread's stack has a similiarity intension ```cpp .align 8 .func pxPortInitialiseStack: csrr t0, mstatus /* Obtain current mstatus value. */ andi t0, t0, ~0x8 /* Ensure interrupts are disabled when the stack is restored within an ISR. Required when a task is created after the schedulre has been started, otherwise interrupts would be disabled anyway. */ addi t1, x0, 0x188 /* Generate the value 0x1880, which are the MPIE and MPP bits to set in mstatus. */ slli t1, t1, 4 or t0, t0, t1 /* Set MPIE and MPP bits in mstatus value. */ addi a0, a0, -portWORD_SIZE store_x t0, 0(a0) /* mstatus onto the stack. */ addi a0, a0, -(22 * portWORD_SIZE) /* Space for registers x11-x31. */ store_x a2, 0(a0) /* Task parameters (pvParameters parameter) goes into register X10/a0 on the stack. */ addi a0, a0, -(6 * portWORD_SIZE) /* Space for registers x5-x9. */ store_x x0, 0(a0) /* Return address onto the stack, could be portTASK_RETURN_ADDRESS */ addi t0, x0, portasmADDITIONAL_CONTEXT_SIZE /* The number of chip specific additional registers. */ chip_specific_stack_frame: /* First add any chip specific registers to the stack frame being created. */ beq t0, x0, 1f /* No more chip specific registers to save. */ addi a0, a0, -portWORD_SIZE /* Make space for chip specific register. */ store_x x0, 0(a0) /* Give the chip specific register an initial value of zero. */ addi t0, t0, -1 /* Decrement the count of chip specific registers remaining. */ j chip_specific_stack_frame /* Until no more chip specific registers. */ 1: addi a0, a0, -portWORD_SIZE store_x a1, 0(a0) /* mret value (pxCode parameter) onto the stack. */ ret .endfunc ``` For some unknown problem, we can not directly use the mechanism provided by FreeRTOS to do context switching. So we have to invent our own task switching routine through save / restore context macro: ```cpp .macro portSAVE_CONTEXT /* make room in stack */ addi sp, sp, -REGBYTES * 31 /* Save Context */ STORE x1, 0(sp) STORE x2, 1 * REGBYTES(sp) STORE x3, 2 * REGBYTES(sp) STORE x4, 3 * REGBYTES(sp) STORE x5, 4 * REGBYTES(sp) STORE x6, 5 * REGBYTES(sp) STORE x7, 6 * REGBYTES(sp) STORE x8, 7 * REGBYTES(sp) STORE x9, 8 * REGBYTES(sp) STORE x10, 9 * REGBYTES(sp) STORE x11, 10 * REGBYTES(sp) STORE x12, 11 * REGBYTES(sp) STORE x13, 12 * REGBYTES(sp) STORE x14, 13 * REGBYTES(sp) STORE x15, 14 * REGBYTES(sp) STORE x16, 15 * REGBYTES(sp) STORE x17, 16 * REGBYTES(sp) STORE x18, 17 * REGBYTES(sp) STORE x19, 18 * REGBYTES(sp) STORE x20, 19 * REGBYTES(sp) STORE x21, 20 * REGBYTES(sp) STORE x22, 21 * REGBYTES(sp) STORE x23, 22 * REGBYTES(sp) STORE x24, 23 * REGBYTES(sp) STORE x25, 24 * REGBYTES(sp) STORE x26, 25 * REGBYTES(sp) STORE x27, 26 * REGBYTES(sp) STORE x28, 27 * REGBYTES(sp) STORE x29, 28 * REGBYTES(sp) STORE x30, 29 * REGBYTES(sp) STORE x31, 30 * REGBYTES(sp) /* Store current stackpointer in task control block (TCB) */ LOAD t0, pxCurrentTCB //pointer STORE sp, 0x0(t0) .endm ``` ```cpp .macro portRESTORE_CONTEXT /* Load stack pointer from the current TCB */ LOAD sp, pxCurrentTCB LOAD sp, 0(sp) LOAD x1, 0(sp) portasmRESTORE_ADDITIONAL_REGISTERS /* Restore registers, Skip global pointer because that does not change */ LOAD x6, 5 * REGBYTES(sp) LOAD x7, 6 * REGBYTES(sp) LOAD x8, 7 * REGBYTES(sp) LOAD x9, 8 * REGBYTES(sp) LOAD x10, 9 * REGBYTES(sp) LOAD x11, 10 * REGBYTES(sp) LOAD x12, 11 * REGBYTES(sp) LOAD x13, 12 * REGBYTES(sp) LOAD x14, 13 * REGBYTES(sp) LOAD x15, 14 * REGBYTES(sp) LOAD x16, 15 * REGBYTES(sp) LOAD x17, 16 * REGBYTES(sp) LOAD x18, 17 * REGBYTES(sp) LOAD x19, 18 * REGBYTES(sp) LOAD x20, 19 * REGBYTES(sp) LOAD x21, 20 * REGBYTES(sp) LOAD x22, 21 * REGBYTES(sp) LOAD x23, 22 * REGBYTES(sp) LOAD x24, 23 * REGBYTES(sp) LOAD x25, 24 * REGBYTES(sp) LOAD x26, 25 * REGBYTES(sp) LOAD x27, 26 * REGBYTES(sp) LOAD x28, 27 * REGBYTES(sp) LOAD x29, 28 * REGBYTES(sp) LOAD x30, 29 * REGBYTES(sp) LOAD x31, 30 * REGBYTES(sp) LOAD x5, 29 * REGBYTES(sp) addi x5, x5, 0x08 csrrw x0, mstatus, x5 LOAD x5, 2 * REGBYTES(sp) addi sp, sp, REGBYTES * 31 ret .endm ``` Then combine these 2 macros to perform task switching ```cpp .global vPortYield: csrci mstatus, 8 portSAVE_CONTEXT jal vTaskSwitchContext portRESTORE_CONTEXT ``` And then we replace the `ecall` to use our own `vPortYield` to save and restore another context: ```diff diff --git a/Source/portable/GCC/RISC-V/portmacro.h b/Source/portable/GCC/RISC-V/portmacro.h index de26afa..c7ffdcc 100644 --- a/Source/portable/GCC/RISC-V/portmacro.h +++ b/Source/portable/GCC/RISC-V/portmacro.h @@ -92,9 +92,7 @@ not need to be guarded with a critical section. */ /* Scheduler utilities. */ extern void vTaskSwitchContext( void ); +extern void vPortYield(); +#define portYIELD() vPortYield() -#define portYIELD() __asm volatile( "ecall" ); #define portEND_SWITCHING_ISR( xSwitchRequired ) do { if( xSwitchRequired ) vTaskSwitchContext(); } while( 0 ) #define portYIELD_FROM_ISR( x ) portEND_SWITCHING_ISR( x ) /*-----------------------------------------------------------*/ ``` :::warning :warning: It works and the task switching seems to be fine. But this modification breaks the code using by other RISC-V port and cause RISC-V-Qemu-virt port to fail during compiling. ::: ### Replace vPortYield with ecall Because changing the macro provided by FreeRTOS will break the structure. However, executing `ecall` may trigger a SWI and jump to the address of `mtvec`, which is `0x80000020` and its value into can not be changed :::info :information_source: Revised the permission for accessing to `mtvec` seems to be abnormal. It can jump to the address kept in `mtvec`, but it can not print out string through UART. ::: To redirect PC to FreeRTOS trap handler, I rewrite the `start.S` to put the jump instructions to that address. ```cpp .section .init _init: j _start nop nop nop nop nop nop nop main_entry: la a0, freertos_risc_v_trap_handler jr a0 ``` Each instruction is 4-byte-wided, there are 8 instructions in _init block. Thus the start address of main_entry block can be set to 0x20, when we set the code to `.init` section. The main_entry will be set to `0x80000020` After setting the exception vector, we can now use ecall to trigger SWI and jump to the address. ### Illegal instruction Although we have set the exception vector and use `ecall`, the system still gets into trap during task switching. Since the system provides its exception trap, we can simplily dump the value of `t0`, `t1` and `t2` to see which code causes the exception: ```cpp is_exception: csrr t0, mcause /* For viewing in the debugger only. */ csrr t1, mepc /* For viewing in the debugger only */ csrr t2, mstatus j is_exception ``` So we can retrieve the value from `mcause` and `mepc` Then we can use the following table to distinguish the problem we have: | Interrupt | Exception Code | Description | | --------- | --------------- | ------------------------------ | | 1 | 0 | User software interrupt | | 1 | 1 | Supervisor software interrupt | | 1 | 2 | Reserved | | 1 | 3 | Machine software interrupt | | 1 | 4 | User timer interrupt | | 1 | 5 | Supervisor timer interrupt | | 1 | 6 | Reserved | | 1 | 7 | Machine timer interrupt | | 1 | 8 | User external interrupt | | 1 | 9 | Supervisor external interrupt | | 1 | 10 | Reserved | | 1 | 11 | Machine external interrupt | | 1 | ≥12 | Reserved | | 0 | 0 | Instruction address misaligned | | 0 | 1 | Instruction access fault | | 0 | 2 | Illegal instruction | | 0 | 3 | Breakpoint | | 0 | 4 | Load address misaligned | | 0 | 5 | Load access fault | | 0 | 6 | Store/AMO address misaligned | | 0 | 7 | Store/AMO access fault | | 0 | 8 | Environment call from U-mode | | 0 | 9 | Environment call from S-mode | | 0 | 10 | Reserved | | 0 | 11 | Environment call from M-mode | | 0 | 12 | Instruction page fault | | 0 | 13 | Load page fault | | 0 | 14 | Reserved | | 0 | 15 | Store/AMO page fault | | 0 | ≥16 | Reserved | As a result, the value of `mcause` is `2`, which means there is a invalid instruction. The problem is generated by the setting of Briey: ```scala object BrieyConfig{ def default = { <...> new CsrPlugin( config = CsrPluginConfig( <...> mtvecAccess = CsrAccess.NONE, mtvecInit = 0x80000020l, mepcAccess = CsrAccess.READ_WRITE, mscratchGen = false, mcauseAccess = CsrAccess.READ_ONLY, mbadaddrAccess = CsrAccess.READ_ONLY, mcycleAccess = CsrAccess.NONE, minstretAccess = CsrAccess.NONE, ecallGen = false, wfiGenAsWait = false, ucycleAccess = CsrAccess.NONE, uinstretAccess = CsrAccess.NONE ) ), <...> } ``` In the above piece of config, we can see that we were not generated the appropriate setting we want. After rerun the compilation of verilog and verilator, we can now successfully switch between 2 threads: ``` Hello FreeRTOS! 0: Tx: Transfer1 0: Rx: Blink1 0: Tx: Transfer2 0: Rx: Blink2 0: Tx: Transfer1 0: Rx: Blink1 0: Tx: Transfer2 0: Rx: Blink2 0: Tx: Transfer1 0: Rx: Blink1 0: Tx: Transfer2 0: Rx: Blink2 0: Tx: Transfer1 0: Rx: Blink1 0: Tx: Transfer2 ``` ### ISR stack overflow ```cpp BaseType_t xPortStartScheduler( void ) { extern void xPortStartFirstTask( void ); #if( configASSERT_DEFINED == 1 ) { volatile uint32_t mtvec = 0; /* Check the least significant two bits of mtvec are 00 - indicating single vector mode. */ __asm volatile( "csrr %0, mtvec" : "=r"( mtvec ) ); configASSERT( ( mtvec & 0x03UL ) == 0 ); /* Check alignment of the interrupt stack - which is the same as the stack that was being used by main() prior to the scheduler being started. */ configASSERT( ( xISRStackTop & portBYTE_ALIGNMENT_MASK ) == 0 ); #ifdef configISR_STACK_SIZE_WORDS { memset( ( void * ) xISRStack, portISR_STACK_FILL_BYTE, sizeof( xISRStack ) ); } #endif /* configISR_STACK_SIZE_WORDS */ } #endif /* configASSERT_DEFINED */ ``` Since the initialization of `xISRStack`, the stack used by ISR calls, is wrapped in `configASSERT_DEFINED == 1` macro, it is unable to set `xISRStack` properly without the macro. However, the definition of the macro is missed. We should add the missing macros in `FreeRTOSConfig.h` so the interrupt can be proceed successfully. ```diff diff --git a/Demo/RISC-V_VexRiscv-Briey_GCC/FreeRTOSConfig.h b/Demo/RISC-V_VexRiscv-Briey_GCC/FreeRTOSConfig.h index 8356a3b..b6ad54e 100644 --- a/Demo/RISC-V_VexRiscv-Briey_GCC/FreeRTOSConfig.h +++ b/Demo/RISC-V_VexRiscv-Briey_GCC/FreeRTOSConfig.h @@ -68,6 +68,11 @@ #define configGENERATE_RUN_TIME_STATS 0 #define configUSE_PORT_OPTIMISED_TASK_SELECTION 1 +/* Assert definitions. */ +void vAssertCalled( void ); +#define configASSERT_DEFINED 1 +#define configASSERT( x ) do { if ( !( x ) ) vAssertCalled(); } while(0) + /* Co-routine definitions. */ #define configUSE_CO_ROUTINES 0 #define configMAX_CO_ROUTINE_PRIORITIES ( 2 ) ``` :::success :gift: I also send a [PR (#777)](https://github.com/FreeRTOS/FreeRTOS/pull/777) to FreeRTOS repository. It has been merged! ::: ### Use Briey timer instead of machine timer The RISC-V machine timer is implemented depending on the vendor. Since Briey SoC does not provide its own machine timer, I try to use one of the timers to trigger the increment of system ticks. There are 2 functions we need to implement to handle timer interrupt: - `vPortSetupTimerInterrupt` This function in the imeplementation has marked as weak symbol, we can just provide our own function to overwrite the original one ```cpp void vPortSetupTimerInterrupt(void) { asm volatile("li t0, 0x1808\n\t" "csrw mstatus, t0\n\t" "li t0, 0x880\n\t" "csrw mie, t0\n\t"); interruptCtrl_init(TIMER_INTERRUPT); prescaler_init(TIMER_PRESCALER); timer_init(TIMER_A); TIMER_PRESCALER->LIMIT = 500; TIMER_A->LIMIT = 1000; TIMER_A->CLEARS_TICKS = 0x00010002; TIMER_INTERRUPT->PENDINGS = 0xF; TIMER_INTERRUPT->MASKS = 0x1; } ``` - `handle_trap` The function is used to handle timer interrupts. It has to reset the PENDING bit for the timer and increase the system ticks ```cpp void handle_trap(void) { portENTER_CRITICAL(); xTaskIncrementTick(); TIMER_INTERRUPT->PENDINGS = 1; portEXIT_CRITICAL(); } ``` The intension to wrap the function with `portENTER_CRITICAL()` and `portEXIT_CRITICAL()` is to prevent the interrupts to occur during handling. :::warning :warning: Since Briey SoC is lack of the document, I am not sure how to properly set the system tick according to the config of FreeRTOS. For current stage, I just make it works and let the system generate some periodically interrrupts. ::: ## Result The demo is trying to switch between `receiver` and `sender` thread, the `sender` thread will only wake up when the delay time is 0. During the sleep, since receiver thread does not have any message arrived, it will be queued as well. So the system will keep looping in idle thread. When the system ticks is larger than the sender's wake time, the sender will be waked up and send the message to the receiver. ## References - [SpinalHDL / VexRiscv](https://github.com/SpinalHDL/VexRiscv) - [Briey SoC simulation hang at beginning #216](https://github.com/SpinalHDL/VexRiscv/issues/216) - [SpinalHDL / VexRiscvSocSoftware](https://github.com/SpinalHDL/VexRiscvSocSoftware) - [SpinalHDL / openocd_riscv](https://github.com/SpinalHDL/openocd_riscv) - [Dolu1990 / FreeRTOS-RISCV](https://github.com/Dolu1990/FreeRTOS-RISCV) - [Final Project:Run FreeRTOS on VexRiscv and access the peripherals such as VGA.](https://hackmd.io/@4a740UnwQE6K9pc5tNlJpg/H1olFPOCD) - [FreeRTOS Porting Guide - FreeRTOS](https://www.freertos.org/FreeRTOS-porting-guide.html) - [Port Alpine Linux to open source RISC-V platform 系列 - 2021 iThome 鐵人賽](https://ithelp.ithome.com.tw/users/20140421/ironman/4097) - [RISC-V External Debug Support Version 0.13](https://static.dev.sifive.com/riscv-debug-spec-0.13.0512f5d.pdf) - [OscarShiang / FreeRTOS-Briey](https://github.com/OscarShiang/FreeRTOS-Briey) ###### tags: `arch2021`

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.