HackMDRun FreeRTOS on VexRiscv and Measure/Tweak context switch overhead
tags: Computer Architecture
Github : FreeRTOS-on-VexRiscv
Requirement and Expectation
- Reproduce Run FreeRTOS and multitasking on VexRiscv with FreeRTOS 202212.00 released or latest
- Study Reference Link to understand how to measure FreeRTOS context switch
- Understand how to accurately measure cycle count and related latency
- Include timer interrupt.
- Run above at simulator and quantify the performence about context switch
- Study Reducing Context Switching Overhead by Processor Architecture Modification and introduce the cost of reduction about context switch at the instruction level mechanism.
Reproduce past work
- Step-by-step detail in Note
Rebuid and Study Following two projects
- Final Project: VexRiscv
- Final Project:Run FreeRTOS on VexRiscv and access the peripherals such as VGA.
Use 3 terminal to show
- Upper right handle Briey SoC
- Upper left handle GDB server connect to the target
- Lower left handle OpenOCD connect to Briey SoC
Take VGA project in VexRiscvSocSoftware for example.
Run
and type following command in GDB
And see the RGB square in VGA GUI interface.
There are other projects for Briey in VexRiscvSocSoftware
- dhrystone
- timer
- uart
- vga
Rebuid and Study Run FreeRTOS and multitasking on VexRiscv
- This project choose to port Briey SoC to run FreeRTOS
- Installation and setup also follow Final Project:Run FreeRTOS on VexRiscv and access the peripherals such as VGA.
From the description
/sourcecontains the FreeRTOS source code/Democontains a demo application for every official FreeRTOS port./Testcontains the tests performed on common code and the portable layer code.
Use RISC-V_RV32_QEMU_VIRT_GCC as a templete and modify it
The virt board is a platform which does not correspond to any real hardware; it is designed for use in virtual machines. It is the recommended board type if you simply want to run a guest such as Linux and do not care about reproducing the idiosyncrasies and limitations of a particular bit of real-world hardware.
Memory layout
We can find the memory capacity definition in Briey.scala
- There are two storage space
- ram (onChipRam) with base address
0x80000000andonChipRamSize(4kB) - sdram with base address
0x40000000andsdramLayout.capaciry
- ram (onChipRam) with base address
That's why the linker script (<VexRiscvSOcSoftware>/projects/briey/libs/linker.ld) in VexRiscvSocSoftware
mention that
shows that
- Init assembly and stack place on-chip-ram
0x80000000withLENGTH = 4K - Text and data section place on SDRAM
0x40000000withLENGTH = 64M
Modify and rename fake_rom.lds -> linker.ld
Set uart libarary
Copy the header file from VexRiscvSocSoftware/libs (gpio.h, interrupt.h, prescaler.h, timer.h, uart.h, vga.h) to RISC-V_VexRiscv-Briey_GCC/Vex_libs and integrate briey.h(include several base address macro) in gpio.h
Rename riscv-virt to riscv-briey and modify uart port.
And modify associated part.
Implement Briey timer and intergrate uart and timer
Reference Use Briey timer instead of machine timer
main.c
- Reference uart project.
Implement vPortSetupTimerInterrupt() which is marked as weak in Source/portable/GCC/RISC-V/port.c
- The frequency of the interrupt must equal the value of
configTICK_RATE_HZ(which is defined inFreeRTOSConfig.h)
Implement handle_trap(), which is used to handle timer interrupts. It has to reset the PENDING bit for the timer and increase the system ticks
riscv-briey.c
- Notice that the
handle_trap()function is mentioned by-DportasmHANDLE_INTERRUPT=handle_trapin Makefile, and this is assembler macro, not compiler macro.
mstatus
0x1808set MPP[1:0] = 11, which is stand for machine mode, and also set MIE = 1 (enable)
mie
0x880set MEIE and MTIE to 1, which means interrupt enable for machine level external and machine timer interrupts.- Reference Privileged spec
Incompatible problem
- And I find a familiar issue, and follow the discussion and figure it out.
- This problem is already fixed in #838
Memory not fit problem
- Most of the time, VMA and LMA will be the same. But in embedded system, this situation might change.
- Seems that
onChipRam(4K) does not have enough capacity to store our code (text section). - Put the
.textand.datasection to sdram (64M).
Relocation truncated
can solve this problem
Debugging
Copy <FreeRTOS repo>/FreeRTOS/Demo/Common into <FreeRTOS-Briey>/Demo which is not provided in repo.
"Full" vs "Minimal" demo application files
FreeRTOS/Demo/Common/Fulldirectory assume a hosted environment and are only used by demos that run on top of old DOS systems(which is also why the Partest.c filename is cryptic - it could only use short filenames in 8.3 format)FreeRTOS/Demo/Common/Minimaldirectory, none of which assume a hosted environment.
Under /FreeRTOS
Compressed instruction stuck problem
I suffer from some unknown problem and the gdb will stuck and can not enter in main function.
Notice that the project is compile with -march=rv32imac, the compressed instrucion cause nop instruction into 2 bytes (in previous work set 7 nop instruction to push 0x80000020), and I think there is another problem (no figure it out yet, but I think single step(s) in gdb will let pc+4 at each step) that cause gdb stuck.
Also, li instruction is also compressed to 2 bytes. But in riscv64-unknown-elf-gdb, it can't load a1 to 0 properly.
with gdb
And casue the next instruction bne jump to label secondary, which is handle for multicore, and it shouldn't be jump in my case.
So, I decide to build the project without compressed instruction.
Check multi-lib
- And the gdb will run properly
But this method just abort the compressed instruction.
Configure BrieySoc to fit the compressed instruction (RVC)
There are some discusses about error and incompatible feature and some project with RVC compatible implementation.
Some similar project
And follow the cofiguration above, modify IBusCachedPlugin
- Rebuid Briey SoC and It can run properly with compressed instruction without modifying
Makefile
Furthermore, remember that executing ecall will trigger a SWI and jump to the address of mtvec, which is 0x80000020. And that's why Oscar add serveral nop instructions to push the main_entry to 0x80000020.
But in RVC, nop was compressed to 2bytes, that means I need more nop instructions (2 times) to do the same thing.
Linker script lma problem
There is a function in portASM.S to trap exception and view the decription.
run(gdb)info register to check the mcause value
macuse= 0x5, stand for Load access faultmepc= 0x80000074, stand for80000074: 00052283 lw t0,0(a0)mstatus= 0x1800
Check the disassembly
In assembly code is la a0, _data_lma, but in disassembly, is link to _bss_lma.
Put _data_lma lma same to vma.
Now, it can successfully enter in main function. But trap to freertos_risc_v_application_exception_handler again.
- Problem here
- Update gdb by reinstalling riscv-gnu-toolchain
- Now can successfully show the string.
The bug is cause by setting CFLAGS with -march=rv32ima and LDFLAGS with -march=rv32imac and both with -mabi=ilp32
CFLAGSis used to compile codeLDFLAGSis used to link with linker script
It fixed when I used modified BrieySoc (with compressed instruction).
Illegal instruction problem
t0= 2 stand for illegal instruction, reference previous work
Change Briey.scala setting
And it can succefully enter the task.
Machine timer interrupt
But, still trap in exception
- Which means Machine timer interrupt
But, if I use the GCC/RISC-V/portASM.S in FreeRTOS Kernel V10.4.6, It can successfully run context switch.
In previous version, portASM.S use portasmHANDLE_INTERRUPT for handling external interrupts. But in the latest version (FreeRTOS Kernel V10.5.1), the code was re-factoring. And portasmHANDLE_INTERRUPT disappear.
- Detail in #444
Using FreeRTOS on RISC-V Microcontrollers mention that to build FreeRTOS for a RISC-V core need:
- Include the core FreeRTOS source files and the FreeRTOS RISC-V port layer source files in your project.
- Ensure the assembler's include path includes the path to the header file that describes any chip specific implementation details.
- Define either a constant in FreeRTOSConfig.h or a linker variable to specify the memory to use as the interrupt stack.
- Define configMTIME_BASE_ADDRESS and configMTIMECMP_BASE_ADDRESS in FreeRTOSConfig.h.
- For the assembler, #define portasmHANDLE_INTERRUPT to the name of the function provided by your chip or tools vendor for handling external interrupts.
- Install the FreeRTOS trap handler.
So, I change the file to use external interrupts.
portASM.S
For some unknown problem, the results doesn't show as I think. The result should be show as below
which is shown in previous work
Noticed that in previous work, configUSE_PREEMPTION is set to 0, which means cooperative scheduling. So I just add portYIELD(); in the end of the sender and receiver task.
And the context switch will run properly as I think.
Cycle count and Latency in VexRiscv
In VexRiscv.scala
Define a 5-stage pipeline (Fetch, Decode, Execute, Memory, WriteBack)
and Fetch stage has with RVC or not option
which is utilized in Services.scala
And there is serveral regression test that give clock cycle
In src/test/cpp/regression/main.cpp, we can find how regression test work.
There is a ready-valid handshake protocol implemented
Notice that instanceCycles was enclosed by two for loop preCycle() and postCycle()
And these two function definition is depended on IBUS configuration. Take IBUS_SIMPLE as example
It will call every simElement->preCycle() and check if iBus_cmd_valid and iBus_cmd_ready all both in high to make sure data transfer correctely.
For example
-
Slow writer(sender) and fast reader(receiver)
-
Fast writer(sender) and slow reader(receiver)
Under /regression, use the command
by giving TRACE=yes, it will generate .fst file and we can observe it by GTKWave.
Take rv32ui-p-xor.fst for example, which is one of the regression test.
- When
validandreadyare both 1, thepcwill plus 4 at the next clock rising edge. Make sure the PC increment with instructions transfer properly.
And the check() function
- check the regfile it writable (valid bit = 1) and associated payload address is not 0.
- and check user
assertEq, which will throw an exception if two input is not equalpayload_addressandregFileWriteRefIndexpayload_dataandWriteRef
- Which also can observe in GTKWave
If above two function pass, instanceCycles += 1;
And the postCycle() function I think it is related to GDB. (no figure out yet.)
Timer interrupt
mtime will increment and if mTime greater than mTimeCmp, it will generate timerInterrupt
- Still not figure out how timer interrupt behavior in cycle count.
Context Switch performance
From Paper and Reference Link, construct a test method
| Paper | Briey Soc | |
|---|---|---|
| C | Cache size | 4096 byte |
| N | Array size | (process size) x (process number) |
| s | stride | (process size) |
| b | line size | 32 byte |
| a | associativity | 1 way |
And associated definition can find in Briey.scala
And the paper also mentioned several regimes with different size setting to measure time per iteration with or without cache miss
| Regime | Size of Array | Stride | Frequency of Misses | Time per Iteration |
|---|---|---|---|---|
| 1 | \(1 \leq N \leq C\) | \(1 \leq s \leq N/2\) | no misses | \(T_{no-miss}\) |
| 2.a | \(C \lt N\) | \(1 \leq s \lt N/2\) | one miss every \(b/s\) elements | \(T_{no-miss} + Ds/b\) |
| 2.b | \(C \lt N\) | \(b \leq s \lt N/a\) | one miss every element | \(T_{no-miss} + D\) |
| 2.c | \(C \lt N\) | \(N/a \leq s \leq N/2\) | no misses | \(T_{no-miss}\) |
- No misses are generated when \(N \leq C\). When \(N \gt C\), the rate of misses is determined by the stride between consecutive elements. D is the delay penalty
In reference, first simply use an array to test the influence about cache miss.
In Briey, we have 4096 byte cache size, by setting configMINIMAL_STACK_SIZE = 128 and use a function to create number of tasks.
And in the task handler, use xTaskGetTickCount to find the current tick count.
Try to use etime - stime to measure the context switch latency, but it seems not work (stime and etime are the same value at the most time)
Implementation is still working…
Reducing Context Switching Overhead by Processor Architecture Modification
The process of context switching involves storing the context (state) of the current executing task in to the stack and restoring the context of the task to be executed from the stack, the context includes
- Program status word
- Program counter
- Stack pointer
- temporary register values
- data related to the next task
And the overhead can be reduced by migrating kernel services such as scheduling, time tick processing, and interrupt handling to hardware.
This paper is aim to reduce the effect of context switch overhead. One of the method is specializing certain register to a thread, which can eliminate the need for saving and restoring of context, but on the contrary it will reduce the number of registers available for other threads.
This paper provides a mothod that reduce the overhead by restricting the use of memory during context-switching by adding register file to the procerssor. This makes the process to compute at much faster rate therby reducin the overhead.
The implementation involves
- Hardware: Modify processor architecture and hardware implementation of instructions (
scxt, rcxt) - Software: Modify GNU MIPS assembler for adding two new instructions.
Hardware Modification
First, modify orginal Plasma MIPS design by implementing all the "reg_bank" register in FPGA's logic blocks. In order to save context registers on a register file in one CPU clock cycle.
Furthermore, 4 additional register files are implemented with each register file holds 12 context registers.
- 9 saved or temporary registers
- frame pointer register
- global pointer register
- stack pointer register
Modified MIPS Architecture (the red part is modified part)
Next, implementint two context-switching instructions (scxt, rcxt) to access these register files for storing and restoring the context durin context-switching operation.
Software Modification
First, developing a co-operative operating system involing basic function for
- Initializing the OS
- Creating tasks
- Scheduling the tasks
Next, modify GCC compiler to aware of the newly added instruction, which inturn used to compile the MIPS C files. The instruction are specified in GNU binutils, and the file mips-opc.c in folder contains all the instructions supported by the MIPS processor. Like this.
makes GCC compiler compatible to modified architecture.
Applications & Results (in paper)
First application contain four tasks created using createTask(), the firset task deals with incrementing variables followed by adding them. Storing the result in sum variable and finally diplaying number of clock cycles.
Second application comprises of four tasks, two of them are structured to undergo fast context switchin using internal register files and the other two tasks are structured to undergo context switching using external RAM
Conclusion (in paper)
Restricting the process of context-switching to the processir itself by modifing CPU architecture, without having the external memory access the context of tasks, rules out the extra time consumption.
Reference
- 2020 Final Project: VexRiscv
- 2021 Run FreeRTOS and multitasking on VexRiscv
- Final Project:Run FreeRTOS on VexRiscv and access the peripherals such as VGA.
- ‘virt’ Generic Virtual Platform (virt)
- [FreeRTOS] Using FreeRTOS on RISC-V Microcontrollers
- Using FreeRTOS on RISC-V Microcontrollers
- An Introduction to the RISC-V Architecture
- 3.10 Options for Debugging Your Program
- Linker Script VMA and LMA example
- The linker’s warnings about executable stacks and segments
- spinal
- CPU Time
- Understanding Hardware Counter CPU Cycles Profiling Metrics
- Understanding Cache Contention and Cache Profiling Metrics
- mhz: Anatomy of a micro-benchmark
- NCKU Wiki
- Measuring cache and TLB performance and their effect on benchmark runtimes
- GDBWave - A Post-Simulation Waveform-Based RISC-V GDB Debugging Server
- FreeRTOS Study
