# Integrating Linux and the real-time ERIKA OS through the Xen hypervisor
[Integrating Linux and the real-time ERIKA OS through the Xen hypervisor](https://algogroup.unimore.it/people/paolo/papers/Linux-Erika-Xen.pdf)
## Abstract
A dual-OS set-up is described. They introduce ERIKA Enterprise as a *a small-footprint real-time OS suitable for safety-critical control tasks and able to execute commands triggered by Linux.*
## Introduction
They explain their choice of RTOS (ERIKA Enterprise). They have assigned a CPU core to each machine, in order to allow the two OS's to boot in parallel. The introductory work does not seem to use virtualization. Instead the hypervisor is introduced in this project, and thereby virtualization. They say they introduce Xen in order to add an extra abstraction layer, to isolate the two OS's and their memory spaces.
## Related work
The Linux machine runs as dom0 and ERIKA as a domU. They have ported the hypercall `XEN_DOMCTL_memory_mapping` to ARM, in order to get access to I/O-memory areas. They utilize CPU pinning, which assigns each machine a specific CPU core.
## Implementation of inter-domain communication
They list requirements for their project. Regarding priority they say that "notification triggered from the general-purpose operating system must not preempt any high-priority task running on the real-time operating system. The real-time operating system must be able to define priorities and determine which of the tasks running in its container are at a higher priority than interrupts."
### ERIKA Enerprise communication driver
ERIKA requests an unbound event channel from Xen. This is done by using a hypercall. Then it requests the start physical address of the memory region intended for the communication. ERIKA tells Xen to set the memory pages as mappable by dom0. It the exploits the XenStore to pass references to both the event channel and the memory area, to the dom0.
### Linux communication driver
The GP-Linux can for example toggle GPIO-pins. The communication channel needs to be set up correctly in order for ERIKA to utilize the physical peripherals. In order not to interfere with the RTOS, each transactio to ERIKA is seen as a self-contained transaction, and the event channel is only bound for the minimum indeispensable time to perform the needed operation.
### Command delivery and handling
When sending an event between the two domains, the sender populates the shared memory with data neccessary for the operation, and then invokes the appropriate hypercall.
While the shared memory space is masked by the RTOS (used by the RTOS), signals will not be sent to it, in order to allow higher priority to the RTOS.
## Evaluation of the results and future work
This section discusses the functional aspects of the proof-of-concept design.
### Evaluation of the communication mechanism
In the setup-routine, two hypercalls are issued, allowing both domains to access the same memory pages. This eliminates the need to utilize the hypervisor to communicate, other than to signal that a nee command is present. This might be considered a performance bottleneck. They seem to mean that having to use the hypervisor for all communication signalling might cause the machine to be stuck in hypervisor mode for too long and thereby cause misses of critical deadlines. **This is something for us to look out for in our implementation**.
### Evaluation of the design
Xen provides isolation. CPU-pinning reduces temporal interference. Further **regarding boot time**, ERIKA runs as an unprivileged domain and must therefore wait for dom0 to fully boot until it can boot. It does not even guarantee that the RTOS boots at all, as a malfunction in dom0-boot would prevent ERIKA from booting. As of the current state of the project (2015), Xen isn't certified for real-time applications. They speak of different standards that could be easily certified.
### Future work
Having an RTOS run as dom0 and GP-Linux as domU may reduce boot time. Having a microkernel run as dom0 and the RTOS and GP-Linux run as domU's might also be a possibility.
### Porting the solution to other RTOS's
Porting small RTOS's should be fully possible.
## Conclusions
The results of this work is available as open-source. It has generated high interest in the Xen development community. The description is available on the ERIKA Enterprise wiki.
## References
*Xen and the art of virtualization*, pages 164-177 are referenced.
*Architecture for a portable Open Source Real Time Kernel Environment* might be a good read: P. Gai, E. Bini, G. Lipari, M. Di Natale, L. Abeni
[*A fully Open-Source platform for automotive systems*](http://www.evidence.eu.com/embedded-linux-osekvdx-erika-enterprise-dual-core-automotive-cpu-without-hypervisor.html) by P. Gai, C. Scordino and B. Morelli might be relevant.
*RT-Xen: Real-Time Virtualization based on Xen* is published by [The Xen Project](https://sites.google.com/site/realtimexen/).
[*Xen hypervisor porting*](http://erika.tuxfamily.org/wiki/index.php?title=Xen_Hypervisor), by P. Gai, C. Scordino, B. Morelli, P. Valente, A. Avanzini.
---
# Xen hypervisor porting
This is a summary of a [source](http://erika.tuxfamily.org/wiki/index.php?title=Xen_Hypervisor) referenced in the article above. It is from ERIKA's wiki, and thereby **not** a published source, but rather an online source.
## Introduction
ERIKA can be run as a guest OS inside a Xen domU.
## Quick Guide
They provide a [free Virtual Machine](http://www.erika-enterprise.com/#VM).
For this mapping they have used the following config file:
kernel = "A20.bin"
memory = 512
name = "ERIKA_Enterprise"
vcpus = 1
on_crash = 'destroy'
iomem = ["1c20,2"]
To start ERIKA from inside dom0 they use `xenstore-chmod /local b` followed by the usual `xl create ./erika.cfg`.
The implementation allowes for a user to set GPIO-pins via commands.
## Requirements for working environment
The regular requirements listed. "Make sure the platform is supported by Xen" etc.
### Firmware Requirements
The hypervisor boot must happen in Hypervisor mode (EL2).
### dom0 requirements
A device tree blob must exist. A new node in the DTB is named **`chosen`** and contains
* the image location and a classification per compatibility
* additional parameters to be passed to the hypervisor or to the dom0 during boot.
This is the proposed **chosen** node:
chosen {
bootargs = "$xen_cmdline";
modules {
module@0 { /* dom0 kernel */
bootargs = "$dom0_cmdline";
compatible = "xen,linuxzimage", "xen,multibootmodule";
reg = <$kernel_addr $kernel_size>
};
module@1 { /* initramfs */
compatible = "xen,linuxinitrd", "xen,multibootmodule";
reg = <$initramfs_addr $initramfs_size>;
};
};
};
### domU requirements
**domU executable format:**
*This is for ARMv7, that might explain why the header is different*
Xen uses the zImage format to load an ARM kernel (uncompressed). The header must have an offset 0x24 inside the zImage and must contain:
1. a magic number (0x016f2818) that identifies it as a zImage,
2. the starting address of the zImage, and
3. The end address of the zImage.
Here is a given example:
_start:
@ zImage header
.rept 8
mov r0, r0
.endr
b _start_cubieboard2
.word 0x016f2818 @ Magic numbers to help the loader
.word _start_0 @ absolute load/run zImage address
.word _end _
start @ zImage size
@ end of zImage header
_start_cubieboard2:
**Base address for the executable image of the domU**
> The base address for the address space reserved to a domU is 0x80000000. The [xen/include/public/archarm.h:372 linker script] used for the compilation of the ERIKA source code must be adapted to that value.
**Privileged Intructions**
A non-privileged domain cannot access the System Control Register to enable the MMU. A domU exception is raised.
> **IMPORTANT: Commit to be reverted**
>
> If you compile a Xen Repository including commit 878ff4fe1816d7f808f11254b555b9e9c2f121fa, this will provoke a critical shutdown of the ERIKA Enterprise domU at boot time. We did not yet understand why this happens. the only solution is currently to revert the commit.
## Creating a working system
This section describes installing a toolchain, and creating a Linux dom0 together with an ERIKA Enterprise domU.
### Preparing the toolchain
Install a toolchain. In this case it is suggested to use the `crossdev` application to generate a toolchain.
### Compiling and installing the bootloader
Instructions to download and compile U-Boot. They say the booloader should be stored at an offset of 40KB, and the sumbol table at an offset of 8KB. I'm guessing this goes for Xen, and not ERIKA. It doesn't make sense to install a bootloader in a domU.
### Creating the root file system
All this is to create a dom0.
### Compiling and installing the dom0 kernel
This might be interesting, but not really relevant.
### Validating the setup with only the Linux Kernel
It is possible to test that everything works by starting the bootloader separately.
### Compiling and installing the hypervisor
Download the latest version of Xen. Enable diagnostic messages by modifying the `Config.mk` file, setting the option `debug ?= y`.
To enable ERIKA domU's access to the GPIO driver, a patchset is applied that adds the hypercall `XEN_DOMCTL_memory_mapping`. That hypercall provides the complete access to an I/O memory region to a non-privileged domain. The patchset is available in the [mailing list archives](http://markmail.org/thread/wtevxxizyxjhvmg2).
### Preparing the boot scripts
U-Boot needs to know the correct locations for the hypervisor and the dom0. The **chosen** node in the device tree needs to be added as well. They list a boot script in this section.
**How to exclusively assign a core to the dom0**
In order to allow for CPU-pinning, a configuration option is needed. It is added to the command line used by U-Boot to boot Xen.
### Validating the setup of Xen + Linux dom0
Connect to the serial line of the target board. When you reach the login prompt of the Linux dom0, press "Ctrl+AAA" to switch from the Linux console to the Xen console. Repeat "Ctrl+AAA" to go back to the Linux console.
### Compiling the toolstack
It should be possible to start the system daemons needed to interface with Xen by running
# /etc/init.d/xencommons start
but this is not present in the ACU6 implementation. `xl` still obviously works.
### Creating the configuration file for the ERIKA Enterprise domU
To start a minimal version of ERIKA they use the config file
kernel = "myProject_master.bin"
memory = 512
name = "ERIKA_Enterprise"
vcpus = 1
on_crash = 'destroy'
and then starts the VM with
# xl create erika.cfg
and stops it with
# xl destroy ERIKA_Enterprise
**Changes to the configuration for GPIO and XenBus drivers**
Here they add a line to the config file in order to access the shared memory pages used for the event channel.
iomem = ["1c20,2"]
This gives the domU access to two memory pages starting from page numer 0x1c20 (which corresponds to address 0x1c20800). This is the start of the I/O memory as specified by [sunxi](http://dl.linuxsunxi.org/A20/).
If the domU has been compiled with the XenBus driver, the correct read/write permissions needs to be assigned to the space of the names in the XenStore hierarchy:
# xenstorechmod -r "/local" b
**Statically assigning a core to the domU**
Assign physical cpus to a domain by adding squared brackets around the number of cpus in the config file. It might also be possible the use quotes around the number. See this section for mor info. To get information about the CPU usage of the system use `xl vcpu-list`.
## An example ERIKA application
The ERIKA Enterprise port as a domU Xen-on-ARM includes a set of subsystems which can be used to interact with the hypervisor. As a reference for the ERIKA port, another [porting project, of MiniOS ARM](https://github.com/KarimAllah/xen/tree/miniosarmport) was used. There is a figure in this sections which depicts the architecture of the application. ***Look up what a grant table is***.
### GPIO Driver
This is a simple port of the main functionalities of the [Linux port driver implemented by sunxi in Linux](http://linuxsunxi.org/GPIO).
### Handling the GIC and the event channels
The Xen implementation handles bith interrupts and the Xen notification distribution mechanism (or [event channels](http://wiki.xen.org/wiki/Event_Channel_Internals)).
A modified GIC driver was implemented to handle the interrupt dirstribution in a better way.
There is a lot to unpack in this section. I need to come back to it.
### Communication with the XenBus
The same goes for this section. There is a figure which illustrates the use of a shared ring buffer to pass messages. ***Look into this***.
### Shared Memory Handling
A grant table is a Xen data structure available for every domain, containing info about shared pages and its R/W-permissions. This allows for an unprivileged domain to export pages of its own address space, for other domains to access in a safe way.
dom0 can access the shared page(s) using a hypercall. Some configurations are done, and after that the two domains can share memory without having the hypervisor intervene.
### Synchronization between domains
Xen does not provide a mechanism to synchronize the access to the shared memory area between the two domains. This is therefore implemented, with possibility for mutual exclusion for the memory space. The implementation is intended to use when Linux needs to ask for soma action from the ERIKA domain.
1. The Linux dom0 writes the action to be performed in the Erika domU shared memory.
2. dom0 sends a notification to domU through the event channel.
3. The domU handles the interrupt and executes the command. It signals back to dom0 when done.
4. The dom0 handles the interrupt and gets the response from domU.
This mechanism can be paired with a mutex mechanism based on shared spin lock, but it not recommended since it causes spin locks (duh).
---
# MiniOS-Arm port
This is a GitHub [repository](https://github.com/KarimAllah/xen) which is references in the ERIKA porting. The text here is a summary of the README file of the repo.
## What is Xen?
It's a VMM - nothing new.
## Quick-Start Guide
There are some prerequisites listed, in order to build a Xen source release. You need a dom0 kernel.
## Python Runtime Libraries
Install some stuff to compile and run Xen.
## Intel(R) Trusted Execution Technology Support
Some stuff about trusted boot (`tboot`).
## From browsing the files
in the file `xen/xen/arch/arm/arm64/head.S` we are back to the header we are used to, as opposed to the one proposed by the ERIKA port. As a comment it says that the image header is as expected by Linux boot-loaders.
Sign in with Wallet
Connect another wallet