---
title: "Platform Rearchitecting, November 21, 2022"
tags: unikraft, platform
datetime: 2022-11-21T12:00:00+02:00
location: Online, Discord (https://bit.ly/UnikraftDiscord), the `#maintainers-voice` voice channel
teams:
- maintainers
participants:
- TODO
---

## :dart: Agenda

- Plat rearchitecting design principles
- Hackathon tasks

## :closed_book: Discussions

Brainstorming / minutes: https://docs.google.com/document/d/1Kz0fWUe7pLl2kIx_CnNotY53h8vvSH8owUAd-ZNtB3I/edit

Tasks: https://github.com/orgs/unikraft/projects/36/views/1

## :wrench: Decisions and TODOs

## :evergreen_tree: Source Code Layout

arch/
    arm/
    x86_64/
drivers/ 
    blkdev/
    boot/
        efi/
        multiboot/
    bus/
       pci/
    console/
           ns16550/
           pl011/
    irq/
        gic/       
    lcpu/
        armv8/
        linuxu/ /* platform-specific variants (eg different TLS implementation, syscall handling) can be added here */
        xen-pv/
        x86_64/
    mmu/
        arm-mmu/
                include/
                       driver_headers.h
                       /* If necessary, it is possible to define headers that
                        * are available to the subsystem. This is useful for
                        * use-cases such as ukpaging where the implementation
                        * header defines static inlines that must be accessible to 
                        * the implementation of the API.
                        *
                        * Convention: include/<parent_lib_name>/
                        */
                       uk/bits/ukpaging/
                                       my_header.h
        x86_64_mmu/
        xen-pv/        /* Other non-native implementations also fit here. */
    netdev/
          vmxnet/
          xennet/
    virtio/ /* we allow deviation from strict structure for better code organization */
          mmio/
          pci/
          net/
             |_ README.md     // library specific documentation (to be automatically collected into unikraft docs)
             |_ tests/        // uktest
             |_ *.c 
             |_ *.ld          // needs to be specified in Makefile.uk like any other source / header
             |_ *.h           // private headers
             |_ include/uk/   // public headers
             |_ exportsyms.uk // optional (see https://unikraft.org/docs/develop/porting/#internals-of-the-unikraft-build-system)
             |_ localsyms.uk  // optional (see https://unikraft.org/docs/develop/porting/#internals-of-the-unikraft-build-system)
             |_ Config.uk
             |_ Makefule.uk
             |_ Makefile.rules
plat/
    kvm/
    linuxu/
    xen/
lib/ /* flat set of services - APIs */
    posix-fdtab/
    posix-socket/
    posix-process/
    ukboot/
    ukpaging/
        include/uk/paging.h
        paging.c
    ukbus/
    uklcpu/
    ukblkdev/
    uknetdev/

**TODO:**
- Verify whether existing subsystems fit proposed layout (all)

### arch/

`arch` can contain:
- Definitions provided by the architecture (registers etc).
- Architecture-specific compiler definitions.
- sizes of integer types (library-independent types: `__u16`, `__s64`, `__sz`), limits, etc.
- ABIs like int types, traps etc (eg arm64/lcpu.h).

The definitions in `arch` are always available to all libraries.

`arch` must NOT contain:
- Architecture-specific implementation. All implementation must be part of a library.

**TODO:**
- "These are always available to all libraries" are they already?
- Standardizing the ABIs (naming conventions, prefixes, document, ...)
- Decide on where do the APIs that move out of arch/ go to.

### drivers/

Hardware-centric low-level implementation. We consider all libraries under drivers to
implement some type of device, even if that is a cpu native device such as an mmu.

These usually implement part of a higher-level API defined under `lib`. It may be
possible to define also drivers that do not belong to an API defined under `lib`.
This is okay as we will be allowing direct access to drivers from the application.

## include/

All headers under `include/uk` must not depend on any libraries (except headers that are always available, arch, or libraries that provide "basic headers"). Because of that we don't also use libc types, instead use architecture-provided types (`__u64`, ...).

### Config.uk

1) Standarized feature flags (KConfig) - populated by selected Platform's Config.uk and/or drivers? Yes, but on a high level abstraction for items that don't depend on a specific API implementation of that feature, e.g., `HAVE_SMP` so that we know that multiple cores could be initialized and spinlocks might be necessary. COunter-example where this makes less sense is `HAVE_PCI` because interacting with PCI bus requires to talk with a specific API and this would be provided by a driver. So here you rather depend on `DRIVER_PCI_LEGACY`. Then we have one general exception with `HAVE_LIBC` which makes possible to replace libc easily - but APIs are assumed to be the same.

/* probably standarized driver APIs */
HAVE_DMA
HAVE_PAGING
HAVE_NATIVE
HAVE_XENPV
HAVE_PCIBUS

/* Today already: */
HAVE_LIBC
HAVE_SCHED

2) Driver's Config.uk
Dependency on features

menuconfig MYDRIVER
depends on !HAVE_X86 && HAVE_PAGING
depends on DRIVERCPUNATIVE || DRIVERXENCPU
depends on DRIVERPCI
select HAVE_DMA /* announce this driver implements this feature */

3) Platform's Config.uk
CONFIG PLATQEMU
/* some basic drivers have to be hard-selected by platform also kick-start */
select DRIVER_CPU_NATIVE /* required*/
select DRIVER_BOOT_MULTIBOOT1 /* image format definition */
imply DRIVER_APIC_NATIVE if CONFIG_CPUS > 1
imply DRIVER_PAGING_NATIVE /* Always paging? --> depends on CPU_NATIVE */
imply DRIVERPCI
imply DRIVERVIRTIO /* good default to have but can still be unselected */

CONFIG PLATLINUXU
select DRIVER_CPU_LINUXU
select DRIVER_BOOT_ELF
imply DRIVER_TAPBUS
imply DRIVER_TAPNET

CONFIG PLATCOQOS
select DRIVERCPUCOQOS
imply DRIVERMMIOBUS

/* choose default set of drivers */

XX) Probably we have to differentiate between two types of drivers:

1) Basic system drivers (where you can select only implementation, never multiple) - e.g., CPU, PAGING
--> these might be the ones that need to be hard selected
as starting pooint by a platform library
--> API definition (Marc's idea) mostly consists only of headers with #if MACRO, #ifnot MACRO then function prototype -- include a standarized uk/bits/ header path. This bits header path is provided by the selected basic system driver.

2) General drivers, that depend on specific basic drivers to be there (e.g., CPUXEN) and optionally other drivers (e.g., a bus driver like XENBUS). Example are network card drivers, disk drivers, ... -> these would be just be "imply" by platform

Decision:
- Define libraries for basic APIs like CPU etc (Simon's Basic System Drivers)
Deprecates HAVE_FEATURE options (HAVE_ name would have been just a duplication of the API library name).
Reasoning:
- Makes sure that implementations do not diverge.
- Fits well everything-is-a-library model.
- Allows defining overriding (alternative) APIs in external libraries that can be
selected by implementing drivers.

New Tasks:
- Identify and define missing library APIs
- Update existing code to replace HAVE_FEATURE with library config option.

Basic drivers are only selected by the platform

SK: Still open question: things like HAVE_LIBC?
MR: Next open question: How linker scripts are built

### Access to platform-specific/driver headers from apps

**Decision:** Should be supported.

**TODO:**
- Continue the discussion on how to do that.

### Support multiple platforms in a single build

**Decision:** Deprecate.

### Macro definitions that are required even if a feature is not enabled (eg UK_ASSERT)

Introduce the concept of "basic macro definitions", ie definitions that are always available
in a build (`UK_ASSERT`, `uk_pr_debug()`), so that calling code won't need to wrap them around
ifdefs.

Make all headers from all libraries available to the build, regardless of whether the library
is enabled. Use guards in headers to protect definitions if the library is not enabled, or for
empty declarations like `#define UK_ASSERT`.

Example: uk/assert.h

#ifdef CONFIG_UKDEBUG
#define UK_ASSERT(condition) do_assert...
#else
#define UK_ASSERT(condition) do {} while(0)
#endif

In the Makefile change:

CINCLUDES-$(CONFIG_UK_DEBUG) := assert.h

to

CINCLUDES-y := assert.h

:warning: External libraries are allowed to provide such definitions, yet no core unikraft code
must depend on external libraries.
:warning: Definitions that are always made available (independent if library is included or not) must use basic unikraft types provided with `/arch` --> otherwise we always force a code dependency to another library (e.g., a particular libc)

## Platforms / Bootcode / Linker scripts / Image creation

Boot order:
1. boot-entry driver (U-Boot, multiboot, uefi, pcbios, mbr, ...)
2. CPU init (utilize functions provided by lcpu driver, set up runtime etc)
3. Jump back to boot-entry to prepare bootinfo structure (as defined API by ukboot)
4. Jump to generic entry point (ukboot)
5. Callback table before ctortab to do further driver initialization
6. ukboot will continue then boot procedure with ukinit (previously called ukboot). The split so that the highlevel init (scheduling, networking, application launching) can be replaced by custom ukinit implementations. ukboot is really about low-level init of hardware (irq/events/...)

- platforms consist of a single Config.uk
- We don't use defconfig as main driving because:
1. We would need to combine defconfig with app config
2. An internal Config.uk provides us with more flexibility on supported features + visual menu and dependency building - we can also limit valid options for configuraiton.
3. However, defconfigs can be still provided to pre-set some spcific platform flavors, e.g., qemu-efi, qemu-multiboot, qemu-microvm, qemu-q35.
- Move logic into drivers/ukboot that takes care of:
- Boot-entry drivers
- Default linker script
- That should only contain the smallest number of sections
- goal is to split linker script definitons as much as possible and to the library actual needing and using them so that ukcontext introduces TLS sections, ukdebug introduces debug sections, ukboot introduces ctortab and inittab, ...
- These sections must be defined along with the required order (insert-after)
-
- Default Linker.uk
- Create lcpu library with:
- Existing lcpu functionality
- Arch-specific cpu init code

lib/ukboot/
          boot.c         /* low level boot tasks (irq, events, ...) */
          Makefile.rules /* additional linker rules for image generation to avoid duplication */
          link64.h       /* header with common sections for linker scripts
                          * (minimal set of sections - libraries provide their own additional headers)
                          */
          bootinfo
          memory.c
drivers/ukboot/          /* boot-entry drivers, entry point */
              mbr/           
                  Linker.uk /* Can use rules provided by ukboot/Makefile.rules along with custom logic */
                  link64.ld.S /* Uses sections from ukboot/link64.h along with custom sections */
              multiboot/
              pcbios/
              uefi/
              u-boot/
              baremetal/ /* not applicable for x86 (doesn't make sense) - it is basically a low-level SoC format that applies to (almost) all (Arm) SoCs */
                       arm64/
              ...
lib/lcpu/                /* existing lcpu functionality + arch-specific bootstrapping */
        arm64/
             lcpu.c        /* arch-specific implementation of lcpu APIs */
             lcpu_start.S  /* arch-specific bootstrapping */
        x86_64/
              ...
        lcpu.c           /* enumeration, initialization, management, smp, ... */
        lcpu.h           /* lcpu API */

lib/ukinit/              /* high level boot: everythinig else (scheduling, networking, jump to application etc) */
          ...

External platform libraries then can define their own config.
Extra care must be taken to avoid changes in boot common lib breaking external platform libraries. Keep boot common minimal!

## Fine-graned platform configuraiton
Example: Allow disabling virtio, but if enabled make sure you use v2.

implies DRIVER_VIRTIO
select DRIVER_VIRTIO_V2

TODO: Investigate whether Kconfig syntax allows setting specific integer values, strings etc

## essentials.h

Split into:
1. `include/uk/compiler.h`
2. `include/uk/essentials.h`

Namespace macros (`MIN`, `MAX` etc clash with libc). Use `UK_MIN`, ...
`container_of` moves into `compiler.h`

## variable types

TODO:

GCC defines types nativel (doc?). We can use these instead of defining ours. Include these types in architecture headers, if not possible define (in the architecture) types manually.

See: https://patchwork.kernel.org/project/linux-arm-kernel/patch/[e-mail]/

## Platform-specific topics

Platforms (eg Xen) can provide definitions (like hypcall IDs) that are required for platform-specific drivers.
Define these under the platform, and have the platform-specific drivers under `drivers/`. Simlarly for Hyper-V.

## VMM Naming

Rename KVM to QEMU

plat/qemu/
         system-aarch64/
                        virt/
                             Config.uk

## Replace internal libraries with external ones

Probably a new make variable, like `R=<name of lib to replace>:<path-to-replacement-lib>`
Ideally via `Makefile.uk` `addlib/importlib` command but how can we deal with set variables and rules from `Makefile.uk`? Maybe `importlib` could parse first instead of include?

2 stages:
1) Build list of libraries and handle replacements
2) Actually include the libraries from the list (`Makefile.rules`, `Makefile.uk`, ...)

## :hourglass: Pending Topics

MR:

SK:

MP: