# GUI Systems on Linux
###### tags: `GNU` `Linux` `GUI` `Operating System` `Kernel` `Computer Graphics` `ABI` `API` `App` `Protocol` `Software` `Hardware` `Cross Platform` `System Design`
A deep dive into GNU/Linux "microkernel-style" (as the GUI infrastructures implemented outside the kernel) GUI system architectures, from high- to low-level.
## Intro
GUI systems at low-level:
- [The BASICS of Adding a Display to Your Arduino, ESP32, STM32, or other MCU Project - Zach of All Trades](https://www.youtube.com/watch?v=j-7pDFmdbc0): devices, drivers, screens, bus interfaces, framebuffers, graphics libraries, glyphs, inputs, event handling, ...
## Linux Desktop Environment
Provides desktop environments & common graphical interfaces, such as `File Manager`, `Windowing System`, `Rendering Graphics Element`, `Input Event`, etc.
More:
- [Free-Software Desktop Environments for the X Window System (X11) and Wayland - freedesktop.org](https://en.wikipedia.org/wiki/Freedesktop.org)
:::info
**Can I mix those applications based on different desktop environments on One System?**
These Desktop Environments have a set of utilities/window managers/design specification to create a more unified desktop. You can mix the two if you feel like it, but you may run into issues with colliding standards and applications, like library conflict issues (which is a bit more common on systems like *Arch*).
Refs:
- [What is KDE, GTK, GTK+, QT, and/or GNOME? - Ask Ubuntu](https://askubuntu.com/questions/249150/what-is-kde-gtk-gtk-qt-and-or-gnome)
:::
### GNOME
*System built by using ++GTK+ GUI toolkit++ on Linux's X11 / Wayland windowing system*
More:
- [GNOME Shell's Architecture - GJS](https://gjs.guide/extensions/overview/architecture.html)
#### GNOME JavaScript (GJS)
- Source Code Repository: [GJS - GNOME GitLab](https://gitlab.gnome.org/GNOME/gjs)
- API Docs & Guides:
- [Developer Guides - GJS](https://gjs.guide/guides/)
- [GJS Docs & Apps - GNOME GitLab](https://gitlab.gnome.org/GNOME/gjs/-/tree/master/doc)
- [GJS Docs - GJSify](https://gjs-docs.gjsify.org/)
- [GJS Docs - GNOME](https://gjs-docs.gnome.org/)
:::warning
**GJS or Node.js?**
Since GNOME currently runs on ++GJS++ runtime integrated with ++GTK\+++ GUI Toolkits, if GNOME migrates from ++GJS++ to ++Node.js++, the GUI applications in GNOME that use ++Node.js++ integrated with other GUI frameworks (e.g., Electon: ++Node.js++ with ++Chromium++ Browser Engine) can now integrate & directly run on native Operating System's GUI system, bypassing the browser application layer.
- Source Code Repository: [node-gtk - GitHub](https://github.com/romgrk/node-gtk)
- API Docs: [node-gtk Docs - GJSify](https://node-gtk-docs.gjsify.org/)
More:
- [Need contribution, Node-GTK project - DEV Community](https://dev.to/donnisnoni/need-contribution-node-gtk-project-19pn)
:::
#### GNOME Shell Extensions
- [Extension Guides - GJS](https://gjs.guide/extensions/)
- **Fatal Error Message:** `GNOME Extensions` > *Extension Package Info*
- **Debug Error/Log Message:** `journalctl -f -o cat /usr/bin/gnome-shell`
- **Reload Extension Package for X11:**
- restart the X11 session: `Alt` + `F2` + `(r)estart`
- via `GNOME Extensions` (if no fatal error):
1. *Extension Package on/off*
2. `gnome-extensions enable/disable <package-directory-name>`
### KDE
*System built by using ++Qt GUI toolkit++ on Linux's X11 / Wayland windowing system*
[Home - KDE Community](https://community.kde.org/Main_Page)
## Cross-Platform GUI Frameworks
*Graphics System Solutions with Cross-Platform ++Windowing++ × ++Graphics++ Libraries*
These frameworks provide a consistent set of APIs & libraries that abstract away the underlying OS differences frameworks for GUIs regardless variations of structures, allowing developers to maintain just one codebase.
:::warning
**Compatibility vs. Performance**
These cross-platform frameworks typically take native app performance & optimization as a quid pro quo for the broad compatibility supports for cross platforms.
:::
> GUI system stack example:
> 
> Different GUI framework's applications on one GNU/Linux's ++Wayland windowing system++ × ++Mesa 3D graphics backend++:
> 
### GTK / GDK
*GTK (GIMP ToolKit & GTK+) / GDK (GIMP Drawing Kit)*
*GDK* is a basic wrapper for standard *Xlib* function calls to access the underlying windowing & graphics systems. GDK lies between the display server and the GTK library, handling basic rendering such as `Drawing Primitives`, `Raster Graphics (Bitmaps)`, `Cursor`, `Font`, `Window Event`, & `User Interaction Functionality`.
[GTK、GDK、GLIB 三者的關係 - Alibaba Cloud](https://topic.alibabacloud.com/tc/a/the-relationship-between-gtk-gdk-and-glib_8_8_20266211.html)
> GTK and GDK traditionally use *GLX* when running on ++X11++ to support OpenGL rendering. *GLX* enables GTK applications to use OpenGL for rendering within ++X11++ windows.
> 
#### GLib
*An Abstraction Layer of Utility Libraries for Cross-Platform Applications*
GLib includes advanced data structures (e.g, objects, linked lists, dynamic arrays, hash tables, trees, strings (including lexical scanner)), memory management, & multi-threading programming functions (e.g., mutexes, asynchronous queues, secure memory pools, callbacks, timers, IPCs), & other software development tools (e.g., macros, assertions, dynamic loading).
For example:
- [/glib/glib/gthread.c](https://gitlab.gnome.org/GNOME/glib/-/blob/main/glib/gthread.c): A cross-platform function wrapper of thread libraries for *POSIX* & *WIN32* operating systems.
```clike=
...
#if defined(THREADS_POSIX)
#include "gthread-posix.c"
#elif defined(THREADS_WIN32)
#include "gthread-win32.c"
#else
#error "No threads implementation"
#endif
...
```
More:
- [向高手学习：glib如何来封装跨平台的线程库 - IOT物联网小镇](https://cloud.tencent.com/developer/article/1823814)
Resources
- Source Code Repository: [GLib - GNOME GitLab](https://gitlab.gnome.org/GNOME/glib)
- API Docs:
- [GLib - GTK Docs](https://docs.gtk.org/glib/)
- [GObject - GTK Docs](https://docs.gtk.org/gobject/)
- [GModule - GTK Docs](https://docs.gtk.org/gmodule/)
- [GIO - GTK Docs](https://docs.gtk.org/gio/)
### Qt
A cross-platform application development framework for creating cross-platform GUI on *Linux*, *Windows*, *macOS*, *Android*, & *embedded systems*.
> 
> 
>
> (Source: [Qt常见类、窗口类继承关系树图 - CSDN](https://blog.csdn.net/m0_73443478/article/details/130953630))
- **Input Events:**
```sh!
evdev -> /dev/input/* devices -read/select/epoll-> windowing system -/tmp/-> QApplication::exec (QEventLoop::exec) -select/epoll-> Qt gui elements (mousePressEvent, QML onClicked, ...)
```
- **Graphics Rendering:**
:::info
- **Using software rendering**
```sh!
Qt app -> Qt rendering backend, CPU rendering -UDS-> windowing system, shm
```
- **Using graphics library**
Using direct rendering:
```sh!
Qt app -> Qt rendering backend -opengl,...-> graphics library -ioctl-> DRM, DMABUF
-UDS-> windowing system, surface
```
Using windowing system (old architecture):
```sh!
Qt app -> Qt rendering backend -egl,...-> windowing system's 2D/3D drivers -ioctl-> kernel driver
-UDS-> windowing system, surface
```
:::
- **Qt app:**
[Qt for Beginners - Qt Wiki](https://wiki.qt.io/Qt_for_Beginners)
- Qt gui elements: signals, slots, properties, treenode-like management, ...
- `QObject`:
```sh!
QObject -> QApplication / QWindow / QNetworkAccessManager / QLayout / QTimer / QWidget / ... -> QLabel / QText / QGraphicsView / QMainWindow / ...
```
- `QWidget`:
- `QMainWindow`: for `QWidget`-based apps.
- Qt rendering targets:
- `QPaintDevice`:
```sh!
QPaintDevice -> QWidget / QImage / ...
```
:::info
- `QPaintEngine`: rendering backends of `QPaintDevice`.
[QPaintEngine - Qt Docs](https://doc.qt.io/qt-6/qpaintengine.html)
- `QPainter`: drawing, clipping, transform, opacity, ...
[QPainter - Qt Docs](https://doc.qt.io/qt-6/qpainter.html)
:::
- `QSurface`:
```sh!
QSurface -> QoffscreenSurface / QWindow -> QVulkanWindow / QPaintDeviceWindow -> QOpenGLWindow / QRasterWindow
```
:::info
- `QOpenGLContext`:
[QOpenGLContext - Qt Docs](https://doc.qt.io/qt-6/qopenglcontext.html)
1. GL provider: OpenGL context, OpenGL ES context.
2. surface context: EGL, GLX, ...
- `swapBuffers`: call the windowing system to flip front/back framebuffers:
> - on-screen framebuffer: for displaying.
> - off-screen framebuffer: for writing.
```C++=
QOpenGLContext context;
QWindow window;
context.setSurface(&window);
context.create();
context.makeCurrent(&window);
```
:::
- `QWindow` (`QObject` x `QSurface`): on-screen surface.
[QWindow - Qt Docs](https://doc.qt.io/qt-6/qwindow.html)
- `QOffscreenSurface`: off-screen surface.
[QOffscreenSurface - Qt Docs](https://doc.qt.io/qt-6/qoffscreensurface.html)
- **graphics library:** graphics driver x dri
- Mesa 3D: gallium 3D (graphics drivers / software rendering), ...
- graphics driver:
- Vulkan: Vulkan loader, Vulkan ICD (Installable Client Driver), ...
- **windowing system:** surface managment, compositor
```sh!
windowing system, display -ioctl-> KMS
windowing system, compositor -opengl,...-> graphics library -ioctl-> DRM, DMABUF
windowing system, surface -egl,...-> graphics library / generic buffer manager (GBM) -ioctl-> DRM, DMABUF
```
### EFL (Enlightenment Foundation Libraries)
EFL grew out of the development of Enlightenment (++X++'s window manager & ++Wayland++'s compositor).
It covers a wide range of functions, including graphics, user interfaces, inter-process communication (IPC), audio and even location services. Other powerful features include file handling utilities, thumbnailing and rendering via scene graph.
- [About EFL - enlightenment.org](https://www.enlightenment.org/about-efl.md)
- [Design UI applications with EFL - Tizen](https://developer.tizen.org/dev-guide/2.4.0/org.tizen.ui.practices/html/native/efl/guides_efl_n.htm)
### SDL (Simple DirectMedia Layer)
A easy-to-use cross-platform development toolkit designed to provide a low-level interface for handling graphics, sound, & input, commonly used in game development & multimedia applications. It supports Windows, macOS, Linux, iOS, & Android.
- **Graphics:** Provides functionality about windows, surfaces, & basic graphics rendering.
- **Input:** Handles input devices (e.g., keyboards, mice, game controllers).
- **Audio:** Provides functionality for playing sounds and music, including support for multiple audio formats.
- **Timing Function:** Manages frame rates and perform tasks at regular intervals.
> Useful for game loops and animations.
[Simple DirectMedia Layer - Wiki](https://en.wikipedia.org/wiki/Simple_DirectMedia_Layer)
:::warning
**What platforms, tools should I choose?**
SDL is a tool more bound to game dev since it not only handles graphics & basic input (like what ++Windowing Systems++ do) but also integrates audio & comprehensive realtime gaming experiences. All contents are bundled as a toolkit / library for developer use, as well as an abstraction layer for cross-platform, cross-device implementation.
It is considered a better alternative option for *GLFW*, but still not perfect. It is a good practice to have an own-designed game engine that is able to handle specific effects or tasks (e.g., multiplayer networking, device integration).
Programming languages, platform environments, & supported infrastructures are also crucial points when talking about what platform to go on. Other considerations & coresponding choices may like:
- **Dedicated Game Engine:** Unity, Unreal Engine, Frostbite, ...
- **Cross-Platform GUI Applications:** Qt, GTK/GDK, WebView, Electron, ...
- **Runtime, Low-Level Infrastructure Support:** WinAPI, GL, Eevee (Realtime Rendering), C#, C/C++, JVM, LLVM, Proton, Wine, Bluetooth, ...
:::
### GLFW, GLUT
*A multi-platform library for OpenGL, OpenGL ES, Vulkan, window and input*
[glfw - GitHub](https://github.com/glfw/glfw)
:arrow_right: Read more: [*OpenGL Libraries*](#OpenGL-Libraries).
## Linux Windowing Systems
Manages the tasks of *display screens*, *window management*, *event handling*, *visual effects*, etc.
Windowing systems are a type of GUI which implements the *WIMP* (`Window`, `Icon`, `Menu`, `Pointer`) paradigm for a user interface.
[Windowing system - Wiki](https://en.wikipedia.org/wiki/Windowing_system)
:::info
**[Extra] Get rid of Windowing System on Embedded Systems?**
*Using DRI/DRM with GBM to replace X11/Wayland work*
Refering to the 3D applications bypassing the windowing system to directly access the underlying graphics facilities, if only running the web applications on browser engines, can we eliminate the windowing system & streamline the user-space graphics stack to increase performance?
[Using the Chrome OS graphics stack on Intel-based Linux desktops - Medium](https://medium.com/joone/using-the-chrome-os-graphics-stack-on-intel-based-linux-desktops-38f56f2ed6e4)
:::
### X Window System (X11)
X supports one or more *screens* containing overlapping *windows* or *subwindows*, & provides ==graphics==, ==text==, & ==raster operations== for those *windows*.
Refs:
- [x(7) - Linux man page - die.net](https://linux.die.net/man/7/x)
- :+1: [[Book] X Window System User's Guide (Definitive Guides to the X Window System) - Valerie Guercia](https://www.oreilly.com/library/view/x-window-system/9780937175149/Chapter01.html)
- :+1: [[Book] Xlib - C Language X Interface - x.org](https://www.x.org/releases/current/doc/libX11/libX11/libX11.html)
- :+1: [Explanations: X Window System Basics - magcius](https://magcius.github.io/xplain/article/)
- [X Window System core protocol - Wiki](https://en.wikipedia.org/wiki/X_Window_System_core_protocol)
#### X Server
A display server that is responsible for coordinating all clients connected via ++UNIX-domain Sockets++ within the *X protocol*, taking input from the mouse and keyboard, & pushing pixels on the output.
> X11 is network-transparent, enabling client & server to run on the same or different machines, and to communicate over the Internet via an encrypted session.
:::info
**Display Server Concept**
A single **X Server** can provide display services for any number of *screens*. A *screen* is a physical monitor and hardware that can be *color*, *grayscale*, or *monochrome*. A set of *screens* for a single user with ++one keyboard++ and ++one pointer++ (e.g., a mouse) is called a *display*.
A *window* can only be created as a *subwindow* of a parent *window*. This causes the *windows* to be arranged hierarchically in a tree. The **X server** automatically creates the root of the tree, called the *root window*. The top-level *windows* are exactly the direct *subwindows* of the *root window*.
> Visibly, the *root window* is as large as the screen, and lies behind all other *windows*.
A *window* can have a border that is any pattern (`Pixmap`) or solid color, so as other *window* attributes.
:::
:::warning
**Window Repainting Occurrence**
Unless the *window* has backing store on the server, the output to areas covered by other *windows* is composited & suppressed (only render the crucial parts) by the windowing system.
X does not guarantee to preserve the contents of *windows*. When part or all of a *window* is hidden and then brought back onto the screen, its contents may be lost. The server then sends the client program an Expose event to notify it that part or all of the *window* needs to be repainted.
Programs must be prepared to regenerate the contents of *windows* on demand.
:::
:::info
**Window Compositor & Manager**
**The X display server** fundamentally supports *window* rendering but lacks advanced features like *multi-monitor support*, *native transparency effects*, *high-resolution displays*, *vertical synchronization (vsync)*, & other modern graphical demands. To overcome these issues, a compositor is introduced into the architecture.
> This is where ++Wayland++ steps in. ++Wayland++ isn't a *Display Server* like ++X11++, but instead just describes a way for applications to communicate with a compositor directly. ++Wayland++ attempts to fix many of the problems of **X**, and for most it's at a stage of being a complete replacement (and even a solid improvement in some areas).
>
> [What are the differences between display servers, window managers and compositors? Also how do these change when comparing Xorg and Wayland? - Reddit](https://www.reddit.com/r/linux4noobs/comments/16o83wp/what_are_the_differences_between_display_servers/)
The compositor acts as an intermediary, receiving frames from the **X Server**, applying visual effects such as shadows, transparency, and animations, and then rendering these enhanced frames to the screen. This additional layer allows for smoother and more visually appealing user interfaces by managing ++off-screen buffers++, which store the visual content and styling information of each *window* for each application, known as ==Compositor Overlay Windows (COWs)==.
Each frame from each running *window* goes through the compositor. The compositor grabs the `Pixmap` of the *windows* from the **X server** and renders it onto the ++OpenGL++ scene.
Refs:
- [The compositing manager as the display server - Wayland](https://wayland.freedesktop.org/docs/html/ch01.html#sect-Compositing-manager-display-server)
:::
#### XLib
*A library that exposes X Server's API for X clients' use*
Most of the functions in **Xlib** just add requests to an `Output Buffer`. These requests later execute asynchronously on the **X Server**. Functions that return values of information stored in **X Server** do not return (that is, they block) until an explicit reply is received or an error occurs. If there exist error handlers, they will be called when the error is reported.
If a client does not want a request to execute asynchronously, it can follow the request with a call to `XSync`, which blocks until all previously buffered asynchronous events have been sent and acted on. As an important side effect, the `Output Buffer` in **Xlib** is always flushed by a call to any function that returns a value from **X Server** or waits for input.
- **Sharable Objects:**
Many **Xlib** functions will return an integer resource ID, which allows you to refer to objects stored on **X Server**. These can be of type `Window`, `Font`, `Pixmap`, `Colormap`, `Cursor`, or `GContext`, as defined in `X11/X.h`. These resources are created by requests and are destroyed (or freed) by requests or when connections are closed.
- `Window` is manipulated explicitly by ++window manager program++.
- `Font` and `Cursors` are shared automatically across ++multiple screens++.
- `Fonts` are loaded and unloaded as needed and are shared by multiple clients, & often cached in **X Server**.
> **Notes:** **Xlib** provides no support for sharing `Graphics Contexts` between applications.
- **Events:**
Client programs are informed of events. `Event` may either be side effects of a request (for example, restacking windows generates Expose events) or completely *asynchronous* (for example, from the keyboard). A client program asks to be informed of events. Because other applications can send events to your application, programs must be prepared to handle (or ignore) `Event` of all types.
`Input Event` arrive *asynchronously* from **X Server** & are *queued* until they are requested by an explicit call (e.g,, `XNextEvent` or `XWindowEvent`). In addition, some library functions (e.g,, `XConfigureWindow`, `XMoveWindow`, `XResizeWindow`, `XRestackWindows`, `XSetWindowBorderWidth`) generate `Expose Event` & `ConfigureRequest Event`. These events also arrive *asynchronously*, but the client may wish to explicitly wait for them by calling `XSync` after calling a function that can cause **X Server** to generate events.
#### X11 Logs
```sh
cat /var/log/Xorg.0.log
```
#### X Extensions
Since X11 is an extensible protocol, it is possible to add additional features without breaking the existed rules.
- **[XRender](https://en.wikipedia.org/wiki/X_Rendering_Extension):** Enables hardware-accelerated rendering of 2D graphics (e.g., alpha blending, anti-aliasing, image compositing operations) on the *X Server*.
### Wayland
A protocol that applications can use to talk to a display server in order to make themselves visible and get input from the user (a person). A Wayland server is called a **compositor**. Applications are Wayland clients. And it makes use of **EGL**.
[Wayland (protocol) - Wiki](https://en.wikipedia.org/wiki/Wayland_(protocol))
#### Weston - Wayland Server
*Weston is the reference implementation of a Wayland server*
[Weston - Arch Linux Wiki](https://wiki.archlinux.org/title/Weston)
:::warning
**Why is Wayland Bad?**
*Should I choose Wayland or X11?*
It's not a 1 & 0 problem, it's a trade-off between system performance, compatibility, & workaround.
More:
- [[Article] Think twice before abandoning Xorg. Wayland breaks everything! - probonopd](https://gist.github.com/probonopd/9feb7c20257af5dd915e3a9f2d1f2277)
- [比較X11與Wayland。2024年了，你是否該切換到Wayland，拋棄X視窗系統 - Ivon的部落格](https://ivonblog.com/posts/should-you-switch-to-wayland/)
:::
## Platform-Specific Graphics Interfaces
*Graphics Interfaces between User Clients & Windowing Servers*
### Direct Rendering History
Direct Rendering provides a highly optimized path for sending 3D data directly to the graphics hardware.
:::info
**XFree86's X Server vs. SGI's GLX vs. Mesa vs. DRI**
The DRI was initially developed by *Precision Insight (PI)* funded by *Red Hat*, *SGI*, *3Dfx*, *Intel*, & *ATi* in order to integrate the GLX extension for 3D rendering in an X11 window.
The 3D core rendering component is the Mesa library. SGI has open-sourced GLX extension framework, which essentially provides the glue between the 3D library (i.e., Mesa) & the windowing system (i.e., XFree86 X Server).
*Precision Insight* has integrated these components into the XFree86 X Server & added a Direct Rendering Infrastructure (DRI). This release provides a complete implementation of direct rendering support for the graphics cards (e.g., 3Dfx *Banshee*, *Voodoo3*, *Voodoo5*, & ATI *Rage 128*) & integrated GPU (e.g., Intel *i810*, *i815*).
[Overview of XFree86 4.x - xfree86.org](https://www.xfree86.org/4.2.0/RELNOTES4.html)
> The initiative of the DRI:
> 
> (Source: [The DRI project history - freedesktop.org](https://dri.freedesktop.org/wiki/DriHistory/))
:::
### GLX, WGL, AGL, (EGL)
*OpenGL Extension implemented by Each Windowing System Interface*
An interface that synchronizes ++underlying native platform windowing systems++ (e.g., *X11*, *Wayland*) & ++Khronos rendering APIs++ (e.g., *OpenGL ES*, *OpenVG*).
[OpenGL、OpenGL ES、OpenVG、GLX、EGL 的简介 - CSDN](https://blog.csdn.net/Cappuccino_jay/article/details/125225930)
:::warning
**Differences in Graphical Interfaces across Platforms**
*GLX* is the implementation of the OpenGL interface for the ++X Server++; *WGL* is for ++Windows++; and *AGL* is for ++Apple++.
On the other hand, *EGL* for ++OpenGL ES++ is *platform-independent* (windowing system vendors implement its specs), & is assumed to support *direct rendering* only.
More:
- [[Book] EGL 1.5 Specs - Khronos](https://registry.khronos.org/EGL/specs/eglspec.1.5.pdf)
- [[Book] OpenGL® on Silicon Graphics Systems](https://www-f9.ijs.si/~matevz/docs/007-2392-003/sgi_html/)
:::
:::info
**Porting OpenGL Apps on Specific Platform to Another**
For example, porting apps from ==X11 + OpenGL (via GLX)== to ==Win32 + OpenGL (via WGL)==.
1. **Rewrite the windowing system specific code:** window creation, event handling, ...
> E.g., ++X11++ instances & functions -> ++Win32++ instances & functions
2. **Translate the OpenGL interface specific code:** pixel format functions, pixmaps, framebuffers, device context functions, rendering context functions, ...
> E.g., ++GLX++ functions -> ++WGL++ functions
:::warning
**For Example, from GLX's `Pixmap` to WGL's `Bitmap`**
On ++X11++, GLX provides `glXCreateGLXPixmap` & `glXDestroyGLXPixmap` to create & destroy `Pixmap` (for off-screen rendering) respectively.
On the other hand, ++Windows++ uses device-independent `Bitmap` that serve the same function as GLX's `Pixmap`.
[Porting GLX Pixmap Code - Microsoft Learn](https://learn.microsoft.com/en-us/windows/win32/opengl/porting-glx-pixmap-code)
:::
[Porting X Window System Applications - Microsoft Learn](https://learn.microsoft.com/en-us/windows/win32/opengl/porting-x-window-system-applications)
:::
#### Rendering Objects
- **Windowing System's Drawing Surfaces**
The surfaces that represent the rendering areas, such as:
- **Window:** *(Back-buffered) on-screen rendering*. It is tied to native `Window` API, like an ++Android++ `Surface` or an ++X11++ `Window`.
- **Pixmap (Image Buffer with 2D x N-Depth):** *(Single-buffered) off-screen storage of graphics objects*. It is tied to native `Pixmap` API, like an ++X11++ `Pixmap`. It is used to define a pattern / tile & is interchangeable with `Window`.
- **Pixel Buffer (pBuffer):** *(Back-buffered) off-screen rendering*. After rendering to a `pBuffer`, it can be used as a texture or further manipulate it before displaying or storing it. It is considered more flexible than `Pixmap` because this buffer isn't tied to any native instance.
> `pBuffer` is obsolete. Consider using `Framebuffer Object (FBO)` instead.
> [pBuffer - OpenGL Wiki](https://www.khronos.org/opengl/wiki/P-buffer)
- **Rendering Models**
++EGL++, ++OpenGL++, & ++OpenGL ES++ support 2 rendering models:
- **Back Buffered Rendering:** Memory for the color buffer used during rendering is allocated and owned by ++EGL++.
> `pBuffer` Surface has a *Back Buffer* but no associated `Window`, so the *Back Buffer* needn't be copied.
- **Single Buffered Rendering:** Memory for the color buffer is specified at surface creation time in the form of a native `Pixmap`, and client APIs are required to use that memory during rendering. When the client is finished drawing a frame, the native `Pixmap` contains the final image.
> `Pixmap` Surface typically does not support multisampling anti-aliasing, since the native `Pixmap` used as the color buffer is unlikely to provide space to store multisample information.
:::info
**The Lifespan of a Drawable Surface**
0. **Rendering Context Construction:** Using `eglCreateContext` to create a `Rendering Context` instance.
1. **Surface Construction:** Using `eglCreateWindowSurface` / `eglCreatepBufferSurface` / `eglCreatePixmapSurface` to create a `Surface` instance.
2. **Binding to Rendering Context:** Using `eglMakeCurrent` to bind a *Surface* to a `Rendering Context`.
3. **Buffer Swapping:** When the client is finished drawing a frame, the *Back Buffer* may be copied to a visible `Window` using `eglSwapBuffers`. This operation swaps the front and back buffers to display the rendered content.
4. **Surface Destruction:** When no longer needed, *Surface* can be destroyed using `eglDestroySurface` to free up resources.
:::
- **Windowing System Drawables (in GLX):**
A drawable is something into which ++windowing systems++ (here ++X++ for example) can draw, either a `Window` or a `Pixmap`. An exception is a `Pixel Buffer (pBuffer)`, which is a *GLX drawable* but cannot be used for ++X++ rendering.
A *GLX drawable* (++GL++ + ++X++) is something into which both ++X++ and ++OpenGL++ can draw, either an OpenGL capable window or a *GLX pixmap*.
### DRI (Direct Rendering Infrustructure)
*An interface that ++windowing system++'s clients can directly access graphics hardware by **DRM***
[DRI Project - SourceForge](https://sourceforge.net/projects/dri/)
- **DRI Client:**
An application like an ++X client++ that performs *direct rendering*. It requires a hardware-specific driver, often provided as a shared library, to interact with the graphics adapter. These drivers usually implement a 3D API such as OpenGL (`libGL.so`), and may be provided by the hardware vendors (e.g., 3dfx, ATi, Nvidia, Intel) or projects (e.g., Mesa 3D).
:::warning
**Example of Usage**
Video games outsource rendering calculations to the GPU over graphics API (e.g., OpenGL) in real-time. Shaders are written in GLSL (source code) or SPIR-V (shader IR) and compiled on the CPU. then, send the compiled shader programs to the GPU for execution.
:::
- **Windowing System Server as a DRI Client:**
The **X Server** includes a `DRI Extension` for coordinating with both ++windowing system++ & ++DDX driver++ (Device-Dependent-X driver for video card's 2D hardware acceleration). It often links to the same `DRI Driver` as the clients to enable *hardware-accelerated 3D rendering* through the *GLX* extension for indirect rendering (e.g., for *remote X clients*). For *2D rendering*, the ++DDX driver++ must manage both `DRI Clients` and other uses of the graphics hardware device.
- **Kernel's Direct Rendering Manager (DRM):**
A kernel component regulates access to the graphics hardware. Both the **X Server**'s ++DDX driver++ & the `DRI Clients` use `DRM` to manage *synchronization* and *security*, ensuring that concurrent processes do not interfere and that each client only accesses the hardware as needed for rendering.
:::info
:arrow_right: For more detail on DRM, please heading to below [DRM Direct Rendering Manager](#DRM-Direct-Rendering-Manager) section.
:::
:::info
**The History of Windowing Systems & Applications with DRI on Linux**
[Graphics Stack總結(一)Linux Graphics Stack簡介 - 青山牧雲人 - 博客園](https://www.cnblogs.com/ArsenalfanInECNU/p/15787816.html)
- **User Apps -> X Server (DIX -> XAA -> DDX) -> 2D GPU**
The display server of X receives clients' graphics rendering commands & makes use of the *XAA (XFree86 Acceleration Architecture) driver* to turn those *Device-Independent-X (DIX)* commands into *Device-Dependent-X (DDX)* commands.
:::warning
**Cross-Platform 2D Graphics Driver**
The XFree86 X server has a built-in runtime loader, which enables loading normal object files and libraries in most of the commonly used formats.
Since the loader doesn't rely on an operating system's native dynamic loader support, it works on platforms that don't provide this feature, and makes it possible for the modules to be operating system independent.
[Overview of XFree86 4.x - xfree86.org](https://www.xfree86.org/4.2.0/RELNOTES4.html)
:::
- **User Apps / 3D Apps -> X Server (2D Driver / GLX Driver) -> 2D / 3D GPU**
X Clients only can communicate with the unified **X Server** via *X protocol* or *GLX* -- OpenGL interface on X.
- **3D GPU Direct Rendering Access Support (via DRI Driver)**
With user-space `DRI` support, clients now can directly communicate with in-kernel `DRM` & manipulate GPU drivers directly via graphics library (e.g., Mesa 3D).
:::warning
Dedicated design for intensive 3D rendering or frequent GPU access apps.
:::
:::
Refs:
- [Direct Rendering Infrastructure - Wiki](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure)
- [Direct Graphics Access - Wiki](https://en.wikipedia.org/wiki/Direct_Graphics_Access)
## Cross-Platform Graphics APIs
*An Abstraction Layer for Cross-Platform Apps to Use Underlying Graphics API Calls*
### WebGL
*A Cross-Platform Graphics API Abstracting OpenGL ES as One of the JavaScript API*
WebGL is an API for rendering 2D & 3D graphics within web browsers typically using JavaScript, it translates JavaScript's WebGL instructions to OpenGL ES.
:::info
On browsers, ++WebGL 1, 2++ introduce an API that closely conforms to ++OpenGL ES 2.0, 3.0\+++ that can be used in HTML `<canvas>` elements.
:::
```!
WebGL API Calls (1 of JavaScript API)
--translate--> OpenGL ES API Calls (inside browser)
Does native graphics drivers support OpenGL ES call?
--yes--> Call OpenGL ES driver
--no--> Translate into Vulkan, OpenGL, ... (via ANGLE, see below table)
```
| Platforms | Underlying Compatibility Layer | Native Graphics Driver |
|:-----------:|:------------------------------:|:----------------------------------------------:|
| **Linux** | X / ANGLE | native OpenGL ES driver (via Mesa 3D) / Vulkan |
| **Windows** | ANGLE | Direct3D, Vulkan |
| **macOS** | ANGLE | Metal (via Vulkan2Metal) |
| **Mobiles** | X / ANGLE | native OpenGL ES driver / Vulkan |
:::info
:arrow_right: For more on ANGLE translation layer for WebGL, please head to [*ANGLE*](#ANGLE-Almost-Native-Graphics-Layer-Engine).
:::
:::info
:arrow_right: For more on native graphics API, please head to [*Graphics APIs*](#Graphics-APIs).
:::
More:
- [WebGL Wiki - Khronos](https://www.khronos.org/webgl/wiki/Main_Page)
- [WebGL: 2D and 3D graphics for the web - Web APIs - MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API)
## Cross-Platform Graphics Libraries
*Cross-Platform Rendering & Graphics Function Libraries that Abstract Different Graphics APIs (e.g., OpenGL, Vulkan, Direct3D, Metal, ...) & Platforms (e.g., Linux, Android, Windows, macOS, ...)*
### ANGLE (Almost Native Graphics Layer Engine)
*A Runtime Translation Layer that Translates ++OpenGL ES++ Calls into Specific Underlying ++Graphics API++ Calls for WebGL (a Kind of JavaScript API) Use*
| from \ to | D3D 9 | D3D 11 | OpenGL | OpenGL ES | Vulkan | Metal |
|:-------------:|:-----:|:------:|:-----------:|:-----------:|:------:|:-----:|
| OpenGL ES 2.0 | O | O | O | O | O | O |
| OpenGL ES 3.0 | | O | O | O | O | O |
| OpenGL ES 3.1 | | | O | O | O | |
| OpenGL ES 3.2 | | | in progress | in progress | O | |
> ++OpenCL 1.0, 1.1, 1.2, 3.0++ to ++Vulkan++, ++OpenCL++ backends is also in progress.
| platform supported graphics APIs | D3D 9 | D3D 11 | OpenGL | OpenGL ES | Vulkan | Metal |
|:--------------------------------:|:-----:|:------:|:------:|:---------:|:------:|:-----:|
| Windows | O | O | O | O | O | |
| Linux | | | O | | O | |
| macOS | | | O | | | O |
| iOS | | | | | | O |
| Android | | | | O | O | |
> On Linux, apps can use ++OpenGL ES++ implementation of *Mesa 3D* as the ++OpenGL ES++ renderer instead of ++Vulkan++ drivers by GPU vendors (translated via ANGLE).
:::warning
**Example of Usage**
ANGLE is the ++WebGL++'s backend for **Chrome** & **Firefox**.
:::
Source:
- [ANGLE - Almost Native Graphics Layer Engine - Chromium - Google Source](https://chromium.googlesource.com/angle/angle)
#### Cross-Platform Components
Portions of the ANGLE shader compiler are used as a ***shader validator*** and ***translator*** by ++WebGL++ implementations across multiple platforms, such as on macOS, Linux, and in mobile variants of the browsers.
- **Shader Validator:**
Having single validator helps ensure that a consistent set of ++GLSL ES shaders++ is accepted across browsers and platforms.
- **Shader Translator:**
Translates shaders to other shading languages, and tweaks a little to work around bugs or quirks in the native graphics drivers.
### Skia
*High-Level 2D Rasterization Graphics Library*
Provides common APIs that work across a variety of hardware and software platforms. Features *Shapes*, *Bézier Curves*, *Transform*, *Text Rendering*, *Path Effects*, *Shaders*, ...
:::warning
**Example of Usage**
Rendering ++2D graphics++ like text, images, shapes, ... on **Chromium**.
:::
#### Components
- **SKPDF**
*Skia’s PDF backend (SkPDF) via the SkDocument & SkCanvas APIs*
Refs:
- [Using Skia's PDF Backend - Skia](https://skia.org/docs/user/sample/pdf/)
Example code:
```cpp=
void write_pdf_document(SkWStream* output_stream,
void (*draw_page)(SkCanvas*, int),
int page_length,
SkSize page_size) {
auto pdf_document = SkPDF::MakeDocument(output_stream, metadata);
for (int page_index = 0; page_index < page_length; ++page_index) {
SkCanvas* page_canvas = pdf_document->beginPage(page_size.width(),
page_size.height());
draw_page(page_canvas, page_index);
pdf_document->endPage();
}
pdf_document->close();
}
void draw_pdf_page(SkCanvas* canvas, int page_index) {
const SkScalar R = 115.2f, C = 128.0f;
SkPath path;
path.moveTo(C + R, C);
path.lineTo(x0, y0);
path.lineTo(x1, y1);
...
SkPaint paint;
paint.setStyle(SkPaint::kStroke_Style);
canvas->drawPath(path, paint);
}
int main() {
SkDynamicMemoryWStream buffer;
write_pdf_document(&buffer,
&draw_pdf_page,
1,
(SkSize){8.5f * 72, 11.0f * 72});
sk_sp<SkData> pdfData = buffer.detachAsData(); // pdfData.get(); ...
return 0;
}
```
- **Skottie**
*Lottie Animation Player*
A performant, secure native player for JSON animations derived from the *Bodymovin plugin* for ++Adobe After Effects++.
Refs:
- [Skottie - Lottie Animation Player - Skia](https://skia.org/docs/user/modules/skottie/)
Resources:
- [LottieFiles: Download Free lightweight animations for website & apps.](https://lottiefiles.com/)
- **CanvasKit**
*Skia Canvas with WebAssembly*
Easy deployment of graphics APIs on the web with WebAssembly support.
CanvasKit provides a playground for testing new `Canvas` and `SVG` platform APIs. It can help support for deployment of other features like *Skia’s Lottie animation*.
Refs:
- [CanvasKit - Skia + WebAssembly - Skia](https://skia.org/docs/user/modules/canvaskit/)
:::info
**Main Features**
- Encapsulating `WebGL Context` as an `SkSurface`, allowing for direct drawing to an HTML `<canvas>`.
- Available core set of Skia `canvas` / `paint` / `path` / `text` APIs
- Draws to a *hardware-accelerated* backend
- Security tested with Skia’s fuzzers
:::
- **SkSL**
*Skia’s Shading Language Similar to ++GLSL++*
The core difference is GL is for building programmable shaders on actual GPU pipeline (e.g., fragment shaders that control rasterizers & the blending stages), whilst *SkSL* is for shaders on Skia's pipeline.
Skia's pipeline only creates one single GPU fragment shader that calls all of the functions defined in the `SkShader`, `SkColorFilter`, `SkBlender` tree as nodes (1-to-1 function).
> `SkRuntimeEffect` is a Skia C++ object that can be used to create `SkShader`, `SkColorFilter`, & `SkBlender` objects with behavior controlled by *SkSL* code.
Refs:
- [SkSL & Runtime Effects - Skia](https://skia.org/docs/user/sksl/)
This specialized fragment shader covers:
- **Clipping:** Evaluates whether a pixel falls in/outside of the shape/clip being drawn (or on the border, where it might apply antialiasing).
- **`SkPaint` Features**:
- `SkShader` functions (may be a tree from `SkShaders::Blend`)
- `SkColorFilter` functions (may be a tree from `SkColorFilters::Compose`, `SkColorFilters::Blend`)
- `SkBlender` functions (set from `SkBlendModes`, or customized via `SkPaint::setBlender`).
- **Color Space Conversion:** Applies color in `XYZ D50` color space (a linear combination of RGB axis).
- [Skia Color Management - Skia](https://skia.org/docs/user/color/)
- [CIE 1931 color space - Wiki](https://en.wikipedia.org/wiki/CIE_1931_color_space)
- [Color space conversion - Takao Fujiwara](https://fujiwaratko.sakura.ne.jp/infosci/colorspace/index_e.html)
- `sRGB` in Full CIE Color Space (in XYZ space)
- `sRGB` in `P3-D65` (in XYZ space)
Example code:
```cpp=
/*
* The shader object has a .eval() method instead of using "sample" to
* make clear that the code is applied on Skia's pipeline, not the GL
* pipeline.
*
* Also, Skia has simple methods for creating an SkShader from an SkImage,
* so it’s easy to use images like this on runtime effect:
*/
sk_sp<SkShader> imageShader = image->makeShader(
SkSamplingOptions(SkFilterMode::kLinear)
);
// Sample "image", then swap red & blue
const char sksl[] = R""""(
uniform shader image;
half4 main(float2 coord) {
return image.eval(coord).bgra;
}
)"""";
auto [effect, err] = SkRuntimeEffect::MakeForShader(SkString(sksl));
sk_sp<SkShader> myShader = effect->makeShader(nullptr,
{ { imageShader }, 1 });
// Fill the surface with `myShader`:
SkPaint p;
p.setShader(myShader);
canvas->drawPaint(p);
```
### Cairo
*High-Level 2D Rasterization Graphics Library*
Offers features similar to PostScript & PDF, including *stroking / filling Bézier splines*, *image transformations*, *antialiased text rendering*, etc. And all drawing operations can be transformed by any affine transformation (*scale*, *rotation*, *shear*, etc.)
Cairo supports various output targets, including *X Window System*, *Win32*, *PDF files*, *SVG files*, *image buffers*,...
> High-quality CPU-based rendering, but less support & integration for the GPU backends & platforms (specific on Linux with GTK).
:::warning
**Example of Usage**
- Rendering whole graphical control elements on **GTK**-based softwares.
- **Firefox**'s **Gecko Layout Engine** uses Cairo as the graphics back-end for rendering both web page content & the UI (or **Chrome**).
- **WebKit Framework** uses Cairo for all rendering in the **GTK** and **EFL** ports. Support has also been added for **SVG** and `<canvas>` content using Cairo.
:::
[Cairo - cairographics.org](https://www.cairographics.org/)
Example code:
```cpp=
#include <cairo-svg.h>
#include <stdio.h>
int main(int argc, char **argv) {
cairo_surface_t *surface = cairo_svg_surface_create("Cairo_example.svg", 100.0, 100.0);
cairo_t *cr = cairo_create(surface);
/* Draw the squares in the background */
for (int x = 0; x < 10; ++x)
for (int y = 0; y < 10; ++y)
cairo_rectangle(cr, x * 10.0, y * 10.0, 5, 5);
cairo_pattern_t *pattern = cairo_pattern_create_radial(50, 50, 5, 50, 50, 50);
cairo_pattern_add_color_stop_rgb(pattern, 0, 0.75, 0.15, 0.99);
cairo_pattern_add_color_stop_rgb(pattern, 0.9, 1, 1, 1);
cairo_set_source(cr, pattern);
cairo_fill(cr);
/* Writing in the foreground */
cairo_set_font_size (cr, 15);
cairo_select_font_face (cr, "Georgia", CAIRO_FONT_SLANT_NORMAL, CAIRO_FONT_WEIGHT_BOLD);
cairo_set_source_rgb (cr, 0, 0, 0);
cairo_move_to(cr, 10, 25);
cairo_show_text(cr, "Hello");
cairo_destroy(cr);
cairo_surface_destroy(surface);
}
```
### Text Rendering Libraries
*A CPU-bound operation of rasterizing glyph indicies to textures or bitmaps, then passing them to GPU for composition (e.g., integrated in Skia, Cairo, ...).*
[Modern text rendering with Linux: Overview - mrandri19](https://mrandri19.github.io/2019/07/24/modern-text-rendering-linux-overview.html)
#### Font Formats
- **TrueType:**
- **Static TTF:** only support predefined weights (e.g., *Thin*, *Light*, *Regular*, *Bold*, *Black*).
> May with multiple seperate files for each weight.
- **Variable TTF:** with ++TrueType GX Variations (Apple's technology)++ or ++OpenType Font Variations++, a TTF with variations can include a weight axis (`wght`) allowing smooth transitions between weights.
> Only need single variable font file.
<video autoplay playsinline loop width="300px" src="https://www.gstatic.com/images/icons/material/apps/fonts/1x/catalog/checkout/variable.mp4"></video>
- Usage:
```css=
@font-face {
font-family: 'FAMILY_NAME';
font-stretch: PERCENTAGE_RANGE_LOW_TO_HIGH; /* Note that font-stretch is a % of normal width */
font-style: NORMAL_OR_ITALIC;
font-weight: NUMERIC_RANGE_LOW_TO_HIGH;
src: url(FONT_FILE_NAME.woff2) format('woff2'); /* Compressed Web File */
}
h1 {
font-family: 'FAMILY_NAME', sans-serif;
font-variation-settings: 'opsz' var(--text-opsz);
}
```
- **OpenType:**
- **Static OTF:** only support predefined weights (e.g., *Thin*, *Light*, *Regular*, *Bold*, *Black*).
> May with multiple seperate files for each weight.
- **Variable OTF:** support custom weights & variations.
> Only need single variable font file.
More:
- [How to Use - Google Fonts](https://fonts.google.com/selection)
- [Introducing variable fonts - Google Fonts](https://fonts.google.com/knowledge/introducing_type/introducing_variable_fonts)
- [Variable Fonts - Google Fonts](https://fonts.google.com/knowledge/glossary/variable_fonts)
- [Styling type on the web with variable fonts - Google Fonts](https://fonts.google.com/knowledge/using_type/styling_type_on_the_web_with_variable_fonts)
- [Difference between OTF (Open Type) or TTF (True Type) font formats? - Stack Exchange](https://superuser.com/questions/96390/difference-between-otf-open-type-or-ttf-true-type-font-formats)
> OTF is more likely to be a “better” font, as it supports more advanced typesetting features (*smallcaps*, *alternates*, *ligatures* and so on actually inside the font rather than in fiddly separate expert set fonts). It can also contain either ++spline (TTF-style)++ or ++Bezier curves (PostScript Type 1-style)++, so hopefully you're getting the shapes the font was originally designed in and not a potentially-poorer-quality conversion.
>
> On the other hand, if you're downloading free fonts from shovelware sites, you're unlikely to get any of that. Indeed, you may simple be getting a TTF font renamed to OTF.
#### HarfBuzz
*Vector Text Composition Engine*
#### FreeType
*Raster Text Rendering Engine for Linux, BSDs, Android, ChromeOS, iOS, browsers (Chromium, WebKit, & Gecko)*
- **Text Antialiasing:**
- **Subpixel (for LCD):** Works by treating each pixel as three separate light sources: *Red*, *Green*, *Blue*.
- **Grayscale (Standard):** Instead treats each pixel as a single light source, emitting white light.
Refs:
- [Modern text rendering with Linux: Setup FreeType - mrandri19](https://mrandri19.github.io/2019/07/18/modern-text-rendering-linux-ep1.html)
- [Modern text rendering with Linux: Antialiasing - mrandri19](https://mrandri19.github.io/2019/08/08/modern-text-rendering-linux-ep2.html)
:::info
**[Reference] Other Platform's Text Rendering Engines**
- **DirectWrite**
*Text Rendering API for Win32 apps*
> 
>
> (Source: [Text Rendering with Direct2D and DirectWrite - Microsoft Learn](https://learn.microsoft.com/en-us/windows/win32/direct2d/direct2d-and-directwrite))
[Tutorial: Getting Started with DirectWrite - Microsoft - GitHub](https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/DirectWrite/getting-started-with-directwrite.md)
- **CoreText**
*Text Rendering Engine for macOS*
> 
>
> (Source: [CoreText 学习笔记 - zbblogs - 博客园](https://www.cnblogs.com/zbblog/p/15746080.html))
[Core Text - Apple Dev Docs](https://developer.apple.com/documentation/coretext/)
:::
## Graphics APIs
Such as *OpenGL*, *OpenGL ES*, *OpenVG*, *Vulkan*, *DirectX*, *Metal*, *WebGPU*, etc. wrap function calls & define ABIs that encompass how CPU & operating systems communicate with GPU & complete the rendering tasks.
### 3D Graphics Rendering Pipeline
### Computer Graphics History
- **OpenGL** (Cross-Platform Graphics Library Standard, Workstation-Use, Fixed Graphics Pipeline vs. Programmable Graphics Pipeline, Shader Compilation / Caching, Extensions, Vulkan, Mobile-Use: OpenGL ES)、**Direct3D** (PC-Use, Nonextensible, DX12)、**Proton** (WINE: Windows API reimplemented for Unix-Like Systems in a POSIX API way, DXVK: shaders from Direct3D to Vulkan):
- **Graphics Pipeline / PC Evolution:**
- [【OpenGL 篇】为什么游戏总要编译着色器？ - Redknot-乔红](https://www.youtube.com/watch?v=x7_LSjWe2rc)
- [【Direct3D 篇】为什么游戏总要编译着色器？ - Redknot-乔红](https://www.youtube.com/watch?v=6ZraACggaBU)
- [SteamDeck 搭载 Linux，凭什么可以玩 Win 游戏？ - Redknot-乔红](https://www.youtube.com/watch?v=SJTOIzxRU0s)
- [凭什么连 Mac 都能玩 Windows 游戏了 - Redknot-乔红](https://www.youtube.com/watch?v=MP5GcjV2zRM)
- **OpenGL (ES) vs. Vulkan on Mobiles:**
- [安卓換個圖形API就能效能加倍？但谷歌這事幹得太晚了！- 差评硬件部](https://www.youtube.com/watch?v=HIGaO5QP75k)
- **Shader IR Evolution:**
| | Shader IR |
|:------------:|:------------------------------------------------------------------:|
| **OpenGL** | **X** (GLSL-Based) -> **SPIR-V** (OpenGL 4.6~) |
| **Direct3D** | **X** (HLSL-Based) -> **DXBC** (DX11~) -> **DXIL** (DX12~) -> **SPIR-V** |
| **Vulkan** | **SPIR-V** (Open Extensible Standard) |
| **Metal** | **Metal IR** (LLVM bitcode) |
> **Note:** Since the ++Metal++ generate its IR via Clang x LLVM (as *LLVM bitcode*), it is possible to transform *Metal IR* to *SPIR-V* by the tool developed by Khronos: [LLVM/SPIR-V Bi-Directional Translator - Khronos Group - GitHub](https://github.com/KhronosGroup/SPIRV-LLVM-Translator) (whereas the whole graphics API Vulkan->Metal translation can be done via [MoltenVK - Khronos Group - GitHub](https://github.com/KhronosGroup/MoltenVK))
:::warning
**Graphics API Compatibility Issues**
Since the drastic innovation & evolution in the computer graphics industry within only a few decades, the older graphics library API standards are typically not forward-compatible.
:::
:::info
**Why still Slow with Using IR?**
One said that because nowadays the IRs among those graphics APIs are devised & run based on the *LLVM* (compiler backend & utilizing its bitcode), which is considered a massive toolchain that may lead to performance degradation issues during the shader compilation.
:::
### OpenGL Libraries
- **Basics**
- **Core:** (header: `GL/gl.h`, library: `libGL.so`) Defines OpenGL 1.1 function and token declarations (with non-programmable rendering pipelines).
- **Extensions**
- **GLEW (OpenGL Extended Library):** (header: `GL/glew.h`, library: `libGLEW.so`) A platform-independent library that determines & loads the specific version of OpenGL extentions at run-time on the target platform.
> Support including: ++OpenGL extensions++ (gl, glu), ++WGL extensions++ (wgl), ++GLX extensions++ (glx).
- **GLAD:** A cross-platform graphics development solution by generateing loaders to dynamically linking & loading functions of *OpenGL extended library*.
Similar to *GLEW*, but supports multiple libraries, platforms, & languages:
- **Graphics Libraries:** OpenGL, OpenGL ES
- **Windowing System Supports:** EGL, GLX, WGL
- **Programming Languages:** C/C++, D, Volt, Nim, Pascal
:::warning
**Graphics APIs vs. Driver Vendor's Implementations**
First, GLEW & GLAD help us handle graphics function pointers without manually dynamic loading them at runtime (e.g., `glXGetProcAddress`on Linux, `wglGetProcAddress` on Windows).
The reason why we need using those function pointers is that the code is located in the driver of graphics card, GPU, or GPGPU.
> Some may be the functions inside the hardware that are specified & can only be called internally.
OpenGL, Vulkan, & DirectX are only specifications, they don't provide any implementation; the gpu driver is charged with implementing all of the specification (for example, declarations of OpenGL `.h` from ++Khronos (or SGI)++ × implementations of `.cpp` from the ++driver++ vendors).
Also, the fact that OpenGL is a state machine has nothing to do about needing to load the function pointers (for example, Vulkan is not a state machine but loading the function pointers is still a necessary process).
[Good Explanations of differences between GLFW, GLUT, GLAD, GLEW, etc? - Reddit](https://www.reddit.com/r/cpp_questions/comments/ryr3fk/good_explanations_of_differences_between_glfw/)
:::
:::info
**GLAD × GLFW Example**
```clike=
#include <glad/glad.h>
#include <GLFW/glfw3.h>
// OpenGL math functions
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <glm/gtc/type_ptr.hpp>
#include <stdio.h>
const GLuint WIDTH = 800, HEIGHT = 600;
namespace data {
// attrs of each vertex
GLfloat cube_v[] = {
// position // color
-0.5f, -0.5f, -0.5f, 1.0f, 0.0f, 0.0f,
0.5f, -0.5f, -0.5f, 0.0f, 1.0f, 0.0f,
0.5f, 0.5f, -0.5f, 0.0f, 0.0f, 1.0f,
...
};
// indices for each face
GLuint cube_i[] = {
// 1st tri // 2nd tri
0, 1, 2, 2, 3, 0,
...
};
const char* vert_shader = R"(
#version 330 core
layout (location = 0) in vec3 aPos;
layout (location = 1) in vec3 aColor;
out vec3 ourColor;
uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;
void main() {
gl_Position = projection * view * model * vec4(aPos, 1.0f);
ourColor = aColor;
}
)";
const char* frag_shader = R"(
#version 330 core
out vec4 FragColor;
in vec3 ourColor;
void main() {
FragColor = vec4(ourColor, 1.0f);
}
)";
}
namespace shader {
GLuint compile(const char* src_code, GLenum type) {
GLuint shader = glCreateShader(type);
glShaderSource(shader, 1, &src_code, NULL);
glCompileShader(shader);
GLint res;
glGetShaderiv(shader, GL_COMPILE_STATUS, &res);
if (!res) {
GLchar msg[512];
glGetShaderInfoLog(shader, 512, NULL, msg);
printf("%s", msg);
}
return shader;
}
GLuint init() {
GLuint vert_shader = shader::compile(data::vert_shader,
GL_VERTEX_SHADER);
GLuint frag_shader = shader::compile(data::frag_shader,
GL_FRAGMENT_SHADER);
GLuint shader = glCreateProgram();
glAttachShader(shader, vert_shader);
glAttachShader(shader, frag_shader);
glLinkProgram(shader);
GLint res;
glGetProgramiv(shader, GL_LINK_STATUS, &res);
if (!res) {
GLchar msg[512];
glGetProgramInfoLog(shader, 512, NULL, msg);
printf("%s", msg);
}
glDeleteShader(vert_shader);
glDeleteShader(frag_shader);
return shader;
}
}
namespace graphics {
GLuint shader;
GLuint VAO, VBO, EBO;
glm::mat4 m_model, m_view, m_projection;
GLuint model, view, projection;
void bind_data() {
// gen & bind vertex array objects and buffers
glGenVertexArrays(1, &VAO);
glGenBuffers(1, &VBO);
glGenBuffers(1, &EBO);
glBindVertexArray(VAO);
glBindBuffer(GL_ARRAY_BUFFER,
VBO);
glBufferData(GL_ARRAY_BUFFER,
sizeof(data::cube_v),
data::cube_v,
GL_STATIC_DRAW);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER,
EBO);
glBufferData(GL_ELEMENT_ARRAY_BUFFER,
sizeof(data::cube_i),
data::cube_i,
GL_STATIC_DRAW);
// vertex positions
glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 6 * sizeof(GLfloat), (GLvoid*)0);
glEnableVertexAttribArray(0);
// vertex colors
glVertexAttribPointer(1, 3, GL_FLOAT, GL_FALSE, 6 * sizeof(GLfloat), (GLvoid*)(3 * sizeof(GLfloat)));
glEnableVertexAttribArray(1);
}
void init() {
bind_data();
// custom shader program
shader = shader::init();
// perspective transformation matrices
m_model = glm::mat4(1.0f);
m_view = glm::translate(glm::mat4(1.0f), glm::vec3(0.0f, 0.0f, -3.0f));
m_projection = glm::perspective(glm::radians(45.0f), (float)WIDTH / (float)HEIGHT, 0.1f, 100.0f);
// shaders
GLuint model = glGetUniformLocation(shader,
"model");
GLuint view = glGetUniformLocation(shader,
"view");
GLuint projection = glGetUniformLocation(shader,
"projection");
}
void update() {
glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glUseProgram(shader);
// update perspective transformation matrices
model = glm::rotate(model, (GLfloat)glfwGetTime() * glm::radians(50.0f), glm::vec3(1.0f, 1.0f, 0.0f));
glUniformMatrix4fv(modelLoc, 1, GL_FALSE, glm::value_ptr(model));
glUniformMatrix4fv(viewLoc, 1, GL_FALSE, glm::value_ptr(view));
glUniformMatrix4fv(projectionLoc, 1, GL_FALSE, glm::value_ptr(projection));
// render the object
glBindVertexArray(VAO);
glDrawElements(GL_TRIANGLES, 36, GL_UNSIGNED_INT, 0);
glBindVertexArray(0);
// swap buffers
glfwSwapBuffers(window);
glfwPollEvents();
}
void cleanup() {
glDeleteVertexArrays(1, &graphics::VAO);
glDeleteBuffers(1, &graphics::VBO);
glDeleteBuffers(1, &graphics::EBO);
glDeleteProgram(graphics::shader);
}
}
int main() {
if (!glfwInit()) {
// unable to init GLFW
return -1;
}
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
GLFWwindow* window = glfwCreateWindow(WIDTH, HEIGHT, "Title", NULL, NULL);
if (!window) {
// unable to create window
glfwTerminate();
return -1;
}
glfwMakeContextCurrent(window);
glfwSetFramebufferSizeCallback(window, [](GLFWwindow*, int width, int height) {
glViewport(0, 0, width, height);
});
if (!gladLoadGLLoader((GLADloadproc)glfwGetProcAddress)) {
// glad loader unavailable
return -1;
}
/*************** main ***************/
graphics::init();
// update infinite loop
while (!glfwWindowShouldClose(window))
graphics::update();
graphics:cleanup();
/************************************/
glfwTerminate();
return 0;
}
```
:::
- **Cross-Platform Windowing Libraries for OpenGL**
- **[Outdated] GLUT (OpenGL Utility Toolkit):** (header: `GL/glut.h`, library: `libglut.so.3`) A simple utils toolkit that implements portable platform-independent **windowing system APIs** for OpenGL, typically on ++Windows++ & ++X11++.
> The successor *FreeGLUT* is compatable with *GLUT* & *GLFW* APIs.
- **GLFW (OpenGL Utility Toolkit):** (header: `GLFW/glfw3.h`, library: `libglfw.so.3`) Similar to *GLUT*, but is newly designed for game development.
- **Utilities**
- **[Outdated] GLU (OpenGL Utility Library):** (header: `GL/glu.h`, library: `libGLU.so`). A set of convenience methods for utils functions with OpenGL core.
### OpenGL API
*Graphics Library Standards*
> A platform/machine-independent open standard for graphic card driver functions.
#### OpenGL Basics Review
- :+1: [Learn OpenGL - learnopengl.com](https://learnopengl.com/)
- [CS 330 Graphics OpenGL Basics - Loren K. Rhodes - Juniata Collage](https://jcsites.juniata.edu/faculty/rhodes/graphics/opengl1.htm)
- [Introduction to Computer Graphics - NCKU CSIE](https://slideplayer.com/slide/17317162/)
- [[Slides] OpenGL Overview - Introduction to Graphics and Game Development - Arizona State University](https://speakerdeck.com/javiergs/ser332-lecture-03)
:::info
**OpenGL Rendering Pipeline**
[OpenGL Rendering Pipeline - songho.ca](https://www.songho.ca/opengl/gl_pipeline.html)
:::
#### Rendering Contexts
Contain the current OpenGL rendering graphics state, an instance of an OpenGL state machine.
> Only one `Rendering Context` can be bound to at most one `Window` or `Pixmap` in a given thread. If a `Context` is bound, it is considered the current `Context`.
#### OpenGL Extension's Objects
Enhence OpenGL's efficiency by minimizing CPU-GPU data transfers and enabling more complex rendering techniques.
- **Buffer Object on VRAM**
An array of ==unformatted== memory allocated by the `OpenGL Context` (aka the ++GPU++). It is used for *transform feedback processes* or *asynchronous pixel transfers*, as the data type of `Vertex` or `Pixel` retrieved from `Image`, `Framebuffer`, etc.
:::danger
It enables asnyc processing but may cause OpenGL to **block the pipeline if accessed before completion**.
:::
Refs:
- [Buffer Object - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Buffer_Object)
- [API for Buffer Objects - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Category:Core_API_Ref_Buffer_Objects): Create, manage, and delete `Buffer` objects.
- **Framebuffer Object (FBO)**
*R/W Drawing Buffers on VRAM*
An off-screen buffer (other than *default* `Framebuffer` or `Screen`) that can be used to render images, off-screen & directly draw into `Textures` without intermediate copying.
[Framebuffer Object - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Framebuffer_Object)
:::warning
It simplifies rendering pipelines & *eliminate the need for `PBOs` by enabling direct texture use in shaders*.
:::
> 
>
> (Source: [[Slides] Framebuffers - Offscreen Rendering and Post Processing - Francesco Andreussi](https://www.uni-weimar.de/fileadmin/user/fak/medien/professuren/Computer_Graphics/CG_WS_18_19/CPhysiG/05_FramebufferPostProc.pdf))
- **Offscreen Rendering**
- **Post-Process**
E.g., Grayscale, Mirror, Blur, Brightness, Contrast, ...
- **Attachments (Layers)**
E.g., color buffer, depth buffer, stencil buffer, ...
:::info
**The main rendering output target of ++Fragment Shader++**
A `Framebuffer` consists of several attachments, and the fragment shader writes to one or more of these buffers (mainly the color buffer).
:::
- **Renderbuffer Object (RBO)**
*GPU-only Rendering R/W Drawing Buffer (Compared to Pixel Buffer Object)*
E.g., stencil/depth attachments
Similar to `Textures` that contain images, but CANNOT be sampled directly & are optimized for use as render (writing) targets, with native support of *Multisample Anti-Aliasing (MSAA)* available.
If there is no need to read or sample the data from the `Renderbuffer` in subsequent ++shader passes++ (i.e., not using the `Image` data in another ++shader++ after rendering), then `RBOs` are ideal.
> If resampling is required, use `Textures` instead.
:::warning
Useful for off-screen rendering where the data does not need to be used as a `Texture`.
:::
- [Renderbuffer Object - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Renderbuffer_Object)
- [MSAA (Multisample anti-aliasing) - Vulkan](https://docs.vulkan.org/samples/latest/samples/performance/msaa/README.html)
- **Texture**
*Shader-Use Readonly Image-Like Texture (Compared to Texture Buffer Object)*
E.g., color/depth attachments
It can be considered as a container of *`Images` all with the same `Image` format*.
- **Image:** A single array of pixels of a certain dimensionality (2D/3D/cubemap) with specific size (*width*, *height*, & *depth*) & format (*RGB*, *RGBA*).
- **Texture Sampling:** Get the color (data) from the given image & query coordinates.
- **Wrapping:** The behavior when texture sampling falls outside the normalized range of `0.0` ~ `1.0`.
E.g., `GL_REPEAT`, `GL_CLAMP_TO_EDGE` (extends edge color), `GL_CLAMP_TO_BORDER` (filled with specific color), ...
- **Filtering:** Interpolation for upscale/downscale sampling strategies to fit the result into the willing size.
E.g., `GL_TEXTURE_MAG_FILTER`/`GL_TEXTURE_MIN_FILTER` with `GL_LINEAR`, `GL_NEAREST`, `GL_NEAREST_MIPMAP_NEAREST`, ...
- **Mipmapping**
- **Texture Atlases:** Technique that pre-renders all `glyphs` or `images` into a large `Texture` (an atlas) & then can be sampled during rendering to draw each of it.
Refs:
- [Texture - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Texture)
- [Texture atlas - Wiki](https://en.wikipedia.org/wiki/Texture_atlas)
- [The Math of Computer Graphics - TEXTURES and SAMPLERS - FloatyMonkey](https://www.youtube.com/watch?v=DuQDx0ZIxa8)
- **Texture Buffer Object (TBO)**
*Shader-Use General R/W Binary-Data-Based Texture (Compared to Texture)*
E.g., physics simulations, precomputed look-up tables (LUTs), intermediate data buffers, large dataset processing, deformable meshes, ... for shader computing use
A *"large"* (larger than [Uniform Buffer Objects](https://www.khronos.org/opengl/wiki/Uniform_Buffer_Object) size) 1D array of `Texture` that can store large amounts of data that don't fit in traditional Texture types (e.g., 2D/3D/cubemap), but still can be accessed in shaders.
:::danger
It is not designed to use as the rendering buffers in `FBOs`. Use `Textures` or `RBOs` instead.
:::
[Buffer Texture - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Buffer_Texture)
More:
- :+1: [What is the purpose of OpenGL texture buffer objects? - Stackoverflow](https://stackoverflow.com/questions/21424968/what-is-the-purpose-of-opengl-texture-buffer-objects)
- [Uniform Buffers VS Texture Buffers - RasterGrid](https://www.rastergrid.com/blog/2010/01/uniform-buffers-vs-texture-buffers/)
- **Pixel Buffer Object (PBO)**
*CPU-GPU General R/W Drawing Buffer (Compared to Renderbuffer Object)*
E.g., async video streaming, texture uploading, screenshots, ...
Used for *async pixel transfer operations* like:
- Writing to a Buffer & used its data by the Texture
- Reading from `Framebuffer`, `Screen`, `Image`, ... in GPU without blocking the CPU-GPU data transfer or the rendering piplines.
:::info
It allows efficient *GPU-side* handling of operations that traditionally involved costly data transfers.
:::
[Pixel Buffer Object - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Pixel_Buffer_Object)
- **Vertex Array Object (VAO)**
*Vertex-Shader-Use Data Structure*
Encapsulate the state ++(since the OpenGL is a state machine design)++ needed to specify each `Vertex` attribute data.
:::danger
It CANNOT be shared across `OpenGL Contexts`.
:::
- **Vertex Buffer Object (VBO)**
*Vertex Data Buffer*
- **Element Buffer Object (EBO)**
*Index Data Buffer*
Refs:
- [OpenGL學習：VAO和VBO以及幾個頂點繪圖方式比較 - 阿拉格2009](https://blog.csdn.net/arag2009/article/details/52954879)
- [OpenGL: VBO, VAO 與 EBO - 圖學玩家](https://medium.com/@tonytsai225/opengl-vbo-vao%E8%88%87ebo-b6fde4cf4ef1)
Refs:
- [OpenGL Object - OpenGL Wiki](https://www.khronos.org/opengl/wiki/OpenGL_Object)
- [Direct State Access - OpenGL Wiki](https://www.khronos.org/opengl/wiki/Direct_State_Access)
- [VBOs, PBOs and FBOs - The Hacks of Life](https://hacksoflife.blogspot.com/2006/10/vbos-pbos-and-fbos.html)
### OpenGL Shader Language (GLSL)
*A custom program that can be run on the programmable rendering pipeline*
[GLSL Shader - Maya - AutoDesk](https://help.autodesk.com/view/MAYAUL/2022/ENU/?guid=GUID-BD177012-56E1-43D0-B7A5-F4B1A70EA846)
For Example:
- **Vertex Shader**
```glsl=
#version 330 core
// position variable that has attribute position 0
layout (location = 0) in vec3 aPos;
// color output to the fragment shader
out vec4 vertexColor;
void main()
{
// see how we directly give a vec3 to vec4's constructor
gl_Position = vec4(aPos, 1.0);
// set the output variable to a dark-red color
vertexColor = vec4(0.5, 0.0, 0.0, 1.0);
}
```
- **Fragment Shader**
```glsl=
#version 330 core
out vec4 FragColor;
// input variable from the vertex shader (same name & same type)
in vec4 vertexColor;
void main()
{
FragColor = vertexColor;
}
```
## User-Space Graphics Drivers
A Bridge for Graphics Rendering between User Apps (via Graphics API) & In-Kernel Graphics Driver (via DRI-DRM Model).
> 
>
> (Source: [Linux AMD Graphics Stack - Wiki](https://en.wikipedia.org/wiki/AMD_FirePro#/media/File:Linux_AMD_graphics_stack.svg))
### Proprietary Drivers
*Drivers implemented by Graphics Card Manufacturers*
E.g., Vulkan / OpenGL ES drivers (for 3D direct rendering), EGL / GLX drivers (for windowing system's apps)
[NVIDIA® Jetson™ Linux - NVIDIA Docs](https://docs.nvidia.com/jetson/archives/r36.4.3/DeveloperGuide/index.html)
#### Windowing Systems
#### Open Portable Trusted Execution Environment (OP-TEE) Support
*Based on Arm TrustZone Technology*
The overall framework of OP-TEE combines with two major components:
- [optee_os](https://github.com/OP-TEE/optee_os): The trusted side of the TEE (the secure world)
- [optee_client](https://github.com/OP-TEE/optee_client): The untrusted side of the TEE (the normal world).
#### Hardware Acceleration in the WebRTC Framework
*Real-Time Communication for Web Apps with Hardware-Aaccelerated Video De/Encoding*
:::info
**[Reference] Nvidia DriveOS Based on Linux**
*System Stack for Autonomous Vehicle Applications*
[NVIDIA DRIVE OS Linux SDK Developer Guide - NVIDIA Docs](https://developer.nvidia.com/docs/drive/drive-os/6.0.8/public/drive-os-linux-sdk/common/topics/intro_sdk/GettingStarted1.html)
:::
### Mesa 3D
*Open-Source User-Space Graphics Driver for ++OpenGL++, ++OpenGL ES++, & ++Vulkan++ on Linux & BSDs (, & Software Rendering on Windows)*
> **Active Contributors**: Valve, AMD, Intel, Google, VMware, ...
Mesa 3D is an abstraction layer for translating API calls into hardware-specific graphics driver commands. It interacts with the Linux kernel through subsystems like `DRI` - `DRM` & `KMS` to manage direct rendering & graphical output through *Gallium3D*.
:::danger
Vast topic, many issues ought to concern, various scenario ought to deliver different solutions.
:::
#### System Architecture, Platform & Graphics API Support
> 
> 
>
> (Source: [MESA 架构 - Linux GPU 系列 05 - Joy Xu](https://joyxu.github.io/2021/05/13/gpu05/))
More:
- [Platforms and Drivers - Mesa 3D Docs](https://gallium.readthedocs.io/en/latest/systems.html)
#### Gallium3D
*User-Space Abstraction Layer for Accessing GPU Drivers*
:::info
**An Improved & Enhanced Graphics Driver Architecture**
Gallium3D is a new architecture for building 3D graphics drivers. It streamlines complicated DRI (to DRM) drivers, & integrated with multiple graphics APIs, OS & graphics interfaces. Addtitionally, the new driver architecture is an abstraction of modern graphics hardware, rather than a direct `OpenGL -> hardware` translator.
The new driver interface will also assume the presence of programmable vertex/fragment shaders and flexible memory objects.
:::
- **State Trackers:**
The Mesa state tracker is the piece which interfaces core Mesa to the Gallium3D interface. It's responsible for translating Mesa state (blend modes, texture state, etc) and drawing commands (like glDrawArrays and glDrawPixels) into pipe objects and operations.
Traditional fixed-function OpenGL components (such as lighting and texture combining) are implemented with shaders. OpenGL commands such as glDrawPixels are translated into textured quadrilateral rendering. Basically, any rendering operation that isn't directly supported by modern graphics hardware is translated into a hardware-friendly form.
Future state trackers will be created for OpenGL 3.0 and OpenGL-ES 2.x.
More:
- [Gallium3D - Mesa 3D Docs](https://gallium.readthedocs.io/en/latest/)
- [Gallium3D Technical Overview - freedesktop.org](https://www.freedesktop.org/wiki/Software/gallium/)
:::warning
**Graphics Driver Optimization - D3D11 to D3D12 Example**
To benefit from the re-use rendering work, multi-threading scaling, and other changes for more efficient GPU and resource utilization, the system's structures and interfaces need to be redesigned.
- **GPU Command Bundles:**
:::
## Kernel Subsystems Related to GUIs & Graphics
- **GUI Stacks**
- **Graphics Stacks**
- **User Interfaces & Libraries vs. Kernel Modules & Device Drivers**
### DRM (Direct Rendering Manager)
*In-Kernel Direct-Access Graphics/Display Driver System Interface*
Interacts with GPU hardwares for *GPU-side resources management*, *display settings*, *DMA transfer works*, *hardware compositing*, *rendering (shaders, pipelines)*, *parallel computing*, ...
:::success
The glue lying in the kernel between user apps & underlying hardwares (drivers with DMA, or even more, zero-copy UMA)!
> Hardware feature: zero-copy texture upload
> 
>
> (Source: [Using the Chrome OS graphics stack on Intel-based Linux desktops - Medium](https://medium.com/joone/using-the-chrome-os-graphics-stack-on-intel-based-linux-desktops-38f56f2ed6e4))
:::
Open-source examples of DRM interfaces: [Nvidia nouveau](https://nouveau.freedesktop.org/), [AMD's amdgpu](https://dri.freedesktop.org/docs/drm/gpu/amdgpu/index.html), [Intel's i915](https://dri.freedesktop.org/docs/drm/gpu/i915.html).
History:
- [DRM (Direct Rendering Manager) 的发展历史](https://blog.csdn.net/Ciellee/article/details/125741346)
Development guide:
- [GPU Driver Developer’s Guide - Linux Kernel Docs](https://dri.freedesktop.org/docs/drm/gpu/index.html)
> Linux graphics driver stack overview: DRM with KMS
> 
>
> (Source: [DRM KMS overview - STM32 MPU - ST Wiki](https://wiki.st.com/stm32mpu/wiki/DRM_KMS_overview))
> DRM workflow:
> 
> 
>
> (Source: [DRM 图形显示框架 - LubanCat_RK 系列板卡](https://doc.embedfire.com/linux/rk356x/driver/zh/latest/linux_driver/framework_drm.html))
> Graphics driver access architecture & workflow:
> 
>
> (Source: [Ubuntu VPU 硬件编解码 - Firefly](https://www.t-firefly.com/solution/122))
:::warning
**DRM Driver Framework**
1. Kernel brings up GPU driver registered as DRM driver.
2. Kernel then emits user-space `udev`'s `ACTION=="add"` event with:
- **[sysfs] `/sys/class/<genre>` kernel module info**: which points to `/sys/devices/pci0000:000/<bus-addr>/`.
> e.g., `MAJ:MIN` dev number, irq number, power management interface, driver control interface, ...
- **[udev] `/lib/udev/rules.d/<rules>`**: `/dev/<device>` creation rules.
[udev子系統 - linux的裝置管理員](https://handy505.gitbooks.io/rpi/content/udev.html)
3. *udevd* then use above info & interpretes rules at runtime to create user-space `/dev/<device>`.
DRM character drivers:
- High-level GPU render interfaces (e.g., *draw calls*, *GPU computing tasks*) to the users at:
```sh!
/dev/dri/render* # rendering/computing task interfaces (accessing DRM)
```
> User-space graphics drivers with `libdrm` library use `ioctl`, `mmap` (shared memory space), `epoll`, ... to queue & initiate commands to communicate with the underlying DRM.
- High-level display control interfaces (e.g., *display mode setting*, *framebuffer management*, *page flipping*) to the windowing systems at:
```sh!
/dev/dri/card* # display control interfaces (accessing DRM/KMS)
```
For example, Nvidia DRM driver:
- `sys/class/drm/`
- `/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/`
- `50-udev-default.rules`
```sh!
...
# Before c43ff248f94266cfc93e300a2d3d163ed805e55b, the following line in
# 60-drm.rules also sets ID_PATH for all pci, usb, and platform devices:
# ACTION!="remove", SUBSYSTEM=="drm", SUBSYSTEMS=="pci|usb|platform", IMPORT{builtin}="path_id"
# Unfortunately, some existing rules already rely on the unexpected behavior.
# To keep the backward compatibility, let's set ID_PATH for them.
SUBSYSTEM=="pci|usb|platform", IMPORT{builtin}="path_id"
ACTION!="add", GOTO="default_end"
SUBSYSTEM=="tty", KERNEL=="ptmx", GROUP="tty", MODE="0666"
SUBSYSTEM=="tty", KERNEL=="tty", GROUP="tty", MODE="0666"
SUBSYSTEM=="tty", KERNEL=="tty[0-9]*", GROUP="tty", MODE="0620"
SUBSYSTEM=="input", GROUP="input"
SUBSYSTEM=="input", KERNEL=="js[0-9]*", MODE="0664"
SUBSYSTEM=="video4linux", GROUP="video"
SUBSYSTEM=="graphics", GROUP="video"
SUBSYSTEM=="drm", KERNEL!="renderD*", GROUP="video"
SUBSYSTEM=="dvb", GROUP="video"
SUBSYSTEM=="media", GROUP="video"
SUBSYSTEM=="cec", GROUP="video"
SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="render", MODE="0660"
SUBSYSTEM=="kfd", GROUP="render", MODE="0660"
SUBSYSTEM=="sound", GROUP="audio", \
OPTIONS+="static_node=snd/seq", OPTIONS+="static_node=snd/timer"
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", MODE="0664"
...
```
- `60-drm.rules`
```sh!
# See issue in 50-udev-default.rules
ACTION!="remove", SUBSYSTEM=="drm", SUBSYSTEMS=="pci|usb|platform", IMPORT{builtin}="path_id"
# by-path
ENV{ID_PATH}=="?*", KERNEL=="card*", SYMLINK+="dri/by-path/$env{ID_PATH}-card"
ENV{ID_PATH}=="?*", KERNEL=="controlD*", SYMLINK+="dri/by-path/$env{ID_PATH}-control"
ENV{ID_PATH}=="?*", KERNEL=="renderD*", SYMLINK+="dri/by-path/$env{ID_PATH}-render"
```
Internal kernel driver use (i.e., MMIO, PCIe BAR, interrupt handlers, DMA transactions, ... of GPU) remain inside the kernel ---- visable in the core only.
> In other words, merely high-level control/command interfaces are exposed to the user space.
The actual GPU device driver data structure, for example:
- [open-gpu-kernel-modules/kernel-open/nvidia/nv-pci.c](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/30e15d79de62e8955eb8b77a6e292ab9b87f52b0/kernel-open/nvidia/nv-pci.c#L1320)
```c=
struct pci_driver nv_pci_driver = {
.name = MODULE_NAME,
.id_table = nv_pci_table,
.probe = nv_pci_probe,
.remove = nv_pci_remove,
.shutdown = nv_pci_shutdown,
#if defined(NV_USE_VFIO_PCI_CORE) && \
defined(NV_PCI_DRIVER_HAS_DRIVER_MANAGED_DMA)
.driver_managed_dma = NV_TRUE,
#endif
#if defined(CONFIG_PM)
.driver.pm = &nv_pm_ops,
#endif
};
```
More:
- FB Device drivers vs. DRM drivers: [3D Graphics Driver for Linux: DRM Implementation - Ignitarium](https://ignitarium.com/3d-graphics-driver-for-linux-drm-implementation/).
:::
:::info
**Integrating GPU Programs to Modern Computer System Architectures**
Like a normal CPU program running on the kernel-shell computer architecture, the user program would need user runtime support (packaged as a library) to communicate with the kernel, & the kernel would need drivers to communicate with the hardwares seamlessly:
- **User-Space Runtime Support:** Provides & exposes supported graphics APIs to user apps.
> E.g., *OpenGL*, *Vulkan*, *Direct3D*, *Metal*, ...
- **In-Kernel GPU Drivers:** Handles & manages user-space requests & communication between apps & GPU.
> E.g., Implementing DRM interface for resource & display management on Linux.
The Implementation may Be Varied due to Different Computer System Architectures -- for example, on Windows the graphics & GUI functions lie inside the kernel, whereas on Linux GPU functions lie on top of the kernel.
> **Summary:** The graphics system engineering should conform to the architectural design of the target systems & applications.
:::
:::warning
**[Extra] [Nvidia Proprietary Drivers](https://github.com/NVIDIA/open-gpu-kernel-modules/tree/main)**
They have their ~~close-source~~ in-kernel `nvidia-drm` implementation which is dedicatedly designed for their close-source user-space `libGL-nvidia-glx` graphics API implementation *(not Mesa 3D!)*, leading to compatibility & integration issues on GNU/Linux for Nvidia hardwares.
More:
- [nvdia gpu 开源驱动分析 - Joy Xu](https://joyxu.github.io/2022/05/16/gpu-nvdia-open-source/)
:::
:::info
**[Extra] Unofficial Open-Source User Graphics Driver for Nvidia Propretary Kernel Drivers**
Since the Nvidia user-space graphics driver (i.e., `libGL-nvidia-glx`) has not been open-source yet, Mesa 3D has developed an open-source user-space Vulkan driver aiming to communicate with Nvidia kernel driver (not open-source `nouveau`).
[NVK - Mesa 3D](https://docs.mesa3d.org/drivers/nvk.html)
:::
#### KMS (Kernel Mode Setting)
*An In-Kernel Device Driver for [Video Display Controller](https://en.wikipedia.org/wiki/Video_display_controller)*
:::info
KMS is part of the DRM interface.
:::
KMS takes care of those critical low-level graphics tasks that are better for ==kernel controls== rather than ++user space windowing systems++, such as:
- **Display Initialization at Boot Time**
Determine the proper *resolution* & *color depth* at the boot time, allowing for smoother transitions between booting different stages (e.g., from `Bootloader` to `GUI`'s login screen).
- **Switching Virtual Terminals (VTs)**
In the absence of the ++windowing system++ management, the ++kernel++ can handle the display mode, reducing the delay and flickering that may occur when switching between graphical & text modes in `console` or `tty`.
- **Screen Scanout, Framebuffer Swap, & VSync Control**
DRM KMS driver schedules the pageflip at the VBlank event.
> **KMS is designed for Graphics Rendering Stability, Performance, & Power Efficiency**
> With KMS, the kernel directly takes over the display hardware, providing better performance, stability, or power savings depending on various use cases, especially for tasks that require tight control over the display (e.g., *gaming*, *video playback*).
More:
- [Kernel Mode Setting - Direct Rendering Manager - Wiki](https://en.wikipedia.org/wiki/Direct_Rendering_Manager#Kernel_mode_setting)
- [Kernel Mode Setting (KMS) - Linux Kernel Docs](https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#kernel-mode-setting-kms)
### ALSA (Advanced Linux Sound Architecture)
*In-Kernel Direct-Access Audio Driver Interface*
:::success
*ALSA to audio* resembles *DRM to graphics*.
:::
1. Interacts with raw audio drivers (e.g., `/dev/snd/hw*`) for *raw audio streaming*, *playback controlling*, ...
2. Wraps & exposes character device nodes of high-level control interfaces, streaming interfaces, mixers, MIDI functionalities, ... to the users at:
```sh!
/dev/snd/pcm* # PCM audio streaming interfaces
/dev/snd/control* # control interfaces: volume, routing, device settings, codec configuration, ...
/dev/snd/seq # MIDI client sequencer interface
/dev/snd/timer # sync the sequencer time in realtime
```
> Users can interact with those via `aplay` / `arecord` applications, or with `portaudio`, `pulseaudio`, ... libraries to access the ALSA API.
> Linux audio stack:
> 
>
> (Source: [ALSA overview - STM32 MPU - ST Wiki](https://wiki.st.com/stm32mpu/wiki/ALSA_overview))
#### alsamixer
```sh
alsamixer
```
### evdev (Event Device)
*In-Kernel Direc-Access Input Event Interface*
:::success
*evdev to input* resembles *DRM to graphics*.
:::
1. Interacts with raw device drivers (e.g., `/dev/input/mouse0`, `/dev/usb/hiddev0`) for *raw input event handling*, *device configuration*, ...
2. Wraps & exposes character device nodes of high-level events ... to the users at:
```sh!
/dev/input/event*
```
> Users can access & interact those with `libevdev` library.
- **libevdev**
*Exposes evdev API to the User Space*
:::success
Like user-space graphics drivers to the in-kernel DRM!
:::
`libevdev` abstracts evdev `ioctl` through type-safe interfaces and provides functions to change the appearance of the device. It shares similarities with the `read` syscall.
:::warning
**[GUI Systems] Interaction with User-Space Display Servers**
*evdev* is primarily used by *display servers* like ++X/Xserver++ & ++Wayland/Weston++, as well as by games and console emulators making use of USB and Bluetooth controllers:
- **On X Server**:
```!
Kernel -> libevdev -> libinput -> xf86-input-libinput -> X server -> X client
```
- **On Weston Compositor**:
```!
Kernel -> libevdev -> libinput -> Wayland Compositor -> Wayland Client
```
:::
### D-Bus (Desktop Bus)
*IPC System for Desktop Environments (e.g., GNOME)*
> D-Bus supports ++service discovery++, ++method call transactions++, ++signals++, ++broadcasting++, ... across different user processes.
:::success
An RPC-style user daemon on UNIX-domain socket (UDS) -- a user-space message queue middleware as microkernel-architecture high-level IPC.
:::
More:
- [D-Bus and Polkit, No More Mysticism and Confusion - Venam's Blog](https://venam.net/blog/unix/2020/07/06/dbus-polkit.html)
- [Cross-platform communication with dbus - Nick Tune](https://medium.com/nick-tune-tech-strategy-blog/cross-platform-communication-with-dbus-8798e746071b)
- [D-Bus Tutorial - RM5248](https://rm5248.com/d-bus-tutorial/)
:::warning
**[IPC System Comparison] Dbus vs. gRPC**
- **Dbus:** absracts "Objects" & "Types" (i.e., it can do reflection or some high-level things) of data sending across a bus-based IPC connection through a central broker.
- **gRPC:** focuses more on end-to-end marshaling / unmarshaling data (but gRPC can be used to build pub/sub bus systems, streaming applications, apparently).
[gRPC vs DBUS? - groups.google](https://groups.google.com/g/grpc-io/c/v3kLTEKYCcQ)
:::
:::info
**[IPC System Implementations] Kernel Message Queues vs. UNIX-Domain Sockets vs. Loopback Networking vs. Character Devices**
- **Kernel Message Queues (System V, POSIX) -- Limited Platform-Specific OS Modules:**
> **Why not in-kernel message queues?**
>
> D-Bus has more fine-grained, secured security policy control, high-level system support, & portability compared to platform-specific kernel message queues which are limited to apply to.
- **Unix Domain Socket (`AF_UNIX`) -- High-Level User-Space Daemon Message Queue Architecture**
*Simpler, securer & faster than loopback sockets (i.e., localhost)*
Using shared memory on the filesystem (`tmpfs`) as the transport medium rather the network stack.
> 
>
> (Source: [Julia Evans](https://wizardzines.com/comics/unix-domain-sockets/))
:::warning
**[What are UNIX-Domain Sockets under `/tmp/.X11-unix`?](https://unix.stackexchange.com/questions/196677/what-is-tmp-x11-unix)**
The X server use UNIX-domain sockets (as the IPC system) to communicate with clients like `xterm`, `firefox`, etc. via some kind of reliable stream of bytes.
Use *`lsof` (list opened files)* to inspect running sockets:
```sh
lsof -U # UNIX-domain sockets
```
More:
- [Is `/tmp` the right place for UNIX-domain socket files? - StackOverflow](https://stackoverflow.com/questions/43124765/is-tmp-the-right-place-for-unix-domain-socket-files)
:::
- **Lagacy D-Bus:**
- **kdbus (`/dev/kdbus`) -- Low-Level (Device-Focused) Kernel-Space Daemon Message Queue Architecture**
*A character device, low-level, native kernel D-Bus transport layer*
kdbus is a transport layer for the D-Bus IPC system. It is meant as replacement for the UDS (Unix Domain Socket) transport layer which is the de-facto default for local D-Bus communication.
> kdbus is not directly related to D-Bus, nor is it limited to transporting D-Bus messages. However, kdbus was designed specifically for D-Bus so it does exhibit D-Bus specific behavior.
- **D-Bus with In-Kernel kdbus & SMACK (Privilege Separation / Permission Check):**
More:
- [kdbus - freedesktop.org](https://www.freedesktop.org/wiki/Software/systemd/kdbus/)
:::
#### Components
- **Buses (IPC of Desktop Environment):**
- **System Bus:**
Available to all users and processes of the system, that provides access to system services (i.e. services provided by the operating system and also by any system daemons).
- **Session Bus(es):**
For each user login session, that provides desktop services to user applications in the same desktop session, and allows the integration of the desktop session as a whole.
- **udisks (Mount of Desktop Environment):**
A daemon, `udisksd`, that implements well-defined D-Bus interfaces that can be used to query and manipulate storage devices.
- **polkit (SUID of Desktop Environment):**
Provides an authorization API intended to be used by privileged programs (*mechanisms*) offering service to unprivileged programs (*clients*).
> 
>
> (Source: [polkit Reference Manual - freedesktop.org](https://polkit.pages.freedesktop.org/polkit/polkit.8.html))
## Miscellaneous Topics
### Hardware Graphics Acceleration
*Multimedia parallel processing computing units*
:::warning
**Hardware IMR vs. TBR (Tile-Based Rendering) vs. TBDR (Tile-Based Deferred Rendering)**
[PowerVR GPU架构与优化建议 - Yemi](https://yemi.me/2018/09/17/powervr-architecture-overview/)
:::
:::info
:arrow_right: For more on CPU / coprocessor details, please check out: [Assembly - shibarashinu](https://hackmd.io/@shibarashinu/rkIVYJZmA)
:::
### Windows Graphics Architecture
#### DirectX on Windows
> 
>
> (Source: [Overview of the Windows Graphics Architecture - Microsoft Learn](https://learn.microsoft.com/en-us/windows/win32/learnwin32/overview-of-the-windows-graphics-architecture))
Sign in with Wallet
Connect another wallet