QMK Firmware XAP Specs

Comments made may disappear – they're not removed – you can turn on hidden comments in the settings on the top right which should restore them.

Doc generation is now handled from code:

XAP defs + output

QMK Firmware XAP Specs

This document aims to describe the requirements of the new QMK XAP ("extensible application protocol") API. The intention is to provide access to everything currently exposed through the VIA interface, as well as additional subsystems that VIA does not support.

Ultimately the intent is that XAP replaces the current VIA protocol, and the VIA host app (as well as others such as Vial, remap.keys) can leverage XAP going forward. Once sufficient critical mass is reached, support for the VIA protocol can be removed from QMK Firmware.

Comments by @tzarc look like this:

…and should not be considered as part of the final document.

Types

see Types

Definitions

see Definitions

Key requirements

Superset of VIA functionality
Once an externally-visible ID is present in the master branch, it can never be changed, only marked as deprecated
- Deprecation ensures the "reordering" issue previously struck with keycode numbers does not occur
XAP defines the protocol definitions only, and is transport-layer independent
- RAW HID implementation likely to be initially provided
  - Pick another usage page and usage ID for XAP
- Future support may be provided for VIRTSER/SysEx etc.
Some routes may be marked as secure, requiring an unlock sequence to be initiated by the user before allowing execution
- Secure routes may be executed without an unlock sequence at the user's decision, through a #define in their user/keymap config.h.
- At no point should QMK accept a PR with the insecure flag enabled in base keyboard or default keymap.
- The unlock sequence should automatically timeout after a period of inactivity with XAP.
  - Default to 90000ms.
  - Allow override through #define in user/keymap config.h
  - Also disallow in keyboard config.h

High-level Suggestions

Protocol document to be defined using hjson
- Allows for comments and matches existing usage within QMK
- Allows for schema validation
All IDs need to be defined in the protocol document
- Includes things such as:
  - Subsystem IDs
  - Keycodes
  - RGB animation IDs
  - etc. etc.
All IDs should be exported to header file during build, much like info.json etc.
Strongly suggested that routing is handled through codegen.
Possibly look at embedding a compressed copy of info.json into the firmware, to ensure valid information at the time of build is available.
- Requires proper info.json to be present for all builds including XAP.
- Alternatively, provide only the things that matter for the current build – key positions and the like.

Requests and Responses

see Requests and Responses

Example "conversation":

see Example conversation

Broadcast messages

see Broadcast messages

XAP Versions

0.0.1: Initial XAP version. Intentionally only includes XAP subsystem, with only the version query route. Can be used as a fallback to ensure version query can succeed.
0.1.0: Fleshes out the rest of the "base" XAP functionality – adds unlock capability, as well as subsystem query, QMK, keyboard, and user subsystems.
1.0.0: Currently undefined – should include Dynamic Keymap, Encoder, and lighting configuration.

Subsystems

This list is not "locked in" and is non-exhaustive – it is present mainly to cater for initial feature parity.

0x00: XAP (always present, introduced XAP version 0.0.1)
0x01: QMK (always present, introduced XAP version 0.1.0)
0x02: Keyboard-specific passthrough (introduced XAP version 0.1.0) – xap_handler_kb()
0x03: User-specific passthrough (introduced XAP version 0.1.0) – xap_handler_user()
0x04: Dynamic Keymaps
0x05: Dynamic Encoders
0x06: rgblight
0x07: rgb_matrix
Others TBD

Subsystem validity should be queried through the "Enabled-in-firmware subsystem query" under the QMK subsystem (route=0x00,0x01). This is the primary method for determining if a subsystem has been enabled in the running firmware. Example pseudo-C-code running on the host application side:

// elsewhere...
#define XAP_SUBSYSTEM_ID 0x00
#define XAP_SUBSYSTEM_ENABLED_SUBSYSTEMS_QUERY_ID 0x01

// check if subsystem is enabled
bool is_subsystem_enabled(uint8_t subsystem) {
    uint8_t query[] = { XAP_SUBSYSTEM_ID, XAP_SUBSYSTEM_ENABLED_SUBSYSTEMS_QUERY_ID };
    uint8_t response[32] = {0};
    xap_execute(query, sizeof(query), response, sizeof(response));
    return (response[subsystem / 8] & (1 << (subsystem % 8))) ? true : false;
}

// elsewhere...
#define RGB_MATRIX_SUBSYSTEM_ID 0x07
bool is_rgb_matrix_enabled = is_subsystem_enabled(RGB_MATRIX_SUBSYSTEM_ID);

Currently unsure if we want to make a distinction between rgblight and rgb_matrix, or whether or not we just provide a flag. There's considerable overlap between the two subsystems, but also enough difference that it could go either way. Should we just call it "lighting", and add support for backlight too?

The addition of dynamic encoders is subject to decisions regarding how to persist their functionality in other in-flight PRs. If they end up being part of the matrix, then we may not need another subsystem.

XAP Subsystem `0x00` (always present, introduced XAP version 0.0.1)

This subsystem is always present if XAP is enabled, and provides the ability to query information about the XAP protocol of the connected device.

Routes

0x00: XAP protocol version (always present, introduced XAP version 0.0.1)
- Returns the BCD-encoded version in the format of XX.YY.ZZZZ => 0xXXYYZZZZ
  - e.g. 1.1.15 will match 0x01010015.
- Response:
  - u32 value.
0x01: Enabled-in-firmware subsystem query (always present, introduced XAP version 0.1.0)
- Allows for querying which top-level subsystem routes are valid within the running firmware.
- Response:
  - u8[16] = 128 bits = 128 total subsystems.
  - Each bit is the equivalent of (1ul << SUBSYSTEM_ID).
  - For example, a firmware with dynamic keymaps(0x04), and rgb_matrix(0x07) will have the first entry in the array of 0b10011111. (Note: XAP, QMK, keyboard, and user, will always be present.)
- Any matching bit in the response signifies that there is a corresponding handler able to receive a request made to that subsystem.
- e.g.
0x02: Query secure route status (always present, introduced XAP version 0.1.0)
- Allows checking if secure routes are allowed.
- Response:
  - u8:
    - 0 means secure routes are disabled
    - 1 means unlock sequence initiated but incomplete
    - 2 means secure routes are allowed
    - any other value should be interpreted as disabled
0x03: Initiate secure route unlock sequence (always present, introduced XAP version 0.1.0)
- Tells the firmware that the unlocking sequence should be performed by the user.
- Normal keypresses should be disabled until the sequence is completed, or fails through pressing the wrong combination, or through timeout.
- Host apps should query secure route status to determine when the unlock sequence has completed.
- All other subsystems are disabled during unlock.
- Response flags are used to determine success/fail.
0x04: Disable secure routes (always present, introduced XAP version 0.1.0)
- Tells the firmware that any previous unlock sequence should be disregarded, thereby preventing secure routes from executing until another unlock sequence is completed.
- This also serves as a "cancel" for the unlocking sequence.
- Response:
  - u8: 1 means secure routes are disabled, any other value signifies failure. If secure routes were already disabled, a 1 will be returned.

QMK Subsystem `0x01` (always present, introduced XAP version 0.1.0)

This subsystem is always present, and provides routes that are specific to QMK, but not tied to any specific subsystem within QMK, such as jumping to bootloader.

Routes

0x00: QMK subsystem capabilties query (always present, introduced XAP version 0.1.0)
- Allows for querying which QMK subsystem functionality is present.
- Capabilities (QMK_CAPABILITY_ID):
  - 0x01: QMK version (always present)
  - 0x02: Keyboard VID/PID/UniqueID (always present)
  - 0x03: Jump to bootloader (secure)
- Response:
  - u8[8] = 64 bits
  - Each bit is the equivalent of (1ul << QMK_CAPABILITY_ID).
  - For example, a firmware with base keyboard functionality will have the first entry in the array of 0b00011111.
0x01: QMK version (always present, introduced XAP version 0.1.0)
- Returns the BCD-encoded version in the format of XX.YY.ZZZZ => 0xXXYYZZZZ
  - e.g. 0.10.135 will match 0x00100135.
- Response:
  - u32
0x02: Keyboard VID/PID/UniqueID/UserID (always present, introduced XAP version 0.1.0)
- Contains the USB VID:PID pair, as well as a UniqueID defined within QMK, and a UserID for determining if a specific user has had their userspace loaded.
- Allows for identification of the connected keyboard, as opposed to QMK version.
- Response:
  - ```
  struct {
      u16 VendorID;
      u16 ProductID;
      u64 UniqueID; // Format TBD, see below.
      u16 UserID;
  }
```
- UniqueID allows for identifying a specific keyboard without relying on the VID:PID pair.
  - Alleviates the need for finding a unique pairing of VID:PID
  - Needs to be deterministic at time of build, and not dependent on build time or environment setup
  - Format is currently TBD
  - Do we need to go to 128-bit? Or do we drop down, is 32-bit enough?
  - Perhaps something like:
```
% f=$(echo -n planck/rev6 | sha256sum)
% echo ${f:0:8} # 32-bit
c654a00b
% echo ${f:0:16} # 64-bit
c654a00b59e9b67e
% echo ${f:0:32} # 128-bit
c654a00b59e9b67e0c433aaca7437f37
```
- UserID allows for users to define an identifier that can be queried to determine if their firmware is loaded.
  - For example, @tzarc could add #define XAP_USER_ID 0x1337 to his keymap's config.h
  - If the UserID matches, then you know the User-specific Subsystem 0x03 below can accept data associated with @tzarc's handlers.
0x03: Jump to bootloader (secure, introduced XAP version 0.1.0)
- May not be present – if QMK capabilities query returns "true", then jump to bootloader is supported
- Secure operation – will fail if XAP secure routes are not allowed
- Response:
  - u8:
    - 0 means secure routes are disabled, and should be considered as a failure
    - 1 means successful, board will jump to bootloader

Keyboard-specific Subsystem `0x02`

Any keyboard-specific XAP functionality is to be implemented under this route.
No restrictions on payload data or format.
Invokes xap_receive_kb().

User-specific Subsystem `0x03`

Any user-specific XAP functionality is to be implemented under this route.
No restrictions on payload data or format.
Invokes xap_receive_user().

David Hoelscher

2021/06/17 00:23:39

>> Curren

I'm wondering if it might be useful to include "kb" and possibly "user" subsystems. This would allow XAP to support implementation of interfaces not contemplated or broadly by QMK. This would require custom software on the computer to use this interface, but since tools like VIAL are open source, it could be useful. (Edited)

2021/06/17 00:28:19

In any case, whether kb or user routes are supported or not, it would be helpful if there is some mechanism to allow software on the computer to communicate which OS is being used. We've had a request for keymaps that dynamically adapts to the OS. While that exists in a few commercial keyboards, I didn't find a great way to do it in QMK (yet) even assuming some simple custom software is installed on the computer to support the feature.

Nick Brassel

2021/06/17 01:47:15

These are to be included, yes.

2021/06/17 21:20:13

Others TBD

Yep, I wouldn't be surprised if something like this ends up getting added along the way. Might not be part of first drop but should be relatively simple to add afterwards, so long as the foundational stuff is properly designed. (Edited)

Yoichiro Tanaka

2021/06/18 00:28:43

#### Routes * `0x00`: XAP protocol version (**always present**, introduced XAP version 0.0.1) * Returns the BCD-encoded version in the format of XX.YY.ZZZZ => `0xXXYYZZZZ`

Should we define the API to retrieve the QMK Firmware version number to this QMK Subsystem (0x00)? Or, does this QMK Subsystem set have an API set related to the XAP only? (Edited)

2021/06/18 00:33:42

As my opinion, we should not define an API set related to the QMK Firmware in this QMK Subsystem (0x00), and it would be better to define an API set which handles for XAP only and would be better to define another API set handling some QMK firmware features. (Edited)

2021/06/18 02:31:05

The intention along those lines was that XAP would be the "exposed" interface and its version would be decoupled from the QMK version itself. This way the host-side applications would be able to rely on a specific feature set related to that XAP version. Any changes to the exposed interface would need to ensure backwards compatibility -- older versions of tools will naturally continue to work with newer firmwares, and newer tools then can to do their own versioning based on what's returned.

2021/06/18 02:32:10

I've gone ahead and reworked the definitions to replace QMK with XAP, for clarity. (Edited)

2021/08/17 02:13:24

Comments made may disappear -- they're not removed -- you can turn on hidden comments in the settings on the top right which should restore the

Is there a place to discuss this casually? If there is such place (ex. Discord channel, Slack channel and so on), I guess that we can provide more ideas and progress writing this spec. (Edited)

2021/08/29 03:59:57

#qmk_firmware on Discord: https://discord.gg/Uq7gcHh

2021/09/14 21:40:10

Ok, thanks.

QMK Firmware XAP Specs

Comments made may disappear – they're not removed – you can turn on hidden comments in the settings on the top right which should restore them.

Doc generation is now handled from code:

QMK Firmware XAP Specs

Types

Definitions

Key requirements

High-level Suggestions

Requests and Responses

Example "conversation":

Broadcast messages

XAP Versions

Subsystems

XAP Subsystem 0x00 (always present, introduced XAP version 0.0.1)

Routes

QMK Subsystem 0x01 (always present, introduced XAP version 0.1.0)

Routes

Keyboard-specific Subsystem 0x02

User-specific Subsystem 0x03

Read more

tz's QMK todo list

XAP Subsystem `0x00` (always present, introduced XAP version 0.0.1)

QMK Subsystem `0x01` (always present, introduced XAP version 0.1.0)

Keyboard-specific Subsystem `0x02`

User-specific Subsystem `0x03`