# Study Notes for NVMe
2020/08/04

### 1. NVMe Queue
In NVMe base spec. revision 1.4 (hereinafter referred to as spec. 1.4) page 7:

Two types of queues:
**Where each type has:**
*Submission queue*
In spec. 1.4 page 8:
> A Submission Queue (SQ) is a <font color=red>circular buffer</font> with a <font color=red>*fixed slot size*</font> that the host software uses to submit commands for execution by the controller.
>

The submission queue byte layout is as show below:
Sumbission Queue Entry Byte Layout (Reference from spec. 1.4 in page 65-66, figure form https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2012/20120821_TD12_Onufryk.pdf)

spec. 1.4 in page 65
>Each command is 64 bytes in size.
>

*Completion queue*
In spec. 1.4 page 9:
> A Completion Queue (CQ) is a <font color=red>circular buffer</font> with a <font color=red>*fixed slot size*</font> used to post status for completed commands.
>
An entry in the Completion Queue is at least 16 bytes in size.

The completion queue byte layout is as show below:


Completion Queue Entry Byte Layout (Reference from spec. 1.4 in page 77, figure form https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2012/20120821_TD12_Onufryk.pdf)

spec. 1.4 in page 77
>An entry in the Completion Queue is at least 16 bytes in size.
>

#### 1. Admin queues
Admin Command Set may be submitted to the Admin Submission Queue.

#### 2. I/O queues

The NVM Express interface is based on a **paired** <font color=red>**Submission**</font> and <font color=red>**Completion**</font> Queue mechanism.

#### 3. Confused Issues:
In spec. 1.4 page 7:
Does not require uncacheable / MMIO register reads in the command submission or completion path;

In spec. 1.4 page 284 Figure 432: Command Processing
Step 3. The controller transfers the command(s) from in the Submission Queue slot(s) into the controller for future execution.
This operation is not a reading opeartion? or what does controller transfering mean?

*What is the slot? or slot size?*
*How many bytes of one logic block size?* (512 Bytes ??) namespace LBAF

*What contents does exatly the Metadata store in? (an address)*
It depends on the PRP or SGL entry?

### 2. Why are there two different lists, the Physical Region Page Entry and List (PRP)and Scatter Gather List (SGL)?
In spec. 1.4 page 66:
PRPs shall be used for all <font color=red>**Admin commands**</font> for NVMe over <font color=red>**PCIe**</font> implementations.

SGLs shall be used for all Admin and <font color=red>**I/O commands**</font> for NVMe over <font color=red>**Fabrics**</font> implementations. (Just only Fabrics implementation ???)

### 3. Physical Region Page Entry and List (PRP)
#### 1. PRP
In spec. 1.4 page 68:
A physical region page (PRP) entry is a pointer to a physical memory page.
<font color=red>*where the physical memory page size defined in spec. 1.4 page 48*</font>

Offset 14h: CC Controller Configuration
10:07 Memory Page Size(MPS):
The minimum host memory page size is **4 KiB** and the maximum host memory page size is **128 MiB**.

#### 2. PRP List
In spec. 1.4 page 69:
A physical region page list (PRP List) is a set of PRP entries in a single page of contiguous memory.

### 4. Scatter Gather List (SGL)
In spec. 1.4 page 70:
A Scatter Gather List (SGL) is a data structure in memory address space used to describe a data buffer.

<font color=red>*What is the definition for data buffer in NVMe spec. ?*</font>

### 5. Command Set

#### 1. Admin Command Set
To reference the spec. 1.4 chapter 5 "ADMIN COMMAND SET".

#### 2. Confused Issues:
Admin and NVM Command Set, Admin and NVM Vendor Specific Commands..
how to determine??

--> In spec. 1.4 page 94, figure 139. Are that commands determined by combined opcode C0h to FFh ??

#### 3. NVM Command Set
To reference the spec. 1.4 chapter 6 "NVM COMMAND SET".

### 6. Controller Architecture
In spec. 1.4 page 274:
There types of controller: I/O, Administrative, Discovery.
#### 1. I/O Controller
In spec. 1.4 page 275:
An I/O controller is a general purpose controller that supports commands that provide access to an NVM subsystem’s non-volatile storage medium and may support commands that provide management capabilities.

#### 2. Administrative Controller
In spec. 1.4 page 278:
An administrative controller is a controller whose intended purpose is to provide NVM subsystem management capabilities.

#### 3. Discovery Controller
In spec. 1.4 page 283:
A discovery controller is a special type of controller used in NVMe over Fabrics to provide access to a Discovery Log Page.

#### 4. Command Submission and Completion Mechanism
A picture is worth a thousand words. The command processing flow is as show as below (in Spec. 1.4 page 284):

##### 1. Confused issues:
In step 5.
Phase Tag <font color=red>*inverted*</font> from the previous entry to indicate to the host that this completion queue entry is a new entry.
The term inverted does mean the toggle behavior? (0 to 1, 1 to 0)

In step 6.
The controller <font color=red>*optionally*</font> generates an interrupt to the host...
The term optionally does mean that feature not must be done??

#### 5. Fused Operation
When the controller would ececute the fused operation ?? (Atomic ?? When)

#### 6. Resets

##### 1. NVM Subsystem Reset
In spec. 1.4 page 290:
A value of 4E564D65h (“NVMe”) is written to the NSSR.NSSRC field;
in spec. 1.4 page 42:
As shown in Figure 68: Register Definition
NSSR (NVM Subsystem Reset)
in spec. 1.4 page 50
As shown in Figure 80: NSSRC - NVM Subsystem Reset Control

##### 2. Controller Level Reset
In spec. 1.4 page 290:
There are five methods to initiate a Controller Level Reset:
• NVM Subsystem Reset;
• Conventional Reset (i.e., PCI Express Hot, Warm, or Cold reset);
• PCI Express transaction layer Data Link Down status;
• Function Level Reset (i.e., PCI reset); and
• Controller Reset (i.e., CC.EN transitions from ‘1’ to ‘0’).

##### 3. Queue Level
In spec. 1.4 page 291:
A queue level reset is performed by deleting and then recreating the queue.

#### 7. Controller Initialization
The detail could be referenced in spec. 1.4 page 295-296, but we may be summarize the actions as below:
1. Set the PCI and PCI Express register.
2. Waiting for CSTS.RDY to become '0'.
3. Set the AQA, ASQ, and ACQ values.
4. Arbitraion mechanism --> CC.AMS
memory page size --> CC.MPS
I/O Command Set --> CC.CSS
6. CC.EN to '1'
7. CSTS.RDY to '1'