---
tags: David, OpenSSD, Cosmos+, firmware
---
# Introduce to Cosmos+ OpenSSD Platform Firmware
## Abbreviations
### NVMe
- `CPL`: Complete or Completion
### Firmware
- `Fb`: Free Block
- `Llr`: Low-level Scheduler
- `LSA`: Logical Slice Address
- `Q`: queue
### Storage
- `nsc`: NAND Storage Controller
- ??`V2F`: ==Virtual/Vector== to Flash
- ??`V2FMCRegisters`: ==Virtual/Vector== to Flash Memory Controller Registers
## DRAM Structure
```
+-------------------------------------------------+ << 0
~ ~
~ ~
+-------------------------------------------------+ <<
+-------------------------------------------------+ <<
+-------------------------------------------------+ <<
+-------------------------------------------------+ <<
+-------------------------------------------------+
~ ~
~ ~
+-------------------------------------------------+ << COMPLETE_FLAG_TABLE_ADDR
+-------------------------------------------------+ << STATUS_REPORT_TABLE_ADDR
+-------------------------------------------------+ << ERROR_INFO_TABLE_ADDR
+-------------------------------------------------+ << TEMPORARY_PAY_LOAD_ADDR
+-------------------------------------------------+
| |
~ ~
~ ~
| |
+-------------------------------------------------+ << DATA_BUFFER_MAP_ADDR
| |
| dataBuf[NUM_OF_DATA_BUFFER_ENTRY] |
| |
+-------------------------------------------------+ << DATA_BUFFFER_HASH_TABLE_ADDR
| |
| dataBufHash[NUM_OF_DATA_BUFFER_ENTRY] |
| |
+-------------------------------------------------+ << TEMPORARY_DATA_BUFFER_MAP_ADDR
| |
| tempDataBuf[NUM_OF_TEMPORARY_DATA_BUFFER_ENTRY] |
| |
+-------------------------------------------------+ << LOGICAL_SLICE_MAP_ADDR
+-------------------------------------------------+ << VIRTUAL_SLICE_MAP_ADDR
+-------------------------------------------------+ << VIRTUAL_BLOCK_MAP_ADDR
+-------------------------------------------------+ << PHY_BLOCK_MAP_ADDR
+-------------------------------------------------+ << BAD_BLOCK_TABLE_INFO_MAP_ADDR
+-------------------------------------------------+ << VIRTUAL_DIE_MAP_ADDR
+-------------------------------------------------+ << GC_VICTIM_MAP_ADDR
+-------------------------------------------------+ << REQ_POOL_ADDR
+-------------------------------------------------+ << ROW_ADDR_DEPENDENCY_TABLE_ADDR
+-------------------------------------------------+ << DIE_STATE_TABLE_ADDR
+-------------------------------------------------+ << RETRY_LIMIT_TABLE_ADDR
+-------------------------------------------------+ << WAY_PRIORITY_TABLE_ADDR
+-------------------------------------------------+ << FTL_MANAGEMENT_END_ADDR
+-------------------------------------------------+ << RESERVED1_START_ADDR
| |
~ ~
~ ~
| |
+-------------------------------------------------+ << 1 GB
```
## Important Structures
### Data Buffer
- `DATA_BUF_MAP` / `dataBuf`: An 1D data buffer array with `AVAILABLE_DATA_BUFFER_ENTRY_COUNT` entries.
There are N(default 16) x `USER_DIES` entries in this array. Each of the entry is represented by the data structure `DATA_BUF_ENTRY` which is consist of several members:
- `logicalSliceAddr`: 32 bits unsigned integer, the logical address of the slice command
- `dirty`: 1 bit flag, whether this entry is dirty or not. 0 (`DATA_BUF_CLEAN`) for clean, 1 for dirty.
- `prevEntry`: 16 bits unsigned integer, the prev data buffer entry index in the LRU list.
- `nextEntry`: 16 bits unsigned integer, the next data buffer entry index in the LRU list.
- `hashPrevEntry`: 16 bits unsigned integer, the prev data buffer entry index in the current `dataBufHash` bucket
- `hashNextEntry`: 16 bits unsigned integer, the next data buffer entry index in the current `dataBufHash` bucket
- `blockingReqTail`: 16 bits unsigned integer, the data buffer entry index of the last blocking request
At the begining, the `prevEntry` and `nextEntry` just simply initialized to the index of prev/next element in this array. But as the time goes by, some entries may be chosen to store some data, //TODO
This table is used for maintaining the relation between entries in LRU list and data buffer bucket. //TODO
- `DATA_BUF_LRU_LIST` / `dataBufLruList`: A structure that manages the MRU and LRU data buffer entry.
This structure used for choosing the victim data buffer entry.
There are only two members in this structure:
- `headEntry`: 16 bits unsigned integer, the index of MRU data buffer entry. (first data buffer entry by default).
- `tailEntry`: 16 bits unsigned integer, the index of LRU data buffer entry. (last data buffer entry by default).
Note that because this structure only records the index of MRU/LRU data buffer entry, the relation between data buffer entries is managed by the `prevEntry` and `nextEntry` of `dataBuf` described above.
When a LRU entry is chosen to be the victim entry, the previous entry indicated by `prevEntry` and the next entry indicated by the `nextEntry` should be modified as well, concretely:
- evicted.prev: should be `DATA_BUF_NONE`, because this node is the new head of LRU list
- evicted.next: should be the index of original head of LRU list
- evicted.prev.next: if evicted.prev exists, its next should be `DATA_BUF_NONE`, because it's now the new LRU entry
- evicted.next.prev: should be the index of evicted entry
The function for choosing the victim entry is `AllocateDataBuf()` defined in `data_buffer.c`.
- `DATA_BUF_HASH_TABLE` / `dataBufHash`:
The structure `DATA_BUF_HASH_ENTRY` is consist of the following two members:
- `headEntry`: 16 bits unsigned integer that indicates the first data buffer entry in this bucket
- `tailEntry`: 16 bits unsigned integer that indicates the last data buffer entry in this bucket
Because this structure only record the head and tail index, we should keep the relation between the data buffer entries in corresponding data buffer bucket by maintain the `hashPrevEntry` and `hashNextEntry` of those data buffer entries.
// TODO
just simply the `logicalSliceAddr` % the number of hash buckets:
```c
#define FindDataBufHashTableEntry(logicalSliceAddr) ((logicalSliceAddr) % AVAILABLE_DATA_BUFFER_ENTRY_COUNT)
```
Therefore, if we want to find the data buffer of a given request, we can just traverse the correspoding bucket, instead of traverse the whole data buffer or LRU list. (check the implementation of the function `CheckDataBufHit` defined in `data_buffer.c`)
- `TEMPORARY_DATA_BUF_MAP` / `tempDataBuf`:
There are `USER_DIES` entries in this array, namely one temp buffer entry per die, therefore just simply use the `dieNo` to choose the entry of temp data buffer, in current implementation.
Each temp buffer entry is a `TEMPORARY_DATA_BUF_ENTRY`, and this structure have only one member:
- `blockingReqTail`:
TODO: when to use this temp buffer?
### Requests
- `REQ_POOL` / `reqPool`: The request pool, a fixed-sized 1D array of requests.
There are `AVAILABLE_OUNTSTANDING_REQ_COUNT` (typo, to be fixed) entries in this array. Every entry in the request pool is a `SSD_REQ_FORMAT`, and the structure `SSD_REQ_FORMAT` is consist of:
- logicalSliceAddr: the logical address of this slice command
- reqType: 4 bits flag, the type of this request, should be one of the macros `REQ_TYPE_*` defined in `request_format.h`
- reqQueueType: 4 bits flag, the type of queue where this request current stored
- reqCode: 8 bits flag, the operation code of this request
- nvmeCmdSlotTag: 16 bits flag, <!-- TODO -->
- reqOpt;
- dataBufInfo;
- nvmeDmaInfo;
- nandInfo;
- prevReq : 16 bits unsigned integer, the request pool entry index of the prev request
- nextReq : 16 bits unsigned integer, the request pool entry index of the next request
- prevBlockingReq : 16 bits unsigned integer, the request pool entry index of the prev blocking request
- nextBlockingReq : 16 bits unsigned integer, the request pool entry index of the next blocking request
Within this firmware project, the variable `reqSlotTag` is widely used to represent a request pool index, check the `InitReqPool` function implemented in the `request_allocation.c` for example.
- `REQ_QUEUE_TYPE_*` / `*_REQUEST_QUEUE`
There are 7 macros `REQ_QUEUE_TYPE_*` and corresponding 6 queue structures `*_REQUEST_QUEUE` (no queue for `REQ_QUEUE_TYPE_NONE`). Each request in the request pool belongs one of the 6 queues.
These 6 queues has exactly same structure:
- `headReq`: 16 bits unsigned integer, the request pool index of the head request in this queue
- `tailReq`: 16 bits unsigned integer, the request pool index of the tail request in this queue
- `reqCnt`: 16 bits unsigned integer, the number of requests in this requests queue
But with different meaning and usage:
- `REQ_QUEUE_TYPE_FREE` / `FREE_REQUEST_QUEUE` / `freeReqQ`:
a
- `REQ_QUEUE_TYPE_SLICE` / `SLICE_REQUEST_QUEUE` / `sliceReqQ`:
a
- `REQ_QUEUE_TYPE_BLOCKED_BY_BUF_DEP` / `BLOCKED_BY_BUFFER_DEPENDENCY_REQUEST_QUEUE` / `blockedByBufDepReqQ`:
a
- `REQ_QUEUE_TYPE_BLOCKED_BY_ROW_ADDR_DEP` / `BLOCKED_BY_ROW_ADDR_DEPENDENCY_REQUEST_QUEUE` / `blockedByRowAddrDepReqQ`:
a
- `REQ_QUEUE_TYPE_NVME_DMA` / `NVME_DMA_REQUEST_QUEUE` / `nvmeDmaReqQ`:
a
- `REQ_QUEUE_TYPE_NAND` / `NAND_REQUEST_QUEUE` / `nandReqQ`:
The `nandReqQ` is a 2D array with `USER_CHANNELS` rows and `USER_WAYS` cols. Each element of it is a `NAND_REQUEST_QUEUE` that records the following
FIXME: simplify these queue structures
- `sliceReqQ`
- `LOGICAL_SLICE_MAP` `LOGICAL_SLICE_ENTRY`
- `VIRTUAL_SLICE_MAP` `VIRTUAL_SLICE_ENTRY`
### ss
- `BlockedByRowAddrDepReq`:
- `dieState`:
- `FbList`
- `NandXXXXList`:
- `Erase`:
- `Idle`:
- `StatusReport`:
- `Write`:
- `ReadTrigger`:
- `ReadTransfer`:
- `NandWayPriorityTable`:
## Configurations
- `XPAR_TIGER4NSC_[0-7]_BASEADDR` defined in `ftl_config.h`
Sign in with Wallet
Connect another wallet