# FPGA-accelerated TrustZone-enabled IOTA Swarm Nodes

## 動機

在 IOTA 中，一個錢包完整發出交易 (transaction) 請求會被劃分為四個命令：
* `getTransactionsToApprove`
* `attachToTangle`
* `broadcastTransactions`
* `storeTransaction`

也就是先向 Host 取得兩個 tips (trunk, branch) 交易，接著錢包會將 trunk, branch，以及要發出的 transaction trytes 以 `attachToTangle` 送到 Host，讓 Host 簽發交易 (也就是 PoW)，這步驟是發起交易最耗時的部分。接著請求 Host 將這筆交易廣播給其他鄰居，最後將這筆交易存下來。
- [ ] 延伸閱讀: [IOTA API](https://iota.readme.io/v1.3.0/reference)

針對物聯網和大規模部署的需求，我們[實作了 IOTA Swarm nodes 並允許在中低階 Arm 環境佈署](https://hackmd.io/s/SJdnjRWnz)，而在 [2018-06-13 會議記錄](https://hackmd.io/c/rkpoORY4W/%2Fs%2FSy2X3ZTgX) 提及評估透過 FPGA 加速 IOTA PoW (attachToTangle 命令) 並期望在 [Arrow SoCKit Evaluation Board](https://rocketboards.org/foswiki/Documentation/ArrowSoCKitEvaluationBoard) 建立兼具 FPGA 加速和 Arm TrustZone 安全隔離的系統雛形。
- [ ] 未來展望
1. 接受運算能力低的感測器轉包 PoW 到硬體加速的 IOTA Swarm nodes，初步鎖定 [Fujitsu PalmSecure](https://www.fujitsu.com/global/services/security/offerings/biometrics/palmsecure/) 一類的感測器 ([應用場景](https://twitter.com/i_am_pass/status/1006768419436363778));
2. 作為 [IOTA Sandbox](https://hackmd.io/s/ByWQUqfh-) 或 [PoWbox](https://blog.iota.org/relaunching-the-powbox-d392236b6939) 的負載平衡器暨加速器;

現有 IOTA PoW 的 FPGA 加速器實作:
* [IOTA PoW Hardware Accelerator FPGA for Raspberry Pi](https://microengineer.eu/2018/04/25/iota-pearl-diver-fpga/): Cyclone 10, MIT License
* [FPGA based hardware accelerator for IOTA Curl and POW operations](https://github.com/LampaLab/iota_fpga): Cyclone V, MIT License

:::info
用 FPGA 加速的 IOTA PoW 時間分佈 (橫軸單位: ms)

* 25% of all nonces are found within 87ms
* 50% of all nonces are found within 200ms
* 75% of all nonces are found within 433ms
:::

預期 [iota-swarm-node](https://github.com/yillkid/iota-swarm-node) 整合方案
1. 將 FPGA accelerator 包裝 dcurl 介面: [shufps/dcurl](https://github.com/shufps/dcurl) 已有初步實作
2. iota-swarm-node 現有 Python 載入 dcurl 並且驗證 IOTA / TangleID APIs

## 評估

Q1: 是否有TrustZone?
> [name=AjMaChInE] p.102, [SoCKIT to Me!Don’t Just Elevate, Integrate.](https://doc.xdevs.com/doc/ALTERA/Arrow%20SoCkit%20SW%20Workshop_13.0%20%5BRepaired%5D.pptx), 說明Arrow SoCKit Board可以做TrustZone應用

Q2: 選擇 shufps/dcurl或 LampaLab/iota_fpga?

|items\projects | shufps/dcurl | LampaLab/iota_fpga |
| -------- | -------- | -------- |
| board| (old)[Altera DE1](https://www.altera.com/solutions/partners/partner-profile/terasic-inc-/board/altera-de1-board.html) (new)(customize)Altera DE1 | [Intel DE10-Nano Board](https://software.intel.com/en-us/articles/de10-nano-board-schematic)
| FPGA | (old) Cyclone 2 (new) Cyclone 10 LP | Cyclone V
| HDL | VHDL-Core | Verilog/System Verilog |
| HDL source| [IOTA VHDL PoW (Pearl Diver)](https://github.com/shufps/iota_vhdl_pow) | [iota_fpga/curl_accel_soc/hardware/](https://github.com/LampaLab/iota_fpga/tree/master/curl_accel_soc/hardware)

>[name=AjMaChInE]
>
>* 建議兩個都可以試試看，但Altera DE1 Cyclone 2不用試，因為硬體架構上沒有[HPS](https://www.altera.com/products/fpga/features/cyv-soc-hps.html) 與 [Arrow SoCKit board](http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=205&No=816&PartNo=2)有落差
>* 預計先試shufps/dcurl再試LampaLab/iota_fpga

PS:
* "This respository will not only contain VHDL source code and Altera DE1 project-files but also everything needed for **a custom PCB** (**with a modern FPGA Cyclone 10 LP**)which is plugged on top of a Raspberry Pi. Proto-Type is reaching 14.6MH/s" [IOTA VHDL PoW (Pearl Diver)](https://github.com/shufps/iota_vhdl_pow)
* "Intel Cyclone 10 LP FPGAs extend the low-power leadership of the previous generation Cyclone V FPGAs, 參考": [Intel® Cyclone® 10 LP FPGA](https://www.altera.com/products/fpga/cyclone-series/cyclone-10/cyclone-10-lp/overview.html)

Q3: FPGA合成資源初步判定

| Items\Projects | shufps/dcurl | LampaLab/iota_fpga | [Arrow SoCKit](http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=205&No=816&PartNo=2)|
| -------- | -------- | -------- | -------- |
| LEs | [23,877](https://microengineer.eu/2018/04/25/iota-pearl-diver-fpga/)| ? |110K |
| ALMs | ? | 12 377 ALMs, 23 945 flip-flops (30% of 5CSEBA6U23I7 FPGA) | 41509 |
>[name=AjMaChInE] 兩者初步判定皆可以在Arrow SoCKit上使用其資源合成。

PS:
* ALM: Adaptive Logic Module

* LE: Logic Element

Q4: Weakness: FPGA與CPU有共有區塊，此區塊會保存seed，可以透過mmap存取

>[name=AjMaChInE]
>* The SoCKit board的 [Specifications](http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=205&No=816&PartNo=2)並沒有寫到TZC400 (TrustZone® Address Space Controller)
>* 替代解法: 設定normal world的page table讓此區塊不能讀不能寫不可執行且還需對OPTEE修改並撰寫OPTEE TA，這部分想要另外談。

## TODO

1. shufps/dcurl 移植的執行順序
* 1.1 FPGA合成shufps/dcurl Cyclone 10 LP 版本
* 1.2 移植現有[IOTA VHDL PoW (Pearl Diver)](https://github.com/shufps/iota_vhdl_pow)從[Intel DE1-Nano Board](https://www.altera.com/solutions/partners/partner-profile/terasic-inc-/board/altera-de1-board.html)至Arrow SoCKit Board
* [TODO紀錄](https://hackmd.io/vXdPy9ilSYGKS6M1wh4Ppw?both)
* 1.3 iota-swarm-node 現有 Python 載入 dcurl 並且驗證 IOTA / TangleID APIs

2. LampaLab/iota_fpga移植(ED10->Arrow SoCKit)的執行順序
* 2.1 使用Quartus合成LampaLab/iota_fpga，而device從ED10->Arrow SoCKit
* [TODO紀錄-done](https://hackmd.io/2a2x1q9QTJq0MsGWEyJbhA)
* 2.2 移植[LampaLab/iota_fpga/software](https://github.com/LampaLab/iota_fpga/tree/master/pow_accel_soc/software)從[Altera DE10]((https://software.intel.com/en-us/articles/de10-nano-board-schematic))至Arrow SoCKit Board
* [TODO紀錄-done](https://hackmd.io/FMERGjcqTYO4Nn8jBRYOQA)
* 2.3 驗證ccurl
* [TODO紀錄](https://hackmd.io/s/r1RcCnxfQ)
* 2.4 Replace ccurl's POW with dcurl’s POW in LampaLab/iota_fpga
* [TODO紀錄](https://hackmd.io/blMdDCDzTCqLKXMmfHmfDw?view)
* 2.5 [iota-swarm-node](https://github.com/yillkid/iota-swarm-node) 現有 Python 載入 dcurl 並且驗證 IOTA / TangleID APIs

3. RPi3+OPTEE執行順序:
* 3.1 建立RPi3+OPTEE環境
* 3.2 撰寫TA (PoW) for RPi3+OPTEE
PS: seed在normal world與seucre world共享區塊是加密的，TA負責解密

PS: shufps/dcurl的第二步驟如果無法突破將會放棄，執行LampaLab/iota_fpga的移植; 如果成功，將不執行LampaLab/iota_fpga的移植。

## Future Work

Weakness解決方案執行順序:
* 建置OPTEE在The SoCKit board上，需修改編譯流程。
* 設定nornal world的page table 不能讀不能寫不可執行此區塊
* 建立nornal world與sercure world共享區域
* 修改OPTEE的page table建立FPGA使用的記憶體區塊
* 由Tasklet搬此共享區資料至FPGA使用的記憶體區塊並撰寫解加密的seed程式
PS: nornal world與sercure world共享區域存放加密的seed

## 參考資訊

- [ ] [Altera DE1](https://www.altera.com/solutions/partners/partner-profile/terasic-inc-/board/altera-de1-board.html)

- [ ] [Intel DE10-Nano Board](https://software.intel.com/en-us/articles/de10-nano-board-schematic):
* 5CSEBA6U2317

- [ ] [Arrow SoCKit board](http://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=205&No=816&PartNo=2):
* Cyclone V SX SoC—5CSXFC6D6F31C6N
* 110K LEs, 41509 ALMs