# 10820CS 542100, 雲端計算 Cloud Computing, HW1
###### tags: `nthu cloud computing`

:::success

Upload image: `ssh [id]@140.114.91.176 -p20122`

1.上傳image位置: /home/cloud/{學號}
2.1人使用30G
3.server只給學生測試advance效能，不允許長時間佔用server，請學生不要在server上compile kernal

advance score:
advance baseline ->kernel boot time + uesr boot time = 8.155s
60% of advance 只要報告中截圖看出確實有修改kernel，即可拿到60％
40% of advance 以所有修課同學的最快秒數當做40％，baseline當作0％，依照比例配分

:::
# Report checklist
- [x] 1.System Spec (kernel version, OS, CPU core, memory......)
- [x] 2.Show your screenshot to proof that you use Qemu.
- [x] 3.Which modules did you modify in kernel? What do these modules use for?
- [x] 4.Did you find any other OS image boot faster than ubuntu and why? Show your screenshot of your experiment
- [x] 5.Compare the boot time between general config and modify config & Show your screenshot of your experiment (We need two $systemd-analyze result screenshot “baseline + advance”)
- [x] 6.Show your screenshot to proof that your migrate is success.
- [x] 7.Use different number of CPU core to test MNIST execution time and make a table.
- [x] 8.Please simply explain what is the difference between cold migration, warm migration and live migration.

# Basic

### 0. 實驗環境
- Ubuntu 19.10
- kernel: 5.3.0-42-generic
- 12G RAM
- Intel(R) Core(TM) i5-6200U

### 1. Download linux system Ubuntu-16.04.6 (.iso) & cirros image
ubuntu 16.04.6

wget http://releases.ubuntu.com/16.04/ubuntu-16.04.6-desktop-amd64.iso

cirros image

wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

### 2. Create a VM by qemu

懶人作法，用virt-manager創好VM再用`domxml-to-native`導出QEMU參數修改，不僅libvirt會幫忙架好bridge，NAT一併搞定，現在就安裝libvirt

~$ sudo apt install virt-manager -y
~$ brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a5d34f	yes		virbr0-nic

# 使用virt-manager架設好VM導出參數
~$ domxml-to-native qemu-argv --domain [domain]

接著允許qemu使用qemu-bridge-helper在virbr0上創建tap

sudo mkdir -p /etc/qemu
sudo echo "allow virbr0" > /etc/qemu/bridge.conf

安裝vnc viewer

~$ sudo apt install remmina -y

使用cirros測試調好的參數

~$ sudo qemu-system-x86_64 \
-hda /home/meow/Images/cirros-0.5.1-x86_64-disk.img \
-device virtio-net-pci,netdev=hostnet0 -netdev bridge,br=virbr0,id=hostnet0 \
-vnc :7777 \
-monitor stdio

由於vnc port預設開在5900+d，上方的vnc port等同13677，使用remmina連上的畫面如下

(qemu) info vnc
default:
  Server: :::13677 (ipv6)
    Auth: none (Sub: none)
  Server: 0.0.0.0:13677 (ipv4)
    Auth: none (Sub: none)


接著測試ubuntu iso，限制使用
- 4 vcpu
- 4096MB RAM，因為1024/2048MB不足以安裝tensorflow套件

參數如下

~$ sudo qemu-system-x86_64 \
-boot strict=on \
-machine pc-q35-4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
-cpu Skylake-Client-IBRS,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,clflushopt=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,pdpe1gb=on,skip-l1dfl-vmentry=on,hle=off,rtm=off \
-m 4096 \
-smp 1,sockets=1,cores=4,threads=1,maxcpus=4 \
-cdrom /home/meow/Images/ubuntu-16.04.6-desktop-amd64.iso \
-device virtio-net-pci,netdev=hostnet0 -netdev bridge,br=virbr0,id=hostnet0 \
-vnc :7777 \
-monitor stdio


### 3. Run a simple MNIST on a VM
使用前面架設的VM運行python腳本

# 檢查tensorflow所需的avx指令集
~$ lscpu | grep avx

sudo apt update
sudo apt install python3 python3-pip -y
python3 -m pip install tensorflow
wget -O mnist.py https://pastebin.com/raw/g4zTJD6C
python3 mnist.py


### 4. Live migration I - 使用Ubuntu live CD

live migration建議使用shared storage，不過OS都在RAM裏面就沒有這問題了吧？參考[Remove bootable USB after live-booting from it](https://unix.stackexchange.com/questions/430037/remove-bootable-usb-after-live-booting-from-it)於cmdline加入`toram=yes`。


整個OS都在ram裡，此時把光碟退出並用qemu monitor刪除ide-cd0

(qemu) info block
ide2-cd0 (#block172): /home/meow/Images/ubuntu-16.04.6-desktop-amd64.iso (raw, read-only)
    Attached to:      /machine/unattached/device[17]
    Removable device: not locked, tray open
    Cache mode:       writeback

sd0: [not inserted]
    Removable device: not locked, tray closed
(qemu) eject ide2-cd0
(qemu) drive_del ide2-cd0 
(qemu) info block
/machine/unattached/device[17]: [not inserted]
    Attached to:      /machine/unattached/device[17]
    Removable device: not locked, tray open

sd0: [not inserted]
    Removable device: not locked, tray closed

接著執行migration，先使用相同的參數啟動VM，只是多了`-incoming tcp:0:6666`也沒有光碟機，QEMU會啟動一台VM狀態為`paused`等待migrate。

# dst
sudo qemu-system-x86_64 \
-boot strict=on \
-machine pc-q35-4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
-cpu Skylake-Client-IBRS,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,clflushopt=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,pdpe1gb=on,skip-l1dfl-vmentry=on,hle=off,rtm=off \
-m 4096 \
-smp 1,sockets=1,cores=4,threads=1,maxcpus=4 \
-device virtio-net-pci,netdev=hostnet0 -netdev bridge,br=virbr0,id=hostnet0 \
-vnc :7778 \
-monitor stdio \
-incoming tcp:0:6666


下migrate指令，大概花個幾秒完成

使用vnc察看目標

### 5. Live migration II - 使用Cirros image+NFS

創立兩台host，分別是cc-1(192.168.0.21)和cc-2(192.168.0.41)分別作為src和dest，在cc1內創一台VM，目標是live migrate到cc2。為了減少live migration時間，VM使用cirros image生成。

:::info
為了方便，兩台host架在openstack
:::

依據[Migration-KVM](https://www.linux-kvm.org/page/Migration)的要求，VM image必須在shared storage上且建議能以相同路徑存取，不共用同一份image結果就是guest VM有機會在live migration結束後檔案系統損毀，且src和dst啟動VM方式必須一樣。

:::warning
share storage的caching必須關閉避免migrate到dst後src資料尚未寫入
:::

參照[Ubuntu server guide章節](https://help.ubuntu.com/stable/serverguide/network-file-system.html)在cc-1設置nfs server

sudo apt install nfs-kernel-server -y
sudo mkdir -p /mnt/nfs
echo '/mnt/image      192.168.0.0/24(rw,sync,no_root_squash)' | sudo tee -a /etc/exports
sudo systemctl restart nfs-kernel-server.service 

在cc-2掛載

sudo mkdir /mnt/nfs
sudo apt install nfs-common -y
echo '192.168.0.41:/mnt/nfs /mnt/nfs nfs auto,sync,x-systemd.requires=network-online.target     0       0' | sudo tee -a /etc/fstab
sudo mount -a

啟動VM，可看到src上VM狀態是running，dst是paused

# dst
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -O /mnt/nfs/cirros.img # download cirros image

qemu-system-x86_64 -enable-kvm -m 256 \
-smp 1,sockets=1,cores=1,threads=1 \
-drive file=/mnt/nfs/cirros.img,format=qcow2,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-vga cirrus -usb -device usb-tablet -vnc 0.0.0.0:7 -monitor stdio
(qemu) info status
VM status: running

# dst
sudo qemu-system-x86_64 -enable-kvm -m 256 \
-smp 1,sockets=1,cores=1,threads=1 \
-drive file=/mnt/nfs/cirros.img,format=qcow2,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-vga cirrus -usb -device usb-tablet -vnc 0.0.0.0:7 -monitor stdio \
-incoming tcp:0:6666
(qemu) info status
VM status: paused (inmigrate)

接著進行migrate，結束後src上VM的status變為running

# src
(qemu) migrate tcp:192.168.0.21:6666
(qemu) info status
VM status: paused (postmigrate)

# dst
(qemu) info status
VM status: running


### 參考資料
- [Qemu - Invocation](https://www.qemu.org/docs/master/system/invocation.html#sec-005finvocation)
- [Bridged networking using qemu-bridge-helper](https://wiki.archlinux.org/index.php/QEMU#Bridged_networking_using_qemu-bridge-helper)
- [Configuring Guest Networking](https://www.linux-kvm.org/page/Networking)
- [Live Migrating QEMU-KVM Virtual Machines](https://developers.redhat.com/blog/2015/03/24/live-migrating-qemu-kvm-virtual-machines/)

# Advanced

這裡使用libvirt建立虛擬機來編譯、安裝kernel，因為virt-manager建立的image會預留空間，這對視硬碟容量為寸土寸金的使用者來說是壞事，所以先使用qemu-img建立image避開這問題，接著用`ubuntu1604 mini.iso`安裝，只安裝必要元件和`openssh-server`。

qemu-img create -f qcow2 disk.qcow2 40G

### 實驗環境
- Ubuntu 18.04
- kernel: 5.3.0-45-generic
- 128G RAM
- AMD Ryzen 9 3900X 12-Core Processor
- WDC WD1003FZEX, RAID 0+1
- Intel 10G Network Adapter

### VM Spec
- VCPU: 24
- RAM: 16G
- Disk: 40G
- Driver: virtio
- Format: qcow2

### 編譯kernel
取得kernel source code有二
- [kernel.org](https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/)
- apt source

這裡採後者，先在`/etc/apt/sources.list`加入這兩行

deb-src http://archive.ubuntu.com/ubuntu xenial main
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main

依據 [Ubuntu - BuildYourOwnKernel](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel) 安裝必要套件

sudo apt-get build-dep linux linux-image-$(uname -r)
sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf
sudo apt install vim htop iotop git

接著取得源碼

sudo apt update
apt-get source linux-image-unsigned-`uname -r`
cd linux-hwe-4.15.0

編譯方法可參照 [KernelBuild](https://kernelnewbies.org/KernelBuild)

### 優化方法 I - customize kernel image
[A pragmatic guide to boot-time optimization](https://elinux.org/images/6/64/Chris-simmonds-boot-time-elce-2017_0.pdf) 提到 用 `bootgraph.pl` 視覺化開機時間，檔案就位於 `linux-hwe-4.15.0/scripts/bootgraph.pl`

由於 `dmesg` 會定時 `rotate` ，因此重開機馬上執行 `bootgraph` 較為準確。使用 `bootgraph` 前先於 `/etc/default/grub` 的 `GRUB_CMDLINE` 加入 `printk.time=1 initcall_debug` 並 `sudo update-grub`。

整個optimization流程如下：
1. 建立snapshot
2. 產出 bootgraph 分析最佔開機時間的部份，花6分鐘重新編譯kernel

    dmesg | linux-hwe-4.15.0/scripts/bootgraph.pl > boot.svg
    make menuconfig
    make -j24 # 保險起見或加速下次compile，這裡snapshot
    sudo make modules_install
    sudo make install


3. 回到Step2，直到滿意開機時間

此外[NVIDIA -Kernel boot time optimization](https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fkernel_boot_time.html%23wwpID0ERHA)指出:

Additionally, examine the required configurations and verify that they can be programmed as modules so that the drivers are loaded when it is called for use.

因此 `make menuconfig` 時沒有必要對已經是 `module("M")` 的選項做處理。第一次`make menuconfig` 很容易看出不必要的選項(GPIO Support, 1-wire Support...)，一並處理掉。有異常直接倒回上一個 snapshot 或用 `make mrproper` 載入預設config。

大致上砍了這些東西
- ACPI
>虛擬機不需要電源管理
- Device drivers
- Macintosh device drivers
- Fusion MPT device support
- SPI Support
- Network device support
- Fibre
- Ethernet driver support
> 幾乎砍光
- FDDI Driver support
- Wireless LAN
> 沒有 wireless 需求，全砍
- Xen
- VMWare
- FUJITSU...
- Networking over Thunderbolt cable
- Microsoft Hyper-V ..
- ISDN Support
- Watchdog Timer Support
- Voltage and Current Regulator Support
- Graphics support
> 砍到剩
> - /dev/agpgart (AGP Support)
> - VGA Arbitration
- SD card support
- InfiniBand support
- Xen driver support
- Multimedia support
> 只留 Media USB Adapters/Media PCI Adapters
- File systems
- JFS/XFS/GFS2/OCFS2/Btrfs(最肥)/F2FS support

原先系統啟動時間需要 4.1 秒

~$ systemd-analyze time
Startup finished in 2.058s (kernel) + 2.047s (userspace) = 4.105s

調整過後來到 5.2 秒

~$ systemd-analyze time
Startup finished in 2.889s (kernel) + 1.753s (userspace) = 4.642s

雖然載入的模組變少，但因為 `initrd.img` 從 50M 增加到 206MB 的關係導致開機時間變長，從bootgraph 看出大部份時間花在 `populate_rootfs`

- 優化前

- 優化後

### 優化方法 II - strip kernel module
[How to reduce the size of the initrd when compiling your kernel?](https://unix.stackexchange.com/questions/270390/how-to-reduce-the-size-of-the-initrd-when-compiling-your-kernel) 提及同樣的問題，透過下列指令減少 `initrd.img` 大小到 23M。

sudo make INSTALL_MOD_STRIP=1 modules_install && sudo make install

結果如下

~$ systemd-analyze time
Startup finished in 1.421s (kernel) + 1.635s (userspace) = 3.057s


緊接著將 `/etc/initramfs-tools/initramfs.conf` 的 `MODULES=most` 改為 `MODULES=dep` 後再次安裝，`initrd.img` 大小來到 6.7M

~$ systemd-analyze time
Startup finished in 619ms (kernel) + 1.657s (userspace) = 2.277s


### 優化方法 III - disable initrd
借用 [Boot time optimization](https://bootlin.com/doc/training/boot-time/boot-time-slides.pdf) 的圖說明 linux 開機程序


BIOS先載入Boot Loader，Boot loader 再將控制權交給 linux kernel，同時 initramfs (initial RAM-based file system) 也載入記憶體供 kernel 使用，提供一些硬體驅動，裡頭的 init 執行檔負責載入真實根目錄系統，完成後 initramfs 將被清除並交由 systemd 接手。initrd 也是同樣的功能但有些微差距，而且檔案名稱叫 initrd 不一定是真的 initrd ，詳見 [Linux initrd and initramfs](http://weng-blog.com/2017/03/20/linux-initrd-initramfs.html)。

/boot$ file initrd.img-4.15.0-96-generic 
initrd.img-4.15.0-96-generic: ASCII cpio archive (SVR4 with no CRC) # 這不是 initramfs 嗎

從"優化方法 II"的 bootgraph 的圖中我發現 `populate_rootfs` 依然佔了很大一塊，如果能拿掉這塊啟動時間勢必加快。[Is it possible to boot the Linux kernel without creating an initrd image?](https://stackoverflow.com/questions/6405083/is-it-possible-to-boot-the-linux-kernel-without-creating-an-initrd-image) 提到

However if you compile your kernel with all required drivers and modules built into the kernel image, and your root file system device has a fixed name in the system you don't need a initramfs as the kernel can do things by itself then.

依照 [How to inspect initramfs](https://wiki.debian.org/initramfs) 的方法先是察看`initrd`

~$ cpio -i < /boot/initrd.img-4.15.18 
56 blocks
~$ dd if=initrd.img-4.15.18 of=initrd.img bs=512 skip=56; zcat initrd.img |cpio -i
13486+1 records in
13486+1 records out
6905111 bytes (6.9 MB, 6.6 MiB) copied, 0.0187152 s, 369 MB/s
32893 blocks

然後將 `initrd.img` 涵蓋的 module 改為 built-in，然後在 compile kernel 時取消 initramfs support

> 然後就不會動了 :smile: 不熟 Ubuntu 改不了哎

### 優化方法 IV - set acpi off and adjust `initrd.img` compression method

"優化方法 II" 的 bootgraph 指出仍有改進空間，先拿掉
1. Device drivers -> Character devices -> Serial driver -> 8250

接著修改 `/etc/grub/default`，然後執行 `sudo update-grub`

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=off"

關閉 acpi 的副作用有二
1. 關機後畫面出現"你可以放心關機"，然後自行 kill process ，雖然社群不建議這麼做，但拿掉有助於開機速度大幅提升 :smiley:
2. `pci_subsys_init` 花費時間將近2倍，但和 acpi 比起來不算什麼

再來參照 [populate_rootfs](https://elixir.bootlin.com/linux/latest/source/init/initramfs.c#L654)，dmesg 指出 `unpack_to_rootfs` 佔用 `populate_rootfs` 一大段時間，於是修改 `/etc/initramfs-tools/initramfs.conf`，參考 [Linux compressors comparison](https://linuxaria.com/article/linux-compressors-comparison-on-centos-6-5-x86-64-lzo-vs-lz4-vs-gzip-vs-bzip2-vs-lzma) 改用 lzop 壓縮 `initrd.img`

#COMPRESS=gzip
COMPRESS=lz4

重新編譯 kernel 後的 `initrd.img` 大小為 6.7M，應該有稍微增大些。開機時間為

~$ systemd-analyze time
Startup finished in 417ms (kernel) + 781ms (userspace) = 1.199s


bootgraph 指出 `populate_rootfs` 時間幾乎沒變化，整體時間減少還是得益於 `acpi=off`。

### 優化方法 V - convert qcow2 to raw
[FVD: a High-Performance Virtual Machine Image Format for Cloud](https://www.usenix.org/legacy/events/atc11/tech/final_files/Tang.pdf) 的 How a Virtual Disk Works Today 說明 qcow2 是如何運作的。

1. VM 請求位於 virtual block address (VBA) 的資料給 QEMU
2. QEMU 轉換成 image block address (IBA) ，IBA 紀錄資料位於 qcow2 image 何處
3. 轉換方式如下圖，VBA 切成3塊 (d1,d2,d3)，d3 是位於 block 的 offset
4. 起先 qcow2 image 只有 L1 table，沒有 L2 table。
5. block 的位址在第一次資料寫入時才會決定，因此第一次安裝系統時，檔案是擠在一起，難保證資料的 reliablility 和 locality 還在。

綜合上述，把 image 從 qcow2 轉換成 raw 可能會減少開機時間。

qemu-img convert -f qcow2 -O raw newkernel.qcow2 newkernel.img

### 結論

- Default

- Optimized (優化方法 I + II + IV + V )，`.config` [在這](https://pastebin.com/raw/yT8FFvNc)


# Run MNIST on different number of vCPUs
使用 Advanced 所建立的 image 進行實驗
### 實驗環境
- Ubuntu 18.04
- kernel: 5.3.0-45-generic
- 128G RAM
- AMD Ryzen 9 3900X 12-Core Processor
- WDC WD1003FZEX, RAID 0+1
- Intel 10G Network Adapter

### VM Spec
- 1~24 VCPU
- 12G RAM
- 40G DISK
- Driver: virtio
- Format: qcow2

### 測試腳本

#!/bin/bash
cpu=`nproc`
for i in `seq 30`; do
    /usr/bin/time -a -o result.log -f "$cpu,%e" python3 mnist.py 2>/dev/null
done

### 實驗結果
原始資料[在這](https://pastebin.com/0Msm7Xzd)。

1,7.83
1,7.78
1,7.83
1,7.65
1,7.84
...

仿效 [OscarShiang](https://hackmd.io/@oscarshiang/S1mexZhEL) 的統計方式，將程式根據不同的 vCPU 個數（1 ~ 24）執行 30 次並取 95% 的信賴區間進行計算。我用 python 腳本將資料整理自動化

import pandas as pd
import math

vcpu = 24
elapsedtime = pd.DataFrame(columns = range(vcpu))

with open('data.txt', 'r') as f:
    lines = f.readlines()
    for i in range(vcpu):
        arr = []
        for line in lines:
            if int(line.split(',')[0]) == i+1:
                    arr.append(float(line.split(',')[1]))
        elapsedtime[i]=arr.copy()

print(elapsedtime)

# processing the data
mean_bt, std_bt = elapsedtime.mean(axis = 0), elapsedtime.std(axis = 0)
new_elapsedtime = []
print('processing boottime data...')
for i in range(len(elapsedtime.columns)):
    sum = 0
    cnt = 0
    for j in elapsedtime.iloc[:,i]:
        if abs(j - mean_bt[i]) < 2 * std_bt[i]:
            sum = sum + j
            cnt = cnt + 1
    sum = sum / cnt
    new_elapsedtime.append(sum)

out = []
for i, j in zip(range(len(new_elapsedtime)), new_elapsedtime):
	out.append('{},{}'.format(i+1, j))

# output the result
with open('data_.txt', 'w') as f:
    print('file write: {}'.format(f.write('\n'.join(out))))

# plot.gp
reset
set ylabel 'time(s)'
set xlabel 'vCPU'
set title 'Computation time with different vCPUs'
set term png enhanced font 'Verdana, 10'
set output 'output.png'
set datafile separator ','
set grid xtics ytics
set xtics 1
set ytics 1
set grid

plot[1:24][:] \
'data_.txt' using 1:2 with linespoints linewidth 3 title ''

而最後繪製的結果如下圖：


vCPU | computation time (s)
---|---
1|7.822068965517241
2|11.289310344827586
3|11.333214285714286
4|11.733448275862068
5|11.587857142857144
6|11.355862068965514
7|11.61357142857143
8|11.875517241379313
9|11.931428571428572
10|11.9551724137931
11|11.966551724137934
12|12.07357142857143
13|12.380333333333333
14|12.500999999999998
15|12.766896551724136
16|12.845666666666668
17|13.054827586206894
18|13.281379310344828
19|13.6651724137931
20|13.899310344827587
21|14.121071428571431
22|14.48178571428571
23|14.819642857142854
24|15.563928571428573

從圖表看出隨著 vCPU 增加所花費時間反而增加。

# Cold migration/warm migration/live migration

參考
1. [Hot and Cold Migrations; Which Network is Used?](https://blogs.vmware.com/vsphere/2019/12/hot-and-cold-migrations-which-network-is-used.html)
2. [Page 4, Virtual Machine Migration](https://nsrc.org/workshops/2014/sanog23-virtualization/raw-attachment/wiki/Agenda/migration-storage.pdf)

### Cold migration
移動(重建)虛擬機到其他 host，image 通常跟著一起複製過去。

### Warm migration
暫停運行中虛擬機，複製記憶體和暫存器內容到另一台 HOST 後繼續虛擬機器運行。

### Hot migration (live migration)
在虛擬機不停機情況遷移到另一台 host，一開始先同步記憶體，接著同步 dirty page ，等到 dirty page 數量剩下一定值後暫停虛擬機器，複製暫存器和剩餘的記憶體到另一台 host 上啟動虛擬機，downtime 為三者之中最小。

# OS image boot faster than ubuntu
參考 [Which Linux Distribution Boots The Fastest? An 11-Way Linux Comparison](https://www.phoronix.com/scan.php?page=article&item=11-linux-boot&num=2)


### 實驗環境
- Ubuntu 19.10
- kernel: 5.3.0-42-generic
- 32G RAM
- Intel(R) Core(TM) i7-6700 CPU
- Intel 530 SSD 256G
- TP-Link 1G Network Adapter

### VM Spec
- 2 VCPU
- 2G RAM
- 10G DISK
- Driver: virtio
- Format: qcow2

### 實驗
採用下列 linux distribution 做比較，都用 minimal install
- Ubuntu 16.04

- Fedora 31

- Clear linux 32800

clear linux 怎麼做到這點的? Intel 工程師 Feng Tang 在 [Linux Plumbers Conference](https://www.youtube.com/watch?v=A7N_O8pnyTw) 有說明。