Manual - HackMD

# Manual ## 1. Environment: --- **Server** ``` Ip: 100.73.42.5 (server1) 100.73.42.6 (server2) User: ubuntu Password: p@ssw0rd ``` **Network** ``` server1(100.73.42.5) br0: 100.73.42.5 -> 1G Net enp129s0f0: 192.168.105.1 -> 10G Net server2(100.73.42.6) br0: 100.73.42.6 -> 1G Net enp129s0f0: 192.168.105.2 -> 10G Net ``` ## 2. Cuju Execution --- ## Cuju ### 1. NFS - Insert this line in `/etc/exports` to add your NFS folder: ``` /home/[your username]/nfsfolder *(rw,no_root_squash,no_subtree_check) ``` - After editing `/etc/exports`, type `/etc/init.d/nfs-kernel-server restart` or `reboot` - Mount the NFS folder ``` $ sudo mount -t nfs 192.168.11.1:/home/[your username]/nfsfolder /mnt/nfs ``` ``` Script sample: #!/bin/bash sudo mount -t nfs 127.0.0.1:/home/ubuntu/blkserver_nfs/ /home/ubuntu/cuju_blkserver/blkserver/ ``` ### 2. Build Cuju * Clone Cuju on your NFS folder from Github ``` $ cd /mnt/nfs $ git clone https://github.com/Cuju-ft/Cuju.git ``` * Configure & Compile Cuju-ft ``` $ cd Cuju $ ./configure --enable-cuju --enable-kvm --disable-pie --target-list=x86_64-softmmu $ make -j8 ``` * Configure, Compile & insmod Cuju-kvm module `*1` `*2` ``` $ cd Cuju/kvm $ ./configure $ make -j8 $ ./reinsmodkvm.sh ``` >`*1` If you meet `error: incompatible type for argument 5 of '__get_user_pages_unlocked'`, you can use this patch: >``` >$ cd Cuju >$ patch -p1 < ./patch/__get_user_pages_unlocked.patch >``` > >`*2` If you meet `error: implicit declaration of function 'use_eager_fpu' [-werror=implicit-function-declaration]`, you can use this patch: >``` >$ cd Cuju >$ patch -p1 < ./patch/use_eager_fpu.patch >``` ### 3. Execute Cuju * Before launching your VM, you should update kvm module in Primary and Backup nodes: ``` $ cd /mnt/nfs/Cuju/kvm $ ./reinsmodkvm.sh ``` * Boot VM (on Primary Host) * ```runvm.sh``` ``` sudo ./x86_64-softmmu/qemu-system-x86_64 \ -drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \ -device virtio-blk,drive=drive0 \ -m 1G -enable-kvm \ -net tap,ifname=tap0 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \ -cpu host \ -vga std -chardev socket,id=mon,path=/home/[your username]/vm1.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline ``` You need to change the guest image path (`file=/mnt/nfs/Ubuntu20G-1604.img`) and monitor path (`path=/home/[your username]/vm1.monitor`) for your environment * Use VNC to see the console ``` $ vncviewer :5900 & ``` The default `account/password` is `root/root` if you use we provide guest image * Start Receiver (on Backup Host) * ```recv.sh```(sample `/home/ubuntu/cuju_blkserver/blkserver/Cuju/run_primary`) ``` sudo x86_64-softmmu/qemu-system-x86_64 \ -drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \ -device virtio-blk,drive=drive0 \ -m 1G -enable-kvm \ -net tap,ifname=tap1 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \ -vga std -chardev socket,id=mon,path=/home/[your username]/vm1r.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \ -cpu host \ -incoming tcp:0:4441,ft_mode ``` * After VM boot and Receiver ready, you can execute following script to enter FT mode * ```ftmode.sh```(sample `/home/ubuntu/cuju_blkserver/blkserver/Cuju/ft_mode`) ``` sudo echo "migrate_set_capability cuju-ft on" | sudo nc -U /home/[your username]/vm1.monitor sudo echo "migrate -c tcp:192.168.111.2:4441" | sudo nc -U /home/[your username]/vm1.monitor ``` You need to change the ip address and port (`tcp:192.168.111.2:4441`) for your environment, this is Backup Host's IP And change the monitor path (`/home/[your username]/vm1.monitor`) for your environment ## Group Fault Tolerance --- ### 1. GFT work flow in server 100.73.42.5 * 開機後先執行~/nfs/Cuju/kvm裡的reinsmodkvm.sh * 開啟VM: 在~/nfs/Cuju裡執行./open_primary.sh -> ./open_backup.sh * 開啟5台VM: id 0~4 * open_primary.sh會執行run_ft~run_ft5（各自的VM開機script) * 調整gft id/RAM大小/smp數量...等 * open_backup.sh會執行recv_b~recv_b5（各自的backup VM 開機script) * * 若要開超過5台VM: 到100.73.42.6 ~/nfs/Cuju裡執行 ./open_primary6.sh -> ./open_backup6.sh * 開啟5台VM: id 5~9 * 用vncviewer確認VM是否開好 * 開完後執行add_host.sh * add_host.sh內容依group member數量自行調整參數 * gft_add_host [gft_id] [primary的實體機器ip] [GFT PORT] [VM MAC] [SLAVE的實體機器ip] [SLAVE接收備份的port] * 每個gft_add_host都是下到leader裡 * 若超過五台可以使用add_host_more.sh * 若超過五台，需要到100.73.42.6 ~/nfs 下cuju_on_6.sh * 這個script對.6的機器下migrate_set_capability cuju_ft on，因為在.6開的機器monitor無法在.5控制，所以要先設定 * 最後在.5 ~/nfs 執行./gft_init.sh 開始GFT * gft_init.sh分為兩部份： migrate_set_capability cuju_ft on 和 gft_init * migrate_set_capability cuju_ft on 在每台group member都要下 * gft_init只要對leader下就好 #### individual FT ＆ non-buffered FT測試 * 前面步驟一樣開啟VM,確認VM是否開好 * 開完後對每台VM執行ftmode[X].sh, [X]是每台機器順序（ftmode.sh~ftmode10.sh) * non-buffered FT測試要先修改code, 在 $~/nfs/Cuju/hw/net/virtio-net.c裡 ``` // For CUJU-FT //ret = qemu_sendv_packet_async(qemu_get_subqueue(n->nic, queue_index), // out_sg, out_num, virtio_net_tx_complete); ret = qemu_sendv_packet_async_proxy(qemu_get_subqueue(n->nic, queue_index), out_sg, out_num, virtio_net_tx_complete); ``` 修改成 ``` // For CUJU-FT ret = qemu_sendv_packet_async(qemu_get_subqueue(n->nic, queue_index), out_sg, out_num, virtio_net_tx_complete); //ret = qemu_sendv_packet_async_proxy(qemu_get_subqueue(n->nic, queue_index), // out_sg, out_num, virtio_net_tx_complete); ``` #### run stage length測試 * 修改Cuju/include/migration/cuju-kvm-share-mem.h中的 EPOCH_TIME_IN_MS #### memory滿載 * 在VM中執行以下script即可讓VM memory滿載 * #!/bin/bash stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1 ### 2. APM synthetic multi-tier service testing #### work flow 1. 在~/nfs/amtt裡的_amtapp_linear.yaml(可改檔案名）修改topology配置 2. 生成amttcn -> $./amttcngen.sh _amtapp_linear.yaml 3. 確認VM都以開啟完畢 4. 到~/nfs/cmd裡使用amttcn配置topology到各VM中 -> $./amttcn 5. 執行後即可對開啟的ip和port傳request -> ~/nfs 裡 $./curl.sh #### script功能 * _amtapp_linear.yaml (可以改名） -> 設定topology * compute_node -> 設定實體機器 * 因為兩台機器互通且有nfs，故這邊只需要設定一台就可以跑兩台機器的實驗 * 大component -> 不用修改 * application * yaml沒有case_linear application使用，故linear的測試就沿用case_loadbalancer * 配置各個VM，VM名字直接使用VM的ip * forwarder設定arg1~arg5: {forwarder自己的ip / 自己的port / 要轉傳到的VM ip / 要轉傳到的VM port / delay（可以不用改）} * DB server設定arg1~arg3: {DB server自己的ip / 自己的port / delay} * Load balancer設定arg1~arg3: {自己ip / 自己port / delay} * content裡設定有幾台可以forward（fanout) * Multicaster設定同Load balancer * curl.sh * 跑多次curl指令傳POST request的script * 可調整次數（for迴圈） ### 3. GFT * migration.c裡的#define GFT_RESYNC -> 開關resync功能（註解掉即關閉） * migration.c裡的#define ft_debug_mode_enable -> 開關debug message #### leader code flow * hmp_gft_init * qmp_gft_leader_init * set fd handler => fd:group_ft_master_sock / read:gft_master_accept_other_master * set the value of group_ft_leader_inited * distribute group member information to all group nodes * SEND MIG_JOIN_GFT_ADD_HOST & MIG_JOIN_GFT_INIT to master * set fd handler for each master => fd:sd / read:gft_leader_read_master * gft_start_migration() * gft_start_migration * find self index in group_ft_members array and start qmp_migrate respectively for each node * qmp_migrate * cuju_tcp_start_outgoing_migration * cuju_socket_start_outgoing_migration * cuju_migration_channel_connect * migrate_fd_connect * migration_thread * after thread created, enter migrate_ft_trans_get_ready(how?) * gft_connect_internal * gft_master_connect_other_master * gft_leader_read_master * expect MIG_JOIN_GFT_MIGRATION_DONE from individual nodes * when receive enough acks, call gft_leader_broadcast_all_migration_done() * gft_leader_broadcast_all_migration_done * send MIG_JOIN_GFT_MIGRATION_ALL to all nodes, which will be read by gft_master_read_leader #### master code flow * gft_init * set fd handler => fd:group_ft_master_sock / read:gft_master_accept_leader * gft_master_accept_leader * accept MIG_JOIN_GFT_ADD_HOST & MIG_JOIN_GFT_INIT from leader * group members get the info about other group members * set fd handler => fd:server_fd(group_ft_master_sock) / read:gft_master_accept_other_master * set fd handler => fd:fd(group_ft_leader_sock) / read:gft_master_read_leader * gft_start_migration() * gft_master_accept_other_master * accept the * if(GFT_WAIT) call migrate_ft_trans_get_ready(); * gft_master_read_leader - Receive command MIG_JOIN_GFT_MIGRATION_ALL - close fd - set group_ft_members_ready = group_ft_members_size #### master code flow while GFT_WAIT * gft_master_read_master * detect other master VM failed * gft_status = GFT_WAIT * call gft_reset_all * gft_reset_all * gft_reset_connections current migrate state * gft_reset_connections previous migrate state * set s->ft_state to FT_INIT * set n->ft_state to FT_TRANSACTION_PRE_RUN * set fd handler => fd:group_ft_master_sock / read:gft_master_accept_leader * gft_master_accept_leader * qemu_iohandler_ft_pause(false) * recv MIG_JOIN_GFT_ADD_HOST & MIG_JOIN_GFT_INIT * recv info. about other group members * set fd handler => fd:server_fd(group_ft_master_sock) / read:gft_master_accept_other_master * set fd handler => fd:fd(group_ft_leader_sock) / read:gft_master_read_leader * send MIG_JOIN_GFT_INIT_ACK * gft_start_migration() * gft_master_accept_other_master * s1 = migrate_get_current(); * s2 = migrate_get_next(s1); * don't set cur_off, using current cur_off * call migrate_ft_trans_get_ready(); * migrate_ft_trans_get_ready * gft_connect_internal * gft_master_connect_other_master * send MIG_JOIN_GFT_NEW & my_gft_id * gft_master_notify_leader_migration_done * if (group_ft_leader_sock) * **send MIG_JOIN_GFT_MIGRATION_DONE (by group_ft_leader_sock)** * else if group_ft_members_ready == group_ft_members_size * gft_leader_broadcast_all_migration_done * gft_master_wait_all_migration_done * if group_ft_members_size == group_ft_members_ready * gft_prepare_snapshot_bitmap * gft_master_start_listen_other_masters * migration_paused = false * qemu_iohandler_ft_pause(false) * vm_start * migrate_run * change to GFT_PRE ### 4. GFT add/remove member * fail發生後手動add member(需回到前一個commit) * 對failover的member下live migration指令 -> gft_member_live_mig.sh * live mig結束後對leader下add_member指令 -> gft_add_member.sh * fail發生後自動add member * 建立GFT前要在add_host.sh裡加gft_add_backup（Cuju/gft_script資料夾裡) 加入backup pool的VM * 對slave VM下auto_resync.sh 讓slave VM知道backup pool VM * 目前狀況：砍最後一台VM可以正常resync，但砍其他台可能有問題 * add/remove新增變數說明 * group_ft_backup * 儲存由gft_add_backup加入的backup pool備份機器的資訊 * group_ft_failed_members * 儲存偵測到發生錯誤的機器的資訊 * resync_sock * 建立另一個socket給failover member 作live migration初始備份完成時通知leader的管道 * is_gft_adding_new_member * 確認目前的GFT是不是在加新成員的階段（下gft_add_member之後） * is_accept_finish * check trans_get_ready enter after accept_other_master finish in resync process member side * gft_backup_used * 目前backup pool有多少台已經被拿去當正式備份機器 * gft_backup_pool_size * resync_conn_count * 可以調整migrate_run中檢查failover member是否完成初始備份的socket connect的頻率 * resync_is_conn * 確認resync_sock是否建立 * resync_is_recv_flag * 確認resync_sock建立後傳的訊息是否有收到 * add/remove新增function * 1~7 -> dynamic resync流程 1. GFT有機器發生錯誤 2. 其他正常運作的member偵測到 * gft_master_read_master（）偵測到（len == -EINVAL) * 檢查是哪一台成員fail * leader收每一台的message,沒收到的就是failed member * if leader failed -> 換gft_id = 1的VM接手當新leader * 備份failed member的資訊 * call gft_reset_all() 3. 其他member暫停,減少成員, 重新建立少一個成員的GFT * 在gft_reset_all（）裡判斷是不是leader * 是的話就重建GFT info,略過failed member的資訊 * 執行gft_leader_init（） 4. fail的機器做failover到backup機器 5. failover後的機器作live migration初始狀態備份到新的backup VM（從backup pool裡拿） * cuju-ft-trans-file.c裡 cuju_ft_trans_close（） * 檢查backup pool是否還有backup VM可以用 * 若used >= pool_size代表沒有可用的backup VM * 直接繼續VM運行,先不作FT * else -> 讓failover VM和leader建立連線,建完後執行qmp_migrate（）作live migration初始狀態備份 * gft_backup_used++(用掉一台backup pool的VM) * qmp_migrate（）執行到 gft_master_wait_all_migration_done（）後暫停 6. failover VM做完初始狀態備份後暫停,通知leader去作加入成員的動作 * gft_master_wait_all_migration_done（）裡if(is_gft_new_member) * 傳訊息給leader * 傳完後close resync_sock 7. leader通知所有member暫停, 重新建立多一個成員的GFT * migrate_run（） * if(!resync_is_recv_flag && is_leader) * 每次migrate_run（）都嘗試接收failover VM是否做完初始狀態備份的訊（MIG_JOIN_GFT_READY_TO_RESYNC） * 接到MIG_JOIN_GFT_READY_TO_RESYNC後關閉resync_sock,作qmp_gft_add_member（）和gft_reset_all（） * qmp_gft_add_member（） -> 發訊息給所有group member讓他們也從gft_master_read_master（）進入gft_reset_all（）暫停 * leader進入gft_reset_all（） -> 重建GFT,先加入舊有的member * 離開gft_reset_all（）後再加入failover的member,然後qmp_gft_leader_init（）重啟GFT ## Block server --- ### 1. Switch to block server branch and reCompile Cuju ``` $ git checkout feature/blk-server $ make -j8 ``` ### 2. Build Block server - Make script of "server","primary"and"backup" for block server - `blk_server.sh`(sample `/home/ubuntu/cuju_blkserver/blkserver/Cuju/run_blkserver`) ``` sudo ./x86_64-softmmu/qemu-system-x86_64 \ -drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \ -device virtio-blk,drive=drive0 \ -m 1G -enable-kvm \ -net tap,ifname=tap0 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \ -cpu host \ -vga std -chardev socket,id=mon,path=/home/[your username]/vm1.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \ -ft-join-port 4000 -blk-server-listen :5001 ``` - `blk_primary.sh`(sample `/home/ubuntu/cuju_blkserver/blkserver/Cuju/run_primary_blk`) ``` sudo ./x86_64-softmmu/qemu-system-x86_64 \ -drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \ -device virtio-blk,drive=drive0 \ -m 1G -enable-kvm \ -net tap,ifname=tap0 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \ -cpu host \ -vga std -chardev socket,id=mon,path=/home/[your username]/vm1.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \ -blk-server [server IP]:5001 ``` - `blk_backup.sh`(sample `/home/ubuntu/cuju_blkserver/blkserver/Cuju/run_backup_blk`) ``` sudo x86_64-softmmu/qemu-system-x86_64 \ -drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \ -device virtio-blk,drive=drive0 \ -m 1G -enable-kvm \ -net tap,ifname=tap1 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \ -vga std -chardev socket,id=mon,path=/home/[your username]/vm1r.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \ -cpu host \ -incoming tcp:0:4441,ft_mode \ -blk-server [server IP]:5001 ``` ### 3.Execute Flow - 1.Execute server - 2.Execute primary(Use vncviewer to make sure that VM is started) - 3.Execute backup - 4.Execute ft_mode(Make sure that ft_mode is using 10G net) ``` sample of ft_mode(/home/ubuntu/cuju_blkserver/blkserver/Cuju/ft_mode) sudo echo "migrate_set_capability cuju-ft on" | sudo nc -U /home/ubuntu/cuju_blkserver/blkserver/vm1.monitor sudo echo "migrate -d -c tcp:192.168.105.1:4441" | sudo nc -U /home/ubuntu/cuju_blkserver/blkserver/vm1.monitor ``` - 5.Use vncviewer or ssh to connect VM ``` In our server,I set 100.73.42.51 for the cuju VM $sh root@100.73.42.51 ``` ### 4.Code #### Main function of block server #### kvm_blk.c * kvm_blk_server_init * 1 Init block server's Server-terminal * 2 Call by vl.c -> main() * kvm_blk_client_init * 1 Init block server's Client-terminal * 2 Call by vl.c -> main() * 3 Call by migration/cuju-ft-trans-file.c -> cuju_ft_trans_close() #### kvm_blk_client.c * kvm_blk_aio_readv * 1 Get the read request from VM and send to block server's Server-termial * 2 Call from migration/event-tap.c -> blk_aio_preadv_proxy() * kvm_blk_aio_write * 1 Get the write request from VM and send to block server's Server-termial * 2 Call from hw/block/virtio-blk.c -> submit_requests() * kvm_blk_epoch_timer * 1 Do the epoch state of fault tolerance * 2 Call from migration/migration.c -> migrate_timer() * kvm_blk_epoch_commit * 1 Do the commit state of fault tolerance * 2 Call from migration/migration.c -> kvmft_flush_output() * kvm_blk_notify_ft * 1 Start the fault tolerance mode * 2 migration/migration.c -> migration_thread() * kvm_blk_save_pending_request * 1 Call from hw/block/virtio-blk.c -> submit_requests() #### kvm_blk_serv.c * Watch the [HackMD](https://hackmd.io/@QvGjH100RCOnbhPOvK9CrQ/rJIu-yuUX/%2FZf0-nWixRzytkgLGeM7gnQ?type=book) from Cuju ### Interface of block server * vl.c * main() Main function for block server init * migration/event-tap.c * blk_aio_preadv_proxy() Read request will intercept by this function from hw/block/virtio-blk.c -> submit_requests() * migration/migration.c * migration_thread() * Init the fault tolerance of every struct * migrate_timer() * Do the `Snapshot` and `Transfer` state of fault tolerance * migrate_run() * Do the `Run` state of fault tolerance * kvmft_flush_output() * Do the `Commit` state of fault tolerance * send_commit1() * `//printf("%s",s->time_buf)` Can show the time of four state ### 5.Hint - Make sure that backup and block server is in the same node - All the code of block server is in the `/home/ubuntu/cuju_blkserver/blkserver`which is mounted to server1(100.73.42.5) `/home/ubuntu/blkserver_nfs/` folder - `/home/ubuntu/cuju_blkserver/mount.sh` is help to mount the blkserver folder to nfs - Both of the server(100.73.42.5 and 100.73.42.6) use same path for block server

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.