# Cuju on CentOS 7
[CentOS 7 execute Cuju](https://hackmd.io/@4jocGoC0SRi106skGiCKfw/SyrfJW9pL)
## QEMU and Libvirt Version and Source
### source RPM
qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64
http://vault.centos.org/centos/7.8.2003/virt/Source/kvm-common/qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm
libvirt-libs-4.5.0-33el7_8.1.x86_64
http://buildlogs-seed.centos.org/c7.2003.u.x86_64/libvirt/20200512162224/4.5.0-33.el7_8.1.i386/libvirt-4.5.0-33.el7_8.1.src.rpm
QEMU SRPM rebuild Reference:
https://github.com/Akiko97/qemu-kvm-ev
### Binary RPM Install
## QEMU Compile and Install
### Build QEMU(2.12) source rpm on CentOS7
### Pre install
```
$ sudo yum install -y rpm-build pkgconfig
$ sudo yum install -y gnutls-devel cyrus-sasl-devel libaio-devel pciutils-devel libiscsi-devel ncurses-devel libattr-devel libusbx-devel usbredir-devel texinfo spice-protocol spice-server-devel libcacard-devel nss-devel libseccomp-devel libcurl-devel libssh2-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel systemtap-sdt-devel libjpeg-devel libpng-devel libuuid-devel bluez-libs-devel brlapi-devel check-devel libcap-devel rdma-core-devel gperftools-devel iasl lzo-devel snappy-devel numactl-devel libgcrypt-devel device-mapper-multipath-devel systemd-devel libcap-ng-devel libxkbcommon-devel libepoxy-devel libdrm-devel mesa-libgbm-devel
$ sudo yum install zlib-devel SDL-devel gnutls-devel cyrus-sasl-devel libtool libaio-devel pciutils-devel pulseaudio-libs-devel libiscsi-devel ncurses-devel libattr-devel libusbx-devel usbredir-devel texinfo spice-protocol spice-server-devel libseccomp-devel gperftools-devel libcurl-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel systemtap systemtap-sdt-devel nss-devel libjpeg-devel libpng-devel libuuid-devel bluez-libs-devel brlapi-devel check-devel libcap-devel pixman-devel rdma-core-devel cpp lzo-devel snappy-devel libssh2-devel kernel-devel telnet tigervnc
```
### Build source code
```
$ wget http://vault.centos.org/centos/7.8.2003/virt/Source/kvm-common/qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm
$ rpm -i qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm
$ cd SPECS
$ rpmbuild -ba qemu-kvm.spec
$ cd ../BUILD
qemu-2.12.0
```
### Build Cuju-centOS
* Base on the QEMU source (2.12.0) for CentOS 7
* TODO: Complete the Cuju function
```
$ git clone https://github.com/coldfunction/Cuju-centOS.git
$ cd Cuju-centOS
$ ./build_configure.sh /usr /usr/lib64 /etc /var /usr/libexec /usr/share/doc/qemu-kvm qemu-kvm x86_64 qemu-kvm-ev-2.12.0-44.1.el7_8.1 '-O2 -g -pipe -Wall -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' disable enable disable enable enable enable enable enable enable enable enable enable enable enable enable enable disable --target-list=x86_64-softmmu
$ make
```
## Libvirt Compile and Install, Live migration Test
首先移除centos預設的libvirt與刪除殘留的libvirt檔案
```shell=
sudo yum remove libvirt*
sudo find / -name libvirt
sudo rm -r (libvirt的位置)
```
### install libvirt(yum)
開始安裝libvirt的部份，因為yum上提供的版本與我們需求的相同，所以這邊採用yum install 的方式去進行
```shell=
sudo yum install libvirt libvirt-client
```
### install libvirt(source rpm)
一開始要先安裝必要的套件
```shell=
sudo yum install rpm-build gettext-devel readline-devel ncurses-devel libattr-devel libblkid-devel augeas sanlock-devel libpcap-devel libnl3-devel avahi-devel cyrus-sasl-devel libacl-devel parted-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel numactl-devel libcap-ng-devel fuse-devel netcf-devel libcurl-devel audit-libs-devel systemtap-sdt-devel dbus-devel scrub vim autoconf automake libtool gcc libxml2-devel libtasn1-devel libtasn1-devel systemd-devel libpciaccess-devel yajl-devel libselinux-devel device-mapper-devel gnutls-devel
```
下載libvirt的src.rpm
```shell=
wget http://buildlogs-seed.centos.org/c7.2003.u.x86_64/libvirt/20200512162224/4.5.0-33.el7_8.1.i386/libvirt-4.5.0-33.el7_8.1.src.rpm
```
解開src.rpm
```shell=
rpm -i libvirt-4.5.0-33.el7_8.1.src.rpm
```
解開後的資料會存放在\~/rpmbuild中
SOURCE存放原始碼的(包括patch檔)、SPECS存放了如何進行rpm編譯的文件
這邊需要對SPECS資料夾中的檔案進行修改，因為預設的內容中有無法支援的檔案系統存在，所以需要將其排除
```shell=
cd ~/rpmbuild/SPECS
vim libvirt.spec
```
在這段程式碼後加入
```=
# Then the secondary host drivers, which run inside libvirtd
%if 0%{?fedora} || 0%{?rhel} >= 7
%define with_storage_rbd 0%{!?_without_storage_rbd:1}
%else
%define with_storage_rbd 0
```
```=
%define with_storage_rbd 0
```
修改完畢後即可進行編譯
```shell=
rpmbuild -ba libvirt.spec
```
編譯完畢後rpm與srpm檔案會分別存放在\~/rpmbuild/RPMS/x86_64/與\~/rpmbuild/SRPMS，這時即可安裝資料夾中的rpm了
```shell=
cd ~/rpmbuild/RPMS
sudo yum localinstall libvirt-*.rpm
```
### install libvirt(source code)
安裝必要的套件
```shell=
sudo yum install git vim libtool gcc make libxml2-devel gnutls-devel device-mapper-devel python-devel libnl-devel patch libpciaccess-devel yajl-devel
```
安裝完畢後下載libvirt的source code，並切換到指定版本
```shell=
git clone https://github.com/libvirt/libvirt.git
cd libvirt
git checkout -b v4.5-maint origin/v4.5-maint
```
開始進行編譯，如還是有缺少套件請自行安裝
```shell=
sudo ./autogen.sh --prefix=/usr --sysconfdir=/etc --localstatedir=/var
sudo make -j8
sudo make install
sudo ldconfig
```
### 設定libvirt
開始設定libvirtd，要在libvirtd.conf中將下面五行的設定打開後重啟libvirtd並使libvirtd能在開機時自動開啟
```shell=
sudo vim /etc/libvirt/libvirtd.conf
```
* unix_sock_group = "libvirt"
* unix_sock_ro_perms = "0777"
* unix_sock_rw_perms = "0770"
* auth_unix_ro = "none"
* auth_unix_rw = "none"
```shell=
sudo systemctl restart libvirtd
sudo systemctl enable libvirtd
```
並將使用者加入libvirt群組(這邊用的是cuju)
```shell=
sudo usermod -a -G libvirt cuju
```
### start vm
首先如果img檔案是放在nfs系統中時，要打開在selinux中的權限
```shell=
sudo setsebool -P virt_use_nfs on
```
接著建立bridge(本機為eth0，建立br0為例)
```shell=
sudo vim /etc/sysconfig/network-scripts/ifcfg-br0
```
```
DEVICE="br0"
BOOTPROTO="static"
IPADDR="192.168.77.19"
NETMASK="255.255.255.0"
GATEWAY="192.168.77.7"
DNS1="140.96.254.98"
ONBOOT="yes"
TYPE="Bridge"
NM_CONTROLLED="no"
```
```shell=
sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0
```
```
TYPE="Ethernet"
BOOTPROTO="none"
DEVICE="eth0"
ONBOOT="yes"
NM_CONTROLLED=no
BRIDGE=br0
```
重啟網路
```shell=
sudo systemctl restart network
```
重啟後可以使用ifconfig查看是否成功建立birdge
設定完畢後即可開始建立vm，首先先建立vm的xml檔
```shell=
sudo vim test.xml
```
這邊是範例的xml
```xml=
<domain type='kvm'>
<name>test</name>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/nfs/Ubuntu20G-1604.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:64:21:0e'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'>
<listen type='address'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</memballoon>
</devices>
</domain>
```
利用xml來創立vm並查看
```shell=
sudo virsh define test.xml
sudo virsh list --all
```
創建成功後即可開啟vm
```shell=
sudo virsh start test
```
這時即可使用vncviewer進入vm開始操作
```shell=
vncviewer :0&
```
如果vm使用上都沒有問題的話即可關閉vm，開始進行live migration設定與測試
```shell=
sudo virsh shutdown test
```
### live migration in libvirt
這邊首先要注意兩台vm的hostname不能相同
查看方式為
```
hostnamectl
```
修改方式為(以修改為cuju為例)
```
sudo hostnamectl set-hostname cuju
```
以及需要設定讓兩台host都知道對方以便傳輸資料
```shell=
sudo vim /etc/hosts
```
ip位置與hostname請自行根據自己的電腦去設定
開啟firewall上libvirt live migration所需要的port
```shell=
sudo firewall-cmd --zone=public --add-port=49152-49261/tcp --permanent
```
這時就設定完畢，可以開始測試live migration
同樣開啟vm
```shell=
sudo virsh start test
vncviewer :0&
```
在使用到一半時可以在host端輸入(vm名稱與ip位置請自行設定)
```shell=
sudo virsh migrate --live --unsafe test qemu+ssh://cuju@192.168.77.18/system
```
### 範例演示
還未開啟vm的狀況，目前vm的設定是存在左方的host
啟動vm
vm僅在左方的host顯示
進行live migration
live migration後可以看到左邊的host中的vm呈現關機的狀態，右方的host多出了test的vm
接著從右方host遷移到左方的host
可以發現vm重新回到左方的host
### libvirt參考網址
[Akiko97/qemu-kvm-ev](https://github.com/Akiko97/qemu-kvm-ev)
[How to Create A Network Bridge on CentOS 7 / RHEL 7](https://www.itzgeek.com/how-tos/mini-howtos/create-a-network-bridge-on-centos-7-rhel-7.html)
[Red Hat / CentOS Linux 7 查詢、更改主機名稱設定教學](https://blog.gtwang.org/linux/redhat-centos-7-change-hostname-tutorial/)
[libvirt 問題解決記錄集](https://kknews.cc/zh-tw/code/qbm6aro.html)
[kvm错误整理](https://blog.csdn.net/bbwangj/article/details/77963788)
[SELinux](https://wiki.centos.org/zh-tw/HowTos/SELinux)
[CentOS 7 關閉防火牆及 SELinux](https://www.opencli.com/linux/centos-7-disable-firewalld-selinux)
[kvm:訪問NFS池時，Virt-manager權限被拒绝](https://t.codebug.vip/questions-949843.htm)
Sign in via Google
Sign in via Facebook
Sign in via X(Twitter)
Sign in via GitHub
Sign in via Dropbox
Sign in with Wallet
Connect another wallet