# Cuju on CentOS 7

[CentOS 7 execute Cuju](https://hackmd.io/@4jocGoC0SRi106skGiCKfw/SyrfJW9pL)

## QEMU and Libvirt Version and Source

### source RPM
qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64
http://vault.centos.org/centos/7.8.2003/virt/Source/kvm-common/qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm

libvirt-libs-4.5.0-33el7_8.1.x86_64
http://buildlogs-seed.centos.org/c7.2003.u.x86_64/libvirt/20200512162224/4.5.0-33.el7_8.1.i386/libvirt-4.5.0-33.el7_8.1.src.rpm

QEMU SRPM rebuild Reference:
https://github.com/Akiko97/qemu-kvm-ev

### Binary RPM Install

## QEMU Compile and Install
### Build QEMU(2.12) source rpm on CentOS7

### Pre install

$ sudo yum install -y rpm-build pkgconfig
$ sudo yum install -y gnutls-devel cyrus-sasl-devel libaio-devel pciutils-devel libiscsi-devel ncurses-devel libattr-devel libusbx-devel usbredir-devel texinfo spice-protocol spice-server-devel libcacard-devel nss-devel libseccomp-devel libcurl-devel libssh2-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel systemtap-sdt-devel libjpeg-devel libpng-devel libuuid-devel bluez-libs-devel brlapi-devel check-devel libcap-devel rdma-core-devel gperftools-devel iasl lzo-devel snappy-devel numactl-devel libgcrypt-devel device-mapper-multipath-devel systemd-devel libcap-ng-devel libxkbcommon-devel libepoxy-devel libdrm-devel mesa-libgbm-devel
$ sudo yum install zlib-devel SDL-devel gnutls-devel cyrus-sasl-devel libtool libaio-devel pciutils-devel pulseaudio-libs-devel libiscsi-devel ncurses-devel libattr-devel libusbx-devel usbredir-devel texinfo spice-protocol spice-server-devel libseccomp-devel gperftools-devel libcurl-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel systemtap systemtap-sdt-devel nss-devel libjpeg-devel libpng-devel libuuid-devel bluez-libs-devel brlapi-devel check-devel libcap-devel pixman-devel rdma-core-devel cpp lzo-devel snappy-devel libssh2-devel kernel-devel telnet tigervnc

### Build source code

$ wget http://vault.centos.org/centos/7.8.2003/virt/Source/kvm-common/qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm
$ rpm -i qemu-kvm-ev-2.12.0-44.1.el7_8.1.src.rpm
$ cd SPECS
$ rpmbuild -ba qemu-kvm.spec
$ cd ../BUILD
qemu-2.12.0

### Build Cuju-centOS
* Base on the QEMU source (2.12.0) for CentOS 7
* TODO: Complete the Cuju function

$ git clone https://github.com/coldfunction/Cuju-centOS.git
$ cd Cuju-centOS
$ ./build_configure.sh /usr /usr/lib64 /etc /var /usr/libexec /usr/share/doc/qemu-kvm qemu-kvm x86_64 qemu-kvm-ev-2.12.0-44.1.el7_8.1 '-O2 -g -pipe -Wall  -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' disable enable disable enable enable enable enable enable enable enable enable enable enable enable enable enable disable --target-list=x86_64-softmmu
$ make

## Libvirt Compile and Install, Live migration Test
首先移除centos預設的libvirt與刪除殘留的libvirt檔案

sudo yum remove libvirt*
sudo find / -name libvirt
sudo rm -r (libvirt的位置)

### install libvirt(yum)
開始安裝libvirt的部份，因為yum上提供的版本與我們需求的相同，所以這邊採用yum install 的方式去進行

sudo yum install libvirt libvirt-client

### install libvirt(source rpm)
一開始要先安裝必要的套件

sudo yum install rpm-build gettext-devel readline-devel ncurses-devel libattr-devel libblkid-devel augeas sanlock-devel libpcap-devel libnl3-devel avahi-devel cyrus-sasl-devel libacl-devel parted-devel librados2-devel librbd1-devel glusterfs-api-devel glusterfs-devel numactl-devel libcap-ng-devel fuse-devel netcf-devel libcurl-devel audit-libs-devel systemtap-sdt-devel dbus-devel scrub vim autoconf automake libtool gcc libxml2-devel libtasn1-devel libtasn1-devel systemd-devel libpciaccess-devel yajl-devel libselinux-devel device-mapper-devel gnutls-devel

下載libvirt的src.rpm

wget  http://buildlogs-seed.centos.org/c7.2003.u.x86_64/libvirt/20200512162224/4.5.0-33.el7_8.1.i386/libvirt-4.5.0-33.el7_8.1.src.rpm

解開src.rpm

rpm -i libvirt-4.5.0-33.el7_8.1.src.rpm

解開後的資料會存放在\~/rpmbuild中
SOURCE存放原始碼的(包括patch檔)、SPECS存放了如何進行rpm編譯的文件
這邊需要對SPECS資料夾中的檔案進行修改，因為預設的內容中有無法支援的檔案系統存在，所以需要將其排除

cd ~/rpmbuild/SPECS
vim libvirt.spec

在這段程式碼後加入

# Then the secondary host drivers, which run inside libvirtd
%if 0%{?fedora} || 0%{?rhel} >= 7
    %define with_storage_rbd      0%{!?_without_storage_rbd:1}
%else
    %define with_storage_rbd      0

%define with_storage_rbd      0

修改完畢後即可進行編譯

rpmbuild -ba libvirt.spec

編譯完畢後rpm與srpm檔案會分別存放在\~/rpmbuild/RPMS/x86_64/與\~/rpmbuild/SRPMS，這時即可安裝資料夾中的rpm了

cd ~/rpmbuild/RPMS
sudo yum localinstall libvirt-*.rpm

### install libvirt(source code)
安裝必要的套件

sudo yum install git vim libtool gcc make libxml2-devel gnutls-devel device-mapper-devel python-devel libnl-devel patch libpciaccess-devel yajl-devel

安裝完畢後下載libvirt的source code，並切換到指定版本

git clone https://github.com/libvirt/libvirt.git
cd libvirt
git checkout -b v4.5-maint origin/v4.5-maint

開始進行編譯，如還是有缺少套件請自行安裝

sudo ./autogen.sh --prefix=/usr --sysconfdir=/etc --localstatedir=/var
sudo make -j8
sudo make install 
sudo ldconfig

### 設定libvirt
開始設定libvirtd，要在libvirtd.conf中將下面五行的設定打開後重啟libvirtd並使libvirtd能在開機時自動開啟

sudo vim /etc/libvirt/libvirtd.conf

* unix_sock_group = "libvirt"
* unix_sock_ro_perms = "0777"
* unix_sock_rw_perms = "0770"
* auth_unix_ro = "none"
* auth_unix_rw = "none"

sudo systemctl restart libvirtd
sudo systemctl enable libvirtd

並將使用者加入libvirt群組(這邊用的是cuju)

sudo usermod -a -G libvirt cuju

### start vm
首先如果img檔案是放在nfs系統中時，要打開在selinux中的權限

sudo setsebool -P virt_use_nfs on

接著建立bridge(本機為eth0，建立br0為例)

sudo vim /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE="br0"
BOOTPROTO="static"
IPADDR="192.168.77.19"
NETMASK="255.255.255.0"
GATEWAY="192.168.77.7"
DNS1="140.96.254.98"
ONBOOT="yes"
TYPE="Bridge"
NM_CONTROLLED="no"

sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0

TYPE="Ethernet"
BOOTPROTO="none"
DEVICE="eth0"
ONBOOT="yes"
NM_CONTROLLED=no
BRIDGE=br0

重啟網路

sudo systemctl restart network

重啟後可以使用ifconfig查看是否成功建立birdge

設定完畢後即可開始建立vm，首先先建立vm的xml檔

sudo vim test.xml

這邊是範例的xml

<domain type='kvm'>
  <name>test</name>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/nfs/Ubuntu20G-1604.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:64:21:0e'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

利用xml來創立vm並查看

sudo virsh define test.xml
sudo virsh list --all


創建成功後即可開啟vm

sudo virsh start test


這時即可使用vncviewer進入vm開始操作

vncviewer :0&

如果vm使用上都沒有問題的話即可關閉vm，開始進行live migration設定與測試

sudo virsh shutdown test

### live migration in libvirt
這邊首先要注意兩台vm的hostname不能相同
查看方式為

hostnamectl


修改方式為(以修改為cuju為例)

sudo hostnamectl set-hostname cuju

以及需要設定讓兩台host都知道對方以便傳輸資料

sudo vim /etc/hosts

ip位置與hostname請自行根據自己的電腦去設定

開啟firewall上libvirt live migration所需要的port

sudo firewall-cmd --zone=public --add-port=49152-49261/tcp --permanent

這時就設定完畢，可以開始測試live migration
同樣開啟vm

sudo virsh start test
vncviewer :0&

在使用到一半時可以在host端輸入(vm名稱與ip位置請自行設定)

sudo virsh migrate --live --unsafe test qemu+ssh://cuju@192.168.77.18/system

### 範例演示
還未開啟vm的狀況，目前vm的設定是存在左方的host


啟動vm


vm僅在左方的host顯示


進行live migration


live migration後可以看到左邊的host中的vm呈現關機的狀態，右方的host多出了test的vm


接著從右方host遷移到左方的host


可以發現vm重新回到左方的host


### libvirt參考網址
[Akiko97/qemu-kvm-ev](https://github.com/Akiko97/qemu-kvm-ev)
[How to Create A Network Bridge on CentOS 7 / RHEL 7](https://www.itzgeek.com/how-tos/mini-howtos/create-a-network-bridge-on-centos-7-rhel-7.html)
[Red Hat / CentOS Linux 7 查詢、更改主機名稱設定教學](https://blog.gtwang.org/linux/redhat-centos-7-change-hostname-tutorial/)
[libvirt 問題解決記錄集](https://kknews.cc/zh-tw/code/qbm6aro.html)
[kvm错误整理](https://blog.csdn.net/bbwangj/article/details/77963788)
[SELinux](https://wiki.centos.org/zh-tw/HowTos/SELinux)
[CentOS 7 關閉防火牆及 SELinux](https://www.opencli.com/linux/centos-7-disable-firewalld-selinux)
[kvm:訪問NFS池時，Virt-manager權限被拒绝](https://t.codebug.vip/questions-949843.htm)