###### tags: `vagrant` `libvirt` `kvm` `vm` # Vagrant 使用 libvirt provider ## 安裝 [安裝文件](https://github.com/vagrant-libvirt/vagrant-libvirt#installation) KVM libvirt vagrant vagrant-libvirt ## 使用 [文件連結](https://github.com/vagrant-libvirt/vagrant-libvirt) ## Vagrant Provision * 使用Ansible調整VM Vagrantfile ``` Vagrant.configure("2") do |config| config.vm.synced_folder ".", "/vagrant", type: "rsync" config.vm.define :test_vm do |test_vm| test_vm.vm.box = "generic/ubuntu1804" end config.vm.provision "ansible" do |ansible| ansible.playbook = "playbook.yml" end end ``` ## 製作Vagrant Box 1. 在VM上做完想做的事, 建立使用者 vagrant 並設定 sudo 免密碼後關機 2. 建立 vagrant box 資料夾移動到 vagrant box 資料夾 ``` mkdir vagrant cd vagrant ``` 3. 找出vm disk ``` virsh domblklist [domain_name] Target Source ------------------------------------------------ vda /var/lib/libvirt/images/AnsibleTraning_test_vm.img ``` 4. 複製image到vagrantbox資料夾命名box.img ``` cp /var/lib/libvirt/images/AnsibleTraning_test_vm.img box.img ``` 5. 檢查是否有backing file ``` qemu-img info box.img | grep 'backing file:' | cut -d ':' -f2 | xargs ``` 6. 有backing file則執行rebase ``` qemu-img rebase -p -b "" box.img ``` 7. 執行 box.img 初始化 ``` virt-sysprep --no-logfile -a box.img ``` 8. 建立insecure ssh key 和 reconfigure ssh server 的 script rc.local Vagrant 所使用的 insecure key 是固定的不需要修改, reconfig 的方法因系統而異, 這邊是 ubuntu ``` #!/bin/bash mkdir -p /home/vagrant/.ssh echo ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key > /home/vagrant/.ssh/authorized_keys chown -R vagrant:vagrant /home/vagrant/.ssh chmod 0600 /home/vagrant/.ssh/authorized_keys dpkg-reconfigure openssh-server rm $0 # DELETE SELF ``` 9. 將 rc.local 放到 box.img 中 ``` virt-copy-in -a box.img rc.local /etc ``` 10. 建立 meatadata.json 找出 image size ``` qemu-img info --output=json box.img | awk '/virtual-size/{s=int($2)/(1024^3); print (s == int(s)) ? s : int(s)+1 }' ``` ``` { "provider": "libvirt", "format": "qcow2", "virtual_size": $IMG_SIZE } ``` 11. 建立 Vagrantfile ``` Vagrant.configure("2") do |config| config.vm.provider :libvirt do |libvirt| libvirt.driver = "kvm" libvirt.host = "" libvirt.connect_via_ssh = false libvirt.storage_pool_name = "default" end end user_vagrantfile = File.expand_path('../_include/Vagrantfile', __FILE__) load user_vagrantfile if File.exists?(user_vagrantfile) ``` 12. 打包 box ``` tar zcvf vagrant.box box.img metadata.json Vagrantfile ``` 打包好之後就可以使用 box add 匯入 ## 多node Vagrantfile vagrant-libvirt 建立網路很難用使用virt-manager設定好網路後再開啟 box 比較方便 ```ruby= # -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure("2") do |config| # Global config.vm.box = "centos/7" # Global ssh key config config.ssh.insert_key = false config.ssh.private_key_path = ["keys/key", "~/.vagrant.d/insecure_private_key"] # vagrant user config.vm.provision "file", source: "keys/key.pub", destination: "~/.ssh/authorized_keys" config.vm.provision "file", source: "keys/key.pub", destination: "~/.ssh/id_rsa.pub" config.vm.provision "file", source: "keys/key", destination: "/home/vagrant/.ssh/id_rsa" config.vm.provision "shell", inline: <<-EOC sudo sed -i -e "\\#PasswordAuthentication yes# s#PasswordAuthentication yes#PasswordAuthentication no#g" /etc/ssh/sshd_config sudo systemctl restart sshd EOC config.vm.provision "ansible" do |ansible| ansible.playbook = "init.yml" end # Deploy config.vm.define "ka" do |ka| ka.vm.synced_folder ".", "/vagrant", type: "nfs" ka.vm.hostname = "ka" ka.vm.provider :libvirt do |domain| domain.memory = 1024*4 domain.volume_cache = "unsafe" domain.machine_virtual_size = 100 end ka.vm.network :private_network, :libvirt__network_name => "pxe" ka.vm.provision "ansible" do |ansible| ansible.playbook = "init-ka.yml" end end # ctrl-nodes (1..3).each do |i| config.vm.define "ctrl-#{i}" do |node| node.vm.synced_folder ".", "/vagrant", type: "nfs" node.vm.hostname = "ctrl-#{i}" node.vm.provider :libvirt do |domain| domain.memory = 1024*4 domain.volume_cache = "unsafe" domain.machine_virtual_size = 50 end node.vm.network :private_network, :libvirt__network_name => "pxe" node.vm.network :private_network, :libvirt__network_name => "internal_api" node.vm.network :private_network, :libvirt__network_name => "external" node.vm.network :private_network, :libvirt__network_name => "storage" end end # com-nodes (1..2).each do |i| config.vm.define "com-#{i}" do |node| node.vm.synced_folder ".", "/vagrant", type: "nfs" node.vm.hostname = "com-#{i}" node.vm.provider :libvirt do |domain| domain.memory = 1024*4 domain.volume_cache = "unsafe" domain.machine_virtual_size = 50 end node.vm.network :private_network, :libvirt__network_name => "pxe" node.vm.network :private_network, :libvirt__network_name => "internal_api" node.vm.network :private_network, :libvirt__network_name => "external" node.vm.network :private_network, :libvirt__network_name => "storage" node.vm.network :private_network, :libvirt__network_name => "tenant" end end end ``` 關閉預設sync 資料夾設定 ``` config.vm.synced_folder ".", "/vagrant", disabled: true ``` # 在開啟自訂硬碟空間的 box 後 KVM disk resize 在這邊以 xfs 為範例 1. virsh resize ``` virsh domblklist centos7 qemu-img resize /var/lib/libvirt/images/centos7.qcow2 +20G ``` 2. install growpart tool in vm ``` yum install -y cloud-utils-growpart lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 100G 0 disk └─vda1 253:1 0 40G 0 part / growpart /dev/vda 1 CHANGED: partition=1 start=2048 old: size=83884032 end=83886080 new: size=209713119 end=209715167 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 100G 0 disk └─vda1 253:1 0 100G 0 part / ``` 3. growfs ``` xfs_growfs / meta-data=/dev/vda1 isize=512 agcount=4, agsize=2621376 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=10485504, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=5119, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 10485504 to 26214139 ``` ## 已知問題 1. 自己打包的 box 在 vagrant up 中可能出現 NFS mount 卡住 解決方法: firewall 允許使用 nfs3, 或是關閉 firewall ``` # firewall-cmd --permanent --add-service=nfs3 success # firewall-cmd --reload success ```