# kolla-ansible 進行 baremetal
version: stein
OS: CentOS7

:::warning
**注意事項**
使用 bill 提供的 kolla-ansible 工具來建置 請先完成 環境設置
* bill 的 kolla-ansible [參考文件](https://github.com/kjelly/kolla-doc/blob/master/deploy_openstack.md)
* offline registry 可以參考 [這裡](https://github.com/kjelly/kolla-doc/blob/master/use_prebuild_images.md)
* KVM host 要有 docker library > 2.0.0

# test
python -c "import docker; print docker.__version__"
# install 
pip install docker

* docker-py library 與 docker library 會衝突請先完全移除 docker-py library 和 docker library 再重新安裝 docker library

pip uninstall docker
pip uninstall docker-py
rpm -ql python-docker-py | xargs rm -rf

* 當使用 [build_a_lab](https://github.com/MilkBotttle/build_a_lab) 或是 vagrant 建立 baremetal node 需要手動移除 vagrant management network 的網卡(通常是使用 vagrant-libvirt 的 nat 網路) 讓 pxe 可以正常運作
:::

## 準備 ironic image
OS image 原版 [download](https://images.rdoproject.org/stein/rdo_trunk/current-tripleo/overcloud-full.tar)

ironic agent 原版 [download](https://images.rdoproject.org/stein/rdo_trunk/current-tripleo/ironic-python-agent.tar)

放到要裝 kolla-ansible + ironic 的 node 上

## 使用單網卡與 kolla-ansible 所需設定

1. 安裝 ovs
從公司NAS下載編譯好的版本複製到 node 上安裝

wget 'http://172.20.0.22:8080/share.cgi?ssid=0Mry0j9&fid=0Mry0j9&ep=LS0tLQ==' -O openvswitch-2.11.1-1.el7.x86_64.rpm
yum localinstall openvswitch-2.11.1-1.el7.x86_64.rpm

如果有缺乏相依套件請在有網路的環境使用**一樣的**版本系統準備缺少的套件

mkdir ovs
cd ovs
wget 'http://172.20.0.22:8080/share.cgi?ssid=0Mry0j9&fid=0Mry0j9&ep=LS0tLQ==' -O openvswitch-2.11.1-1.el7.x86_64.rpm
yumdownloader $(yum deplist openvswitch-2.11.1-1.el7.x86_64.rpm | grep provider | awk '{print $2}' | sort -u)

複製整個資料夾到缺套件的 node

cd ovs
yum localinstall *

2. 設定網路環境
:::info
* 在這裡使用的網卡叫做 eth1
* ovs-bridge 叫 brex (名稱可以隨意，但是不可以使用特殊字元例如 '-' )
* 使用的 IP range 是 192.168.24.0/24, 使用的 IP 是 192.168.24.1
:::

設定eth1
ifcfg-eth1

ONBOOT=yes
DEVICE=eth1
NM_CONTROLLED=no
TYPE=OVSPort
DEVICETYPE=OVS
OVS_BRIDGE=brex
NAME="eth1"
BOOTPROTO=none

設定brex
ifcfg-brex

DEVICE=brex
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.24.2
NETMASK=255.255.255.0
HOTPLUG=no

3. 啟動網卡

ifup eth1
ifup brex

4. 加上 br-int

ovs-vsctl add-br br-int

5. 設定 selinux 為 permissive
編輯 /etc/selinux/config

SELINUX=permissive

重開機

6. 在後續 globals.yml 設定上網路部分要加上

NEUTRON_BRIDGE: "brex"
neutron_bridge_name: "brex"
network_interface: "brex"
neutron_external_interface: "eth1"
enable_openvswitch: "no"

## 建立 baremetal-ansible-docker container

cd ~/kolla-ansible-docker
sudo ./_run.sh baremetal stein
sudo ./_config.sh baremetal

## kolla-ansible 啟用 ironic
1. 設定 globals.conf
:::info
在 offline 環境要加上 docker_registry: "[registry]:[port]"
:::

/etc/baremetal/globals.conf

config_strategy: "COPY_ALWAYS"
kolla_base_distro: "centos"
kolla_install_type: "source" 
openstack_release: "stein"
kolla_internal_vip_address: "192.168.24.2"
network_interface: "brex"
NEUTRON_BRIDGE: "brex"
neutron_bridge_name: "brex"
neutron_external_interface: "eth1"
enable_haproxy: "no"
enable_fluentd: "no"
enable_horizon: "no"
enable_ironic: "yes"
enable_ironic_ipxe: "yes"
enable_ironic_pxe_uefi: "yes"
enable_openvswitch: "no"
nova_compute_virt_type: "kvm" #如果是VM環境設定成 qemu
ironic_dnsmasq_interface: "brex"
ironic_dnsmasq_dhcp_range: 192.168.24.240,192.168.24.250
ironic_cleaning_network: "ctlplane"

> 補充：
裝完後neutron_server若出現找不到firewall的類似錯誤，多加以下兩個選項
enable_neutron_fwaas: "yes"
enable_neutron_vpnaas: "yes"

2. 準備 ironic introspection 使用的 image
> Note: image 在準備時已經放到 node 上

mkdir -p /etc/baremetal/config/ironic
tar xvf ironic-python-agent.tar -C /etc/baremetal/config/ironic
cd /etc/baremetal/config/ironic/
mv ironic-python-agent.kernel ironic-agent.kernel
mv ironic-python-agent.initramfs ironic-agent.initramfs

## 設定儲存 introspection 的資料
可以跳過只是 introspection 不能存資料而已 不影響部屬 OS
1. 建立 swift 用的硬碟
這邊 swift 會用來存 glance image 和 ironic node 資訊可以依照需求改大小 不一定要設定 10G

free_device=$(losetup -f)
fallocate -l 10G /tmp/swift #
losetup $free_device /tmp/swift
parted $free_device -s -- mklabel gpt mkpart KOLLA_SWIFT_DATA 1 -1
sudo mkfs.xfs -f -L d0 ${free_device}p1

2. 設定 /etc/baremetal/globals.yml
啟用 swift

enable_swift: "yes"

3. 準備 swift 設定

`NODE IP` 就是 `kolla_internal_vip_address` 要修改成符合環境

`KOLLA_SWIFT_BASE_IMAGE` 根據版本修改

NODE=192.168.24.2
KOLLA_SWIFT_BASE_IMAGE="192.168.24.2:4000/kolla/centos-source-swift-base:stein"
SWIFT_CONFIG_DIR="/etc/baremetal/config/swift"
mkdir -p $SWIFT_CONFIG_DIR

# Object ring
docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
    /etc/kolla/config/swift/object.builder create 10 1 1

docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
  /etc/kolla/config/swift/object.builder add r1z1-$NODE:6000/d0 1

# Account ring
docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
    /etc/kolla/config/swift/account.builder create 10 1 1

docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
  /etc/kolla/config/swift/account.builder add r1z1-$NODE:6001/d0 1

# Container ring
docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
    /etc/kolla/config/swift/container.builder create 10 1 1
docker run \
  --rm \
  -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
  $KOLLA_SWIFT_BASE_IMAGE \
  swift-ring-builder \
    /etc/kolla/config/swift/container.builder add r1z1-$NODE:6002/d0 1

for ring in object account container; do
  docker run \
    --rm \
    -v $SWIFT_CONFIG_DIR/:/etc/kolla/config/swift/ \
    $KOLLA_SWIFT_BASE_IMAGE \
    swift-ring-builder \
      /etc/kolla/config/swift/${ring}.builder rebalance;
done

4. 增加 ironic-inspector 的設定

mkdir -p /etc/baremetal/config/ironic-inspector
touch /etc/baremetal/config/ironic-inspector/inspector.conf

ironic-inspector password 可以在 passwords.yml 找到

cat /etc/baremetal/passwords.yml | grep ironic_inspector_keystone_password
ironic_inspector_keystone_password: sFZkdJtDxgaJltBHTT0qUud05fCPg9OdDCAG0XrD

加入設定在 /etc/baremetal/config/ironic-inspector/inspector.conf

[processing]
store_data = swift

[swift]
auth_url=http://192.168.24.2:5000
auth_type=password
password=sFZkdJtDxgaJltBHTT0qUud05fCPg9OdDCAG0XrD
project_domain_name= Default
project_name=service
user_domain_name = Default
username=ironic-inspector

如果要啟用 discovery 把未註冊的 node 註冊成 enroll 再額外增加設定 (這在 VM 環境中會不正常因為 node 因為 vm 無法提供 bmc)

[processing]
node_not_found_hook = enroll

[discovery]
enroll_node_driver = ipmi

:::info
沒有啟用 discovery 需要將 node 的 PXE port mac address 註冊到 baremetal 服務中才可以使用 introspection

openstack baremetal port create --node [baremetal node id] mac_address

:::

## 準備 inventory 和 ssh
設定連到 host 免密碼登入

docker exec -it baremetal-ansible-docker bash
ssh-copy-id root@192.168.24.1

使用 all-in-one inventory 來修改

docker exec -it baremetal-ansible-docker bash 
cp /kolla-ansible/ansible/inventory/all-in-one /etc/kolla-ansible-docker/inventory

編輯 inventory 修改下面 group 的內容

[control]
ka ansible_user=root ansible_password=password ansible_host=192.168.24.1 ansible_become=true validate_certs=False host_key_checking=False

[network]
ka ansible_user=root ansible_password=password ansible_host=192.168.24.1 ansible_become=true validate_certs=False host_key_checking=False

[compute]
ka ansible_user=root ansible_password=password ansible_host=192.168.24.1 ansible_become=true validate_certs=False host_key_checking=False

[storage]
ka ansible_user=root ansible_password=password ansible_host=192.168.24.1 ansible_become=true validate_certs=False host_key_checking=False

[monitoring]
ka ansible_user=root ansible_password=password ansible_host=192.168.24.1 ansible_become=true validate_certs=False host_key_checking=False

## 部屬 kolla-ansible 啟用 ironic

docker exec -it baremetal-ansible-docker bash
ka bootstrap-servers # 這裡會重開 docker 重開完之後再執行一次
ka precheck
ka deploy
ka post-deploy
cd
ln -s /etc/kolla/admin-openrc.sh .

## 使用 Baremetal service
以下操作都在 baremetal-ansible-docker container 中
1. 下載 OS image 或是使用客製化的 image

從[這裡](https://images.rdoproject.org/stein/rdo_trunk/current-tripleo/)下載原版
* ironic-python-agent.tar
* overcloud-full.tar
3. 上傳 image 到 glance

MY_VMLINUZ_UUID=$(openstack image create image-kernel --public --disk-format aki --container-format aki --file overcloud-full.vmlinuz -f value -c id)
MY_INITRD_UUID=$(openstack image create image-ramdisk --public --disk-format ari --container-format ari --file overcloud-full.initrd -f value -c id)
openstack image create overcloud --public --disk-format qcow2 --container-format bare --property kernel_id=$MY_VMLINUZ_UUID --property  ramdisk_id=$MY_INITRD_UUID --file overcloud-full.qcow2

openstack image create bm-deploy-ramdisk --public --disk-format aki --container-format aki --file ironic-agent.initramfs
openstack image create bm-deploy-kernel --public --disk-format ari --container-format ari --file ironic-agent.kernel

4. create neutron flat network

openstack network create --provider-network-type flat --provider-physical-network physnet1 ctlplane
openstack subnet create ctlplane-subnet --host-route destination='169.254.169.254/32',gateway='192.168.24.2' --allocation-pool start=192.168.24.10,end=192.168.24.20 --network ctlplane --dhcp --subnet-range 192.168.24.0/24

5. 建立 nova baremetal flavor

 openstack flavor create baremetal --vcpus 1 \
                                   --ram 1024 \
                                   --disk 40 \
                                   --property resources:CUSTOM_BAREMETAL=1 \
                                   --property resources:VCPU=0 \
                                   --property resources:MEMORY_MB=0 \
                                   --property resources:DISK_GB=0 \
                                   --property capabilities:boot_option="local"

6. 建立 keystone keypair

openstack keypair create default > default.pri

7. 註冊baremetal node

# deploy_kernel = bm-deploy-kernel
# deploy_ramdisk = bm-deploy-ramdisk
openstack baremetal node create --driver ipmi --name ctrl1 \
                                --driver-info ipmi_address=192.168.122.1 \
                                --driver-info ipmi_port=55556 \
                                --driver-info ipmi_username=admin \
                                --driver-info ipmi_password=password \
                                --driver-info deploy_kernel=34b9eb6b-2b3d-489b-b722-ca6ad7ba69eb \
                                --driver-info deploy_ramdisk=9e71eed4-a46e-456a-99b3-1ab85dd5492d \
                                --resource-class baremetal

8. 新增硬體資訊
根據實際硬體來寫, 或者等同於 flavor 所設定的值

openstack baremetal node set --property memory_mb=4096 \
                             --property cpu_arch=x86_64 \
                             --property local_gb=50 \
                             --property cpus=1 \
                             --property capabilities="boot_option:local" \
                             ctrl1

建立 baremetal node pxe port

openstack baremetal port create --node [baremetal-node-uuid] [pxe-port-mac-address]

或者使用 ironic introspection 自動填上硬體資訊跟建立 pxe port

openstack baremetal node manage ctrl1
openstack baremetal introspection start ctrl1

9. 部屬baremetal機器

openstack baremetal node manage ctrl1
openstack baremetal node provide ctrl1
openstack server create ctrl1 --key-name default \
                              --network ctlplane \
                              --flavor baremetal \
                              --image overcloud  \
                              --wait