# Advanced Storage for OpenStack Training Homework
This are the procedures to complete the homework for the OpenStack Advanced Storage Training from Red Hat.
## Scenario
Acme Consolidated has engaged with Red Hat to implement an OpenStack environment. They have defined the following implementation criteria.
:::info
**A single RHOSP 13 Controller Node running the core RHOSP services with two RHOSP 13 Compute nodes**
:::
***Solution***
OpenStack 13 Advanced Storage Lab already has this.
#### The workstation node root user should have a properly configured keystonerc_admin file for administrative access to the OpenStack cluster
***Solution***
SSH to the workstation node.
Generat the SSH key
```bash
[root@workstation-d4f5 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:JJtvyHTAjC0bD4fcuDU+oC6Knb9ZYVSF5sN9jz5vWyg root@workstation-d4f5.rhpds.opentlc.com
The key's randomart image is:
+---[RSA 2048]----+
| .o. |
| . O .o |
| X %+.. |
| . & B+ . . |
| . o X S. . o |
| . + * . .. |
|. . + o .E . .|
|oo . o . o... |
|o o.+. +o. |
+----[SHA256]-----+
[root@workstation-d4f5 ~]#
```
Copy the SSH key to control node with passward ==r3dh4t1!==.
```bash
[root@workstation-d4f5 ~]# ssh-copy-id root@ctrl01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@ctrl01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@ctrl01'"
and check to make sure that only the key(s) you wanted were added.
[root@workstation-d4f5 ~]#
```
Copy the keystonerc_admin file from crtl01 to the workstation.
```bash
[root@workstation-d4f5 ~]# scp ctrl01:~/keystonerc_admin .
keystonerc_admin 100% 327 198.7KB/s 00:00
```
:::info
**The workstation node should have the python-openstackclient package installed**
:::
***Solution***
Install OpenStack client
```shell
[root@workstation-d4f5 ~]# yum install -y python2-openstackclient
```
Test out the client and credentials.
```bash
[root@workstation-d4f5 ~]# source keystonerc_admin
[root@workstation-d4f5 ~(keystone_admin)]# openstack service list
+----------------------------------+------------+--------------+
| ID | Name | Type |
+----------------------------------+------------+--------------+
| 202f5c7a826c4fc7b2d2b592559dd03e | swift | object-store |
| 37704169b33b42b6896b68df92929f8b | cinder | volume |
| 3cb9980330ae49178357937e5905e33a | cinderv3 | volumev3 |
| 43ee8cd030e847a590fac6b38dfab0ed | cinderv2 | volumev2 |
| 57f4ff0a11b64af7b748c3f26e9f31e8 | gnocchi | metric |
| 5c518727354a4535a878223ed26e66ae | glance | image |
| 9c7705f7419e4d42b72ca4ea4d37d829 | aodh | alarming |
| a162bde594804b0dac271269093112e9 | keystone | identity |
| bb5ca8b4508a43d8b9cc631b390d7a0c | ceilometer | metering |
| d6cff27922cb43fba287cd0932ce514f | neutron | network |
| e6c2a5ea1e654d2daf54b424286304f5 | nova | compute |
| e91eb168baed40a2a29ef940a03d3dbc | placement | placement |
+----------------------------------+------------+--------------+
[root@workstation-d4f5 ~(keystone_admin)]#
```
:::info
**There should be an user named swift-user in a new project named swift-project created in OpenStack**
:::
***Solution***
Create the OpenStack Project.
```bash
[root@workstation-d4f5 ~(keystone_admin)]# openstack project create --description "Swift Project" swift-project
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Swift Project |
| domain_id | default |
| enabled | True |
| id | bc0d92bdcf264f95b2be622e72e9eb8f |
| is_domain | False |
| name | swift-project |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+
[root@workstation-d4f5 ~(keystone_admin)]#
```
Create the OpenStack user.
```bash
[root@workstation-d4f5 ~(keystone_admin)]# openstack user create --project swift-project --password r3dh4t1! swift-user
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| default_project_id | bc0d92bdcf264f95b2be622e72e9eb8f |
| domain_id | default |
| enabled | True |
| id | 8ad5de3b53cd473ebbd613df09f7af19 |
| name | swift-user |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
[root@workstation-d4f5 ~(keystone_admin)]#
```
Add the user role.
```bash
[root@workstation-d4f5 ~(keystone_admin)]# openstack role add --user swift-user --project swift-project _member_
```
:::info
**The workstation node root user should have a properly configured keystonerc_user file for access to the OpenStack cluster as swift-user**
:::
***Solution***
Create the keystonerc_user file with the following contents.
```bash
export OS_USERNAME=swift-user
export OS_PASSWORD=r3dh4t1!
export OS_AUTH_URL=http://172.16.7.50:5000/v3
export PS1='[\u@\h \W(swift-user)]\$ '
export OS_PROJECT_NAME=swift-project
export OS_REGION_NAME=RegionOne
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
```
:::info
**A Ceph Storage cluster running on 3 nodes, each containing 2 disk devices
The 3 (three) Ceph MON nodes should be configured on different nodes than OSD nodes
Ceph should be configured so that your OPENTLC account has administrative access to ceph
A RADOS gateway running on MON nodes**
:::
***Solution***
**Install and configure docker-distribution**
Install docker-distribution, skopeo and jq:
```bash
[root@workstation ~]# yum -y install docker-distribution skopeo jq
```
Replace the contents of /etc/docker-distribution/registry/config.yml with those shown below:
```yaml=
version: 0.1
log:
fields:
service: registry
storage:
cache:
layerinfo: inmemory
filesystem:
rootdirectory: /var/lib/registry
http:
addr: 0.0.0.0:5000
host: https://workstation.example.com:5000
tls:
certificate: /etc/docker-distribution/my_self_signed_cert.crt
key: /etc/docker-distribution/my_self_signed.key
```
Create a certificate for the Docker registry:
```bash
[root@workstation-d4f5 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/docker-distribution/my_self_signed.key -out /etc/docker-distribution/my_self_signed_cert.crt
```
```bash
Generating a 2048 bit RSA private key
.......+++
..........................................+++
writing new private key to '/etc/mycerts/my_self_signed.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:HK
State or Province Name (full name) []:Hong Kong
Locality Name (eg, city) [Default City]:HK
Organization Name (eg, company) [Default Company Ltd]:Red Hat
Organizational Unit Name (eg, section) []:Lab
Common Name (eg, your name or your server's hostname) []:workstation.example.com
Email Address []:root@workstation.example.com
```
Enable and start the Docker distribution service:
```bash
[root@workstation ~]# systemctl enable docker-distribution --now
```
Configure RHEL to trust the self-signed certificate:
Using the openssl utility, create a PEM formatted version of my_self_signed_cert.crt that was created
```bash
[root@workstation-d4f5 ~]# openssl x509 -in /etc/docker-distribution/my_self_signed_cert.crt -out /etc/pki/ca-trust/source/anchors/workstation.pem -outform PEM
```
Update the system’s Trust Store using update-ca-trust:
```bash
[root@workstation ~]# update-ca-trust
```
Copy a container image, prior to the latest, from Red Hat to our local registry using skopeo:
```bash
[root@workstation ~]# skopeo copy \
docker://registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest \
docker://workstation.example.com:5000/rhceph/rhceph-3-rhel7:latest
```
Install the ceph-ansible and ceph-common package on the workstation host:
```bash
[root@workstation ~]# yum -y install ceph-ansible ceph-common
```
Create an Ansible hosts inventory file called ~/ceph-nodes.
```bash
[root@workstation-d4f5 ~]# vi ceph-nodes
```
```bash
[ceph]
ceph-node01
ceph-node02
ceph-node03
ceph-mon01
ceph-mon02
ceph-mon03
```
Copy all SSH keys too all nodes for authentication. Using password r3dh4t1!
```bash
[root@workstation-d4f5 ~]# sed 1d ceph-nodes | xargs -i ssh-copy-id root@{}
```
Create an Ansible Playbook called ceph-preqs.yml with the following content.
```bash
[root@workstation-d4f5 ~]# vi ceph-preqs.yml
```
```yaml=
---
- name: Ceph Installation Pre-requisites
hosts: all
gather_facts: no
vars:
remote_user: root
ignore_errors: yes
tasks:
- name: Create the admin user
user:
name: admin
generate_ssh_key: yes
ssh_key_bits: 2048
ssh_key_file: .ssh/id_rsa
password: $6$mZKDrweZ5e04Hcus$97I..Zb0Ywh1lQefdCRxGh2PJ/abNU/LIN7zp8d2E.uYUSmx1RLokyzYS3mUTpipvToZbYKyfMqdP6My7yYJW1
- name: Create sudo file for admin user
lineinfile:
path: /etc/sudoers.d/admin
state: present
create: yes
line: "admin ALL=(root) NOPASSWD:ALL"
owner: root
group: root
mode: 0440
- name: Push ssh key to hosts
authorized_key:
user: admin
key: "{{ lookup('file', '/root/.ssh/id_rsa.pub') }}"
state: present
- name: Copy certificate file from local registry
copy:
src: /etc/pki/ca-trust/source/anchors/workstation.pem
dest: /etc/pki/ca-trust/source/anchors/workstation.pem
- name: Update system's trust store with new certificate
shell:
/usr/bin/update-ca-trust
- name: Get registry catalog to validate certificate trust
get_url:
url: https://workstation.example.com:5000/v2/_catalog
dest: /tmp/registry-catalog.json
```
Run the ceph-preqs.yml playbook:
```bash
[root@workstation ~]# ansible-playbook -i ceph-nodes ceph-preqs.yml
```
Update the root user's SSH config file with new hosts.
```bash
[root@workstation-d4f5 ~]# vi ~/.ssh/config
```
```bash
Host ceph-node01
Hostname ceph-node01
User admin
Host ceph-node02
Hostname ceph-node02
User admin
Host ceph-node03
Hostname ceph-node03
User admin
Host ceph-mon01
Hostname ceph-mon01
User admin
Host ceph-mon02
Hostname ceph-mon02
User admin
Host ceph-mon03
Hostname ceph-mon03
User admin
```
Change the working directory to the ceph-ansible installation directory:
```bash
[root@workstation ~]# cd /usr/share/ceph-ansible/
```
Create the Ansible inventory file:
```bash
[root@workstation ceph-ansible]# vi hosts
```
```bash
[mons]
ceph-node0[1:3]
[mgrs]
ceph-node0[1:3]
[osds]
ceph-node0[1:3]
[rgws]
ceph-mon0[1:3]
```
Create the ansible.cfg file for Ansible:
```bash
[root@workstation ceph-ansible]# mv ansible.cfg ansible.cfg.orig
[root@workstation ceph-ansible]# vi ansible.cfg
```
```bash
[defaults]
inventory = /usr/share/ceph-ansible/hosts
action_plugins = /usr/share/ceph-ansible/plugins/actions
roles_path = /usr/share/ceph-ansible/roles
log_path = /var/log/ansible.log
timeout = 60
host_key_checking = False
retry_files_enabled = False
retry_files_save_path = /usr/share/ceph-ansible/ansible-retry
[privilege_escalation]
become=True
become_method=sudo
become_user=root
become_ask_pass=False
[ssh_connection]
# see: https://github.com/ansible/ansible/issues/11536
control_path = %(directory)s/%%h-%%r-%%p
```
Create the all.yml file:
```bash
[root@workstation ceph-ansible]# vi group_vars/all.yml
```
```yaml=
---
###
# General Options
###
cluster: ceph
fetch_directory: /usr/share/ceph-ansible/ceph-ansible-keys
ntp_service_enabled: true
###
# Ceph Configuration Overrides
###
ceph_conf_overrides:
global:
mon_osd_allow_primary_afinity: true
osd_pool_default_size: 2
osd_pool_default_min_size: 1
mon_pg_warn_min_per_osd: 0
mon_pg_warn_max_per_osd: 0
mon_pg_warn_max_object_skew: 0
client:
rbd_default_features: 1
rbd_default_format: 2
###
# Client Options
###
rbd_cache: "true"
rbd_cache_writethrough_until_flush: "false"
###
# OSD Options
###
journal_size: 1024
public_network: 192.168.99.0/24
cluster_network: 192.168.56.0/24
osd_scenario: non-collocated
###
# Monitor Options
###
monitor_address_block: 192.168.99.0/24
###
# RADOSGW Options
###
radosgw_address_block: 192.168.99.0/24
###
# Docker
###
ceph_docker_image: rhceph/rhceph-3-rhel7
ceph_docker_image_tag: latest
ceph_docker_registry: workstation.example.com:5000
containerized_deployment: true
```
Create the osds.yml file:
```bash
[root@workstation ceph-ansible]# vi group_vars/osds.yml
```
```yaml=
---
copy_admin_key: true
devices:
- /dev/vdb
- /dev/vdc
dedicated_devices:
- /dev/vdd
- /dev/vdd
ceph_osd_docker_memory_limit: 1g
ceph_osd_docker_cpu_limit: 1
```
Create the mons.yml file:
```bash
[root@workstation ceph-ansible]# vi group_vars/mons.yml
```
```yaml=
---
ceph_mon_docker_memory_limit: 1g
ceph_mon_docker_cpu_limit: 1
```
Create the mgrs.yml file:
```bash
[root@workstation ceph-ansible]# vi group_vars/mgrs.yml
```
```yaml=
---
ceph_mgr_docker_memory_limit: 1g
ceph_mgr_docker_cpu_limit: 1
```
Create the rgws.yml file:
```bash
[root@workstation-d4f5 ceph-ansible]# vi group_vars/rgws.yml
```
```yaml=
---
dummy:
ceph_rgw_docker_memory_limit: 1g
ceph_rgw_docker_cpu_limit: 1
```
Create the site-docker.yml playbook:
```bash
[root@workstation ceph-ansible]# cp site-docker.yml.sample site-docker.yml
```
Invoke the site-docker.yml playbook to deploy the Ceph cluster:
```bash
[root@workstation ceph-ansible]# ansible-playbook -i hosts site-docker.yml
```
Copy the Ceph configuration and the administrative keyring to the workstation host:
```bash
[root@workstation ~]# scp root@ceph-node01:/etc/ceph/ceph.conf /etc/ceph/
[root@workstation ~]# scp root@ceph-node01:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
```
Examine the Ceph cluster status and verify access to the cluster:
```bash
[root@workstation ~]# ceph -s
```
:::info
**A Swift Storage cluster running on the differents nodes as Ceph, each also configured with 2 disk devices**
**An additonal storage policy named FOUR should be created, requiring 4 replicas of all objects and using 6 filesystems on 3 nodes.**
:::
***Solution***
Create the ~/rsyncd.conf file.
```bash
[root@workstation-d4f5 ~]# vi ~/rsyncd.conf
```
Enure the global section contains these entries:
```bash
##assumes 'swift' has been used as the Object Storage user/group
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
##address on which the rsync daemon listens
address = <IP>
[account]
max connections = 2
path = /srv/node/
read only = false
write only = no
list = yes
incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
outgoing chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
write only = no
list = yes
incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
outgoing chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
write only = no
list = yes
incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
outgoing chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r
lock file = /var/lock/object.lock
```
Copy the rsyncd.conf to all storage nodes.
```bash
[root@workstation-d4f5 ~]# grep stor swift-nodes | xargs -i scp rsyncd.conf root@{}:/etc
```
Replace the IP in the rsyncd.conf to the according storage node.
```bash
[root@workstation-d4f5 ~]# ssh root@stor01 "sed -i 's/<IP>/172.16.7.21/' /etc/rsyncd.conf && cat /etc/rsyncd.conf"
[root@workstation-d4f5 ~]# ssh root@stor02 "sed -i 's/<IP>/172.16.7.22/' /etc/rsyncd.conf && cat /etc/rsyncd.conf"
[root@workstation-d4f5 ~]# ssh root@stor03 "sed -i 's/<IP>/172.16.7.23/' /etc/rsyncd.conf && cat /etc/rsyncd.conf"
```
Enable and start the rsyncd system service on each Swift storage node:
```bash
[root@workstation-d4f5 ~]# grep stor swift-nodes | xargs -i ssh root@{} "systemctl enable rsyncd"
[root@workstation-d4f5 ~]# grep stor swift-nodes | xargs -i ssh root@{} "systemctl start rsyncd"
```
Install the software packages on the Swift storage nodes:
```bash
[root@workstation-d4f5 ~]# grep stor swift-nodes | xargs -i ssh root@{} "yum -y install openstack-swift-object openstack-swift-container openstack-swift-account memcached"
```
Install the software on the Swift proxy nodes:
```bash
[root@workstation-d4f5 ~]# grep proxy swift-nodes | xargs -i ssh root@{} "yum -y install openstack-swift-proxy memcached python-memcached"
```
Create a random hash value for Swift’s hash_prefix and hash_suffix:
```bash
[root@workstation ~]$ openssl rand -hex 8 | tee hash_prefix
[root@workstation ~]$ openssl rand -hex 8 | tee hash_suffix
```
Create swift.conf.
```bash
[root@workstation-d4f5 ~]# vi swift.conf
```
With the following content. Please replace the prefix and suffix with your own prefix and suffix.
```bash
[swift-hash]
swift_hash_path_prefix = e693e289a2e716a0
swift_hash_path_suffix = 47910b9fd8c8401f
[storage-policy:0]
name = policy-0
policy_type = replication
default = yes
[storage-policy:1]
name = FOUR
policy_type = replication
```
Copy the swift.conf file to /etc/swift/swift.conf at each node in the Swift cluster.
```bash
[root@workstation ~]# cat swift-nodes | xargs -I {} -P 6 scp swift.conf root@{}:/etc/swift/swift.conf
```
Change the bind_ip for container-server.conf, account-server.conf and object-server.conf in all storage ndoes.
```bash
[root@workstation-d4f5 ~]# for n in container-server.conf account-server.conf object-server.conf;do ssh root@stor01 "sed -i 's/127.0.0.1/172.16.7.21/' /etc/swift/${n} && cat /etc/swift/${n}";done
[root@workstation-d4f5 ~]# for n in container-server.conf account-server.conf object-server.conf;do ssh root@stor02 "sed -i 's/127.0.0.1/172.16.7.22/' /etc/swift/${n} && cat /etc/swift/${n}";done
[root@workstation-d4f5 ~]# for n in container-server.conf account-server.conf object-server.conf;do ssh root@stor03 "sed -i 's/127.0.0.1/172.16.7.23/' /etc/swift/${n} && cat /etc/swift/${n}";done
```
Install the python-swift package
```bash
[root@workstation ~]$ yum -y install python-swift
```
Create a directory for creating ring files:
```bash
[root@workstation ~]$ mkdir ~/swift-rings/
[root@workstation ~]$ cd ~/swift-rings/
```
Create all 3 rings (account, container and object)
```bash
[root@workstation-d4f5 swift-rings]# swift-ring-builder ./account.builder create 10 3 0
[root@workstation-d4f5 swift-rings]# swift-ring-builder ./container.builder create 10 3 0
[root@workstation-d4f5 swift-rings]# swift-ring-builder ./object.builder create 10 3 0
[root@workstation-d4f5 swift-rings]# swift-ring-builder ./object-1.builder create 10 4 0
```
Add devices to the account service ring, 2 disks for each storage node:
```bash
swift-ring-builder ./account.builder add z1-172.16.7.21:6202/b1 100
swift-ring-builder ./account.builder add z1-172.16.7.21:6202/c1 100
swift-ring-builder ./account.builder add z1-172.16.7.22:6202/b1 100
swift-ring-builder ./account.builder add z1-172.16.7.22:6202/c1 100
swift-ring-builder ./account.builder add z1-172.16.7.23:6202/b1 100
swift-ring-builder ./account.builder add z1-172.16.7.23:6202/c1 100
```
Add devices to the container service ring, 2 disks for each storage node:
```bash
swift-ring-builder ./container.builder add z1-172.16.7.21:6201/b1 100
swift-ring-builder ./container.builder add z1-172.16.7.21:6201/c1 100
swift-ring-builder ./container.builder add z1-172.16.7.22:6201/b1 100
swift-ring-builder ./container.builder add z1-172.16.7.22:6201/c1 100
swift-ring-builder ./container.builder add z1-172.16.7.23:6201/b1 100
swift-ring-builder ./container.builder add z1-172.16.7.23:6201/c1 100
```
Add devices to the object service ring, 2 disks for each storage node:
```bash
swift-ring-builder ./object.builder add z1-172.16.7.21:6200/b1 100
swift-ring-builder ./object.builder add z1-172.16.7.21:6200/c1 100
swift-ring-builder ./object.builder add z1-172.16.7.22:6200/b1 100
swift-ring-builder ./object.builder add z1-172.16.7.22:6200/c1 100
swift-ring-builder ./object.builder add z1-172.16.7.23:6200/b1 100
swift-ring-builder ./object.builder add z1-172.16.7.23:6200/c1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.21:6200/b1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.21:6200/c1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.22:6200/b1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.22:6200/c1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.23:6200/b1 100
swift-ring-builder ./object-1.builder add z1-172.16.7.23:6200/c1 100
```
Create a binary ring files.
```bash
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
swift-ring-builder object-1.builder rebalance
```
Distribute the ring files to all Swift storage nodes:
```bash
[root@workstation-d4f5 swift-rings]# sed 1d ~/nodes | xargs -i scp -r * root@{}:/etc/swift/
```
Enable the Swift account, container and object server services:
```bash
[root@workstation-d4f5 swift-rings]# sed 1d ~/nodes | xargs -i ssh root@{} systemctl enable openstack-swift-account openstack-swift-container openstack-swift-object
```
Start the Swift account, container and object server services:
```bash
[root@workstation-d4f5 swift-rings]# sed 1d ~/nodes | xargs -i ssh root@{} systemctl start openstack-swift-account openstack-swift-container openstack-swift-object
```
:::info
**Two Swift proxy servers running on nodes proxy01 & proxy02**
:::
***Solution***
Install the Swift client software on the workstation node:
```bash
[root@workstation-d4f5 swift-rings]$ yum -y install python-swiftclient
```
Distribute the ring files to the Swift proxy nodes:
```bash
[root@workstation swift-rings]$ scp -r * root@proxy01:/etc/swift/
[root@workstation swift-rings]$ scp -r * root@proxy02:/etc/swift/
```
We will change the proxy nodes server conf using patch here.
Create the proxy-server.conf.patch file
```bash
[root@workstation-d4f5 ~]# vi proxy-server.conf.patch
```
Put the following contents.
```bash=
--- proxy-server.conf.orig 2019-11-25 21:04:24.068917360 -0500
+++ proxy-server.conf 2019-11-25 21:06:11.521459471 -0500
@@ -8,7 +8,7 @@
# open to access by any client. This is almost always a very bad idea, and
# it's overridden by OSP Director, so it is likely to go away some time
# after Newton.
-pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
+#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
# This sample pipeline uses tempauth and is used for SAIO dev work and
# testing. See below for a pipeline using keystone.
@@ -17,7 +17,7 @@
# The following pipeline shows keystone integration. Comment out the one
# above and uncomment this one. Additional steps for integrating keystone are
# covered further below in the filter sections for authtoken and keystoneauth.
-#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystone copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
+pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystone copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
@@ -39,7 +39,7 @@
[filter:cache]
use = egg:swift#memcache
-memcache_servers = 127.0.0.1:11211
+memcache_servers = 172.16.7.50:11211
[filter:ratelimit]
use = egg:swift#ratelimit
@@ -88,10 +88,14 @@
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
-admin_tenant_name = %SERVICE_TENANT_NAME%
-admin_user = %SERVICE_USER%
-admin_password = %SERVICE_PASSWORD%
-auth_host = 127.0.0.1
-auth_port = 35357
-auth_protocol = http
-signing_dir = /tmp/keystone-signing-swift
+paste.filter_factory = keystonemiddleware.auth_token:filter_factory
+auth_uri = http://172.16.7.50:5000
+auth_url = http://172.16.7.50:35357
+memcached_servers = 172.16.7.50:11211
+auth_type = password
+project_domain_id = default
+user_domain_id = default
+project_name = services
+username = swift
+password = r3dh4t1!
+delay_auth_decision = True
```
Copy the patch to all proxy servers.
```bash
[root@workstation-d4f5 ~]# for n in proxy01 proxy02;do scp proxy-server.conf.patch ${n}:/etc/swift/;done
```
Install patch command in both proxy servers.
```bash
[root@workstation-d4f5 ~]# for n in proxy01 proxy02;do ssh ${n} "yum -y install patch";done
```
Apply the patch to the proxy-server.conf file in both proxy servers.
```bash
[root@workstation-d4f5 ~]# for n in proxy01 proxy02;do ssh ${n} "patch /etc/swift/proxy-server.conf < /etc/swift/proxy-server.conf.patch";done
```
Enable memcached.service and openstack-swift-proxy.service.
```bash
[root@workstation-d4f5 ~]# for n in proxy01 proxy02;do ssh ${n} "systemctl enable memcached openstack-swift-proxy";done
```
Start both services.
```bash
[root@workstation-d4f5 ~]# for n in proxy01 proxy02;do ssh ${n} "systemctl start memcached openstack-swift-proxy";done
```
**Configure HA Proxy**
Connect to haproxy01
```bash
[root@wokstation ~]# scp /etc/hosts haproxy01:/etc/hosts
[root@workstation ~]# ssh root@haproxy01
```
Install haproxy package
```bash
[root@haproxy01 ~]# yum install -y haproxy
```
Back up the orginal haproxy.cfg
```bash
[root@haproxy01 ~]# mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
```
Confgure file /etc/haproxy/haproxy.cfg with the following content.
```bash
[root@haproxy01 ~]# vi /etc/haproxy/haproxy.cfg
```
```bash
global
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen swift-cluster 192.168.56.80:8080
mode http
stats enable
stats auth username:password
balance roundrobin
option httpchk HEAD /healthcheck HTTP/1.0
option forwardfor
option http-server-close
timeout http-keep-alive 500
server proxy01 192.168.56.24:8080 weight 5 check inter 2000
server proxy02 192.168.56.25:8080 weight 5 check inter 2000
```
Enable and start the haproxy service
```bash
[root@haproxy01 ~]# systemctl enable haproxy --now
```
Go back to workstation and auth as swift-user.
```bash
[root@workstation-d4f5 ~]# source keystonerc_user
[root@workstation-d4f5 ~(swift-user)]#
```
Find out the swift auth key for swift-user. It's not using the swift comes with OpenStack. Copy the AUTH key in OS_STORAGE_URL (yours will be different).
```bash
[root@workstation-d4f5 ~(swift-user)]# swift auth
export OS_STORAGE_URL=http://172.16.7.50:8080/v1/AUTH_bc0d92bdcf264f95b2be622e72e9eb8f
export OS_AUTH_TOKEN=69dc6688cddf49c49f682f4caece978a
[root@workstation-d4f5 ~(swift-user)]#
```
Edit keystonerc_user for swift-user.
```bash
[root@workstation-d4f5 ~(swift-user)]# vi keystonerc_user
```
Add the OS_STORAGE_URL with your AUTH token to the end of the file as shown below. Note that the IP is different (swift cluster now).
```bash
export OS_USERNAME=swift-user
export OS_PASSWORD=r3dh4t1!
export OS_AUTH_URL=http://172.16.7.50:5000/v3
export PS1='[\u@\h \W(swift-user)]\$ '
export OS_PROJECT_NAME=swift-project
export OS_REGION_NAME=RegionOne
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_STORAGE_URL=http://192.168.56.80:8080/v1/AUTH_bc0d92bdcf264f95b2be622e72e9eb8f
```
Assign the SwiftOperator role to the created user account in the swift-users project:
```bash
[root@workstation-d4f5 ~(swift-user)]# source keystonerc_admin
[root@workstation-d4f5 ~(keystone_admin)]# openstack role add --user swift-user --project swift-project SwiftOperator
```
To test out if swift is working properly.
```bash
[root@workstation-d4f5 ~(swift-user)]# swift auth
export OS_STORAGE_URL=http://192.168.56.80:8080/v1/AUTH_bc0d92bdcf264f95b2be622e72e9eb8f
export OS_AUTH_TOKEN=1a61712a8bdd482c8527ef5145d9f021
[root@workstation-d4f5 ~(swift-user)]# swift stat
Account: AUTH_bc0d92bdcf264f95b2be622e72e9eb8f
Containers: 0
Objects: 0
Bytes: 0
X-Put-Timestamp: 1574738975.77705
X-Timestamp: 1574738975.77705
X-Trans-Id: tx9bd7be26d03840c5b72c0-005ddc9c1e
Content-Type: text/plain; charset=utf-8
X-Openstack-Request-Id: tx9bd7be26d03840c5b72c0-005ddc9c1e
[root@workstation-d4f5 ~(swift-user)]#
```
:::info
A container named OSPAS should be created that uses storage policy policy-0
:::
***Solution***
Auth as swift user and issue the following command.
```bash
[root@workstation-d4f5 ~(swift-user)]# swift post OSPAS
```
Make sure it's in the policy-0.
```bash
[root@workstation-d4f5 ~(swift-user)]# swift stat OSPAS
Account: AUTH_bc0d92bdcf264f95b2be622e72e9eb8f
Container: OSPAS
Objects: 0
Bytes: 0
Read ACL:
Write ACL:
Sync To:
Sync Key:
Accept-Ranges: bytes
X-Storage-Policy: policy-0
Last-Modified: Tue, 26 Nov 2019 03:32:29 GMT
X-Timestamp: 1574739148.74637
X-Trans-Id: txa2ca3f49cb7a4db8becce-005ddc9d76
Content-Type: application/json; charset=utf-8
X-Openstack-Request-Id: txa2ca3f49cb7a4db8becce-005ddc9d76
[root@workstation-d4f5 ~(swift-user)]#
```
:::info
On the workstation node create a swiftrc file should be created in the root user’s home directory on the workstation node so that the swift-user OpenStack account should be able to write objects into a container called DATA that uses the Swift storage policy FOUR
:::
***Solution***
Create the container DATA with policy FOUR
```bash
[root@workstation-d4f5 ~(swift-user)]# source keystonerc_user
[root@workstation-d4f5 ~(swift-user)]# swift post --header 'X-Storage-Policy:FOUR' DATA
```
Copy the keystonerc_user to swiftrc
```bash
[root@workstation-d4f5 ~(swift-user)]# cp keystonerc_user swiftrc
```
:::info
**Cinder should be configured with 3 backend storage pools, all of the type RBD**
**Volumes >10gb should Only be created in the cinder-large storage pool**
**Volumes <5gb should Only be created in the cinder-small storage pool**
**The third storage pool should be named cinder-med**
**Volumes >5gb & <10gb should be spread evenly across all 3 storage pools**
**Ensure that the grading script can create 5 volumes to evaluate the cinder scheduler configuration**
**Cinder should NOT be dependent on a local volume group on the controller node to start**
**Cinder backups should be configured use a NFS share from the server storage**
:::
***Solution***
Use RBD as backend instead of LVM
From the workstation node, create a all 3 storage pools in the Ceph cluster using 128 PGs. They're called large-vol, med-vol and small-vol.
```bash
ceph osd pool create large-vol 128
ceph osd pool create med-vol 128
ceph osd pool create small-vol 128
```
Create one user to access each pool. They're cender-l, cender-m and cinder-s.
```bash
ceph auth get-or-create client.cinder-l mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=large-vol' -o /etc/ceph/ceph.client.cinder-l.keyring
ceph auth get-or-create client.cinder-m mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=med-vol' -o /etc/ceph/ceph.client.cinder-m.keyring
ceph auth get-or-create client.cinder-s mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=small-vol' -o /etc/ceph/ceph.client.cinder-s.keyring
```
Copy the cinder user’s keyring file to the /etc/ceph directory on ctrl01 to distribute the keyring:
```bash
[root@workstation ~]$ ssh root@ctrl01 yum -y install ceph-common
[root@workstation ~]$ scp /etc/ceph/ceph.client.cinder-*.keyring root@ctrl01:/etc/ceph/
```
Retrieve each cinder user’s key and save it as a file in ctrl01.
```bash
ceph auth get-key client.cinder-l | ssh ctrl01 tee ./client.cinder-l.key
ceph auth get-key client.cinder-m | ssh ctrl01 tee ./client.cinder-m.key
ceph auth get-key client.cinder-s | ssh ctrl01 tee ./client.cinder-s.key
```
Distribute the /etc/ceph/ceph.conf file to the ctrl01 node:
```bash
[root@workstation ~]$ scp /etc/ceph/ceph.conf root@ctrl01:/etc/ceph
```
From the ctrl01 server, view the Ceph status to verify that you can access the cluster as the cinder user:
```bash
[root@workstation ~]# ssh ctrl01
[root@ctrl01 ~]$ ceph --id cinder-l -s
```
On ctrl01, set the ceph.client.cinder-*.keyring file group to cinder to enable Cinder access to the keyring, and update the file permissions to allow group read permissions:
```bash
[root@ctrl01 ~]# chgrp cinder /etc/ceph/ceph.client.cinder-*.keyring
[root@ctrl01 ~]# chmod 0640 /etc/ceph/ceph.client.cinder-*.keyring
```
Generate a UUID and save it in myuuid.txt:
```bash
[root@ctrl01 ~]$ uuidgen | tee ~/myuuid.txt
```
Add an [rbd] section, using the UUID previously generated for the rbd_secret_uuid parameter as the following sample code fragment shows:
```bash
[root@ctrl01 ~]$ vi /etc/cinder/cinder.conf
```
Sample Fragment ==(Please aware that the UUID is the UUID you generated before)==
```bash
[DEFAULT]
...
default_volume_type = cinder-large
enabled_backends = cinder-large,cinder-med,cinder-small
...
[cinder-large]
backend_host = rbd:cinder
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_pool = large-vol
rbd_user = cinder-l
rbd_secret_uuid = b3af1703-1e55-4139-a274-5d370ee35528
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rbd_flatten_volume_from_snapshot = false
filter_function = "5 <= volume.size"
[cinder-med]
backend_host = rbd:cinder
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_pool = med-vol
rbd_user = cinder-m
rbd_secret_uuid = b3af1703-1e55-4139-a274-5d370ee35528
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rbd_flatten_volume_from_snapshot = false
filter_function = "(5 <= volume.size) and (volume.size < 10)"
[cinder-small]
backend_host = rbd:cinder
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_pool = small-vol
rbd_user = cinder-s
rbd_secret_uuid = b3af1703-1e55-4139-a274-5d370ee35528
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rbd_flatten_volume_from_snapshot = false
filter_function = "volume.size < 10"
```
Go back to workstation, auth as keystone admin, then reate a new volume type.
```bash
[root@workstation-d4f5 ~]# source keystonerc_admin
[root@workstation-d4f5 ~(keystone_admin)]# openstack volume type create cinder-large
[root@workstation-d4f5 ~(keystone_admin)]# openstack volume type create cinder-med
[root@workstation-d4f5 ~(keystone_admin)]# openstack volume type create cinder-small
```
Restart both cinder-api and cinder-volume services so that Cinder starts using the RBD back end:
```bash
[root@workstation-d4f5 ~]# ssh ctrl01 systemctl restart openstack-cinder-api openstack-cinder-volume
```
Create a file named ceph.xml for virsh to use to create a libvirt secret: ==(uuid is your uuid)==
```bash
[root@ctrl01 ~]# vi ceph.xml
```
```xml
<secret ephemeral="no" private="no">
<uuid> bdbed7ec-9c95-42d9-8f52-bfc4b5baaa8e </uuid>
<usage type="ceph">
<name>cinder secret</name>
</usage>
</secret>
```
Distribute ceph.xml to each compute node:
```bash
[root@ctrl01 ~]# scp ceph.xml root@compute01:~/
[root@ctrl01 ~]# scp ceph.xml root@compute02:~/
```
Using virsh to define a libvirt secret using the ceph.xml file created:
```bash
[root@ctrl01 ~]# ssh compute01 virsh secret-define --file ~/ceph.xml
[root@ctrl01 ~]# ssh compute02 virsh secret-define --file ~/ceph.xml
```
Copy the cinder client’s key, stored in a file created in a previous step, to the compute nodes:
```bash
[root@ctrl01 ~]# scp client.cinder-*.key root@compute01:~/
[root@ctrl01 ~]# scp client.cinder-*.key root@compute02:~/
```
Using the UUID generated previously, assign the proper value to the libvirt secret:
```bash
ssh compute01 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-l.key)
ssh compute01 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-m.key)
ssh compute01 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-s.key)
ssh compute02 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-l.key)
ssh compute02 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-m.key)
ssh compute02 virsh secret-set-value --secret $(cat myuuid.txt) --base64 $(cat ~/client.cinder-s.key)
```
Configure NFS
Connect to storage node
```bash
[root@ctrl01 ~(keystone_admin)]# ssh storage
```
Install nfs server and configure firewall
```bash
[root@storage ~]# yum install nfs-utils -y
```
Enable service and firewall.
```bash
systemctl enable nfs-server --now
firewall-cmd --add-service=nfs --permanent
firewall-cmd --add-service=rpc-bind --permanent
firewall-cmd --add-service=mountd --permanent
firewall-cmd --reload
```
Configure export
```bash
parted -a optimal /dev/vdb mklabel gpt mkpart primary 0% 30G
mkfs.ext4 /dev/vdb1
echo "/dev/vdb1 /exports ext4 defaults 0 0" >> /etc/fstab
mount -a
echo "/exports 172.16.7.0/24(rw,no_root_squash)" >> /etc/exports
exportfs -a
```
Go back to ctrl01 and install crudini so we can change the config file from command line.
```bash
[root@ctrl01 ~]# yum install -y crudini
```
Configure cinder to use storage as NFS as backup driver.
```bash
echo "storage:/exports" > /etc/cinder/nfs_shares
chown root:cinder /etc/cinder/nfs_shares
chmod 0640 /etc/cinder/nfs_shares
crudini --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_shares
crudini --set /etc/cinder/cinder.conf DEFAULT backup_driver cinder.backup.drivers.nfs
```
Enable SELinux policy
```bash
[root@ctrl01 cinder(keystone_admin)]# setsebool -P virt_use_nfs on
[root@ctrl01 cinder(keystone_admin)]# ssh compute01 setsebool -P virt_use_nfs on
[root@ctrl01 cinder(keystone_admin)]# ssh compute02 setsebool -P virt_use_nfs on
```
Restart services and get list of the pools
```bash
[root@ctrl01 cinder(keystone_admin)]# systemctl restart openstack-cinder-volume.service
```
:::info
**Glance should be configured to use a Ceph RBD storage pool called "images"**
:::
***Solution***
Create a pool using the ceph utility.
```bash
[root@workstation ~]# ceph osd pool create images 128
```
Create a CephX user for glance
```bash
[root@workstation-d4f5 ~(keystone_admin)]# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' -o /etc/ceph/ceph.client.glance.keyring
```
Copy the keyring file to ctrl01
```bash
[root@workstation ~]# scp /etc/ceph/ceph.client.glance.keyring root@ctrl01:/etc/ceph
[root@workstation ~]# ssh root@ctrl01 chown root:glance /etc/ceph/ceph.client.glance.keyring
[root@workstation ~]# ssh root@ctrl01 chmod 640 /etc/ceph/ceph.client.glance.keyring
```
Go to ctrl01 and config Glance.
```bash
[root@workstation-d4f5 ~]# ssh ctrl01
```
Configure glance-api.conf to use RBD
```bash
crudini --set /etc/glance/glance-api.conf DEFAULT default_store rbd
crudini --set /etc/glance/glance-api.conf glance_store default_store rbd
crudini --set /etc/glance/glance-api.conf glance_store stores rbd
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_pool images
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_user glance
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_ceph_conf /etc/ceph/ceph.conf
```
Restart openstack-glance-api service
```bash
[root@ctrl01 ~(keystone_admin)]# systemctl restart openstack-glance-api
```
:::info
**OpenStack should be configured to use the Swift Cluster running on stor01 - stor03, NOT a local Swift AIO on the controller node**
:::
***Solution***
Already configured in keystone_user and swiftrc.
:::info
**SELinux should be enabled on all nodes**
:::
***Solution***
It's done by default. To verify, use the following command in workstation.
```bash
[root@workstation-d4f5 ~]# for HOST in `grep 172 /etc/hosts | awk '{print $3}'`; do echo -e \\n----$HOST; ssh root@$HOST sestatus; done
```
Google Drive
Gist
Clipboard
Sign in via Facebook
Sign in via X(Twitter)
Sign in via GitHub
Sign in via Dropbox
Sign in with Wallet
