HackMD - Collaborative Markdown Knowledge Base

## DF Recovery Proceedure Outline The following procedure is based on Director's ability to deploy clouds using predictable IP addresses and an in-depth knowledge of the Heat database and Engine operational characteristics. **Before beginning this procedure, please read everything within this doc.** #### Administrative Tasks The individual performing the administrative tasks will need to pull information on to the Director node to create a complete set of templates to recover this environment. This procedure assumes all assorted backups have been created, and the deployment is ready for recovery. ###### Setting up Predictable IP Addresses The use of predictable IP addresses follows a specific file format; [see the official documentation for OSP 13 and predictable IP addresses](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-controlling_node_placement#sect-Predictable_IPs) for more information on the requirements. > Please note that the order of the IP addresses corresponds with the numbering order of the host indices; i.e., controller-0 will be the first IP, followed by the second. ###### Example `ips_from_pool.yaml` file (not to be used) * Source file: `/usr/share/openstack-tripleo-heat-templates/environments/ips-from-pool-all.yaml` The following example file is stored within the `stack` user's home folder. ``` yaml Resource_registry: OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml # Management network is optional and disabled by default #OS::TripleO::Controller::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml OS::TripleO::Compute::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Compute::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::Compute::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::Compute::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Compute::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml #OS::TripleO::Compute::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml OS::TripleO::CephStorage::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::CephStorage::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::CephStorage::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::CephStorage::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml OS::TripleO::CephStorage::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml #OS::TripleO::CephStorage::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml OS::TripleO::ObjectStorage::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::ObjectStorage::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::ObjectStorage::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::ObjectStorage::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml OS::TripleO::ObjectStorage::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml #OS::TripleO::ObjectStorage::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml OS::TripleO::BlockStorage::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::BlockStorage::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::BlockStorage::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::BlockStorage::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml OS::TripleO::BlockStorage::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml #OS::TripleO::BlockStorage::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml parameter_defaults: ControllerIPs: # Each controller will get an IP from the lists below, first controller, first IP ctlplane: - 192.168.24.23 external: - 10.46.44.25 internal_api: - 172.17.1.116 storage: - 172.17.3.133 storage_mgmt: - 172.17.4.128 tenant: - 172.17.2.200 ComputeIPs: # Each compute will get an IP from the lists below, first compute, first IP ctlplane: - 192.168.24.24 internal_api: - 172.17.1.17 storage: - 172.17.3.101 tenant: - 172.17.2.171 ``` Once the predictable IP address file has been constructed, add this file to the deployment process as the very last environment file included in the stack. A typical inclusion would look something like so, `-e /home/stack/ips_from_pool.yaml`. > The IP address information can be obtained via the os-net-config JSON file. This JSON file is located under `/etc/os-net-config/config.json` on each Overcloud node. > The usage of each IP range can be found in the `network_data.yaml` file on the Undercloud that is used for the Overcloud deployment. **Remove given Predictable IPs from the provisioning pools. This ensures that reserved IPs for the overcloud nodes are not distributed by accident in the regular DHCP pools.** * Modify `network_data.yaml` (or equivalent) to reduce DHCP pools in order to not conflict with the predictable IPs > If applicable: for more information on modifying the network DHCP pools, please refer to the [official OSP 13 documentation for modifying basic network isolation](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/basic-network-isolation#modifying-isolated-network-configuration). ###### Disable Cleaning To be sure there are no adverse effects, modify the PXE driver to *fake_pxe*, and ensure that node cleaning has been **disabled**. ###### Ceph FSID Setting We believe we should hard code the ceph `FSID` within the customer's templates to ensure further nothing is lost or modified when it comes to the deployment configuration. ###### Database Backups Before running anything potentially destructive, we recommend performing a full backup of the databases. While the database on the Undercloud is in a precarious state, we recommend making a backup of the existing condition to ensure we're able to restore the database in the event of any adverse effects. ``` shell # Escalate your privledges to root sudo su - # Create a plain text dump of all databases on the undercloud mysqldump --all-databases > alldbs.sql ``` ###### Workload Backups It is recommended that the cloud owner perform workload backups before any recovery operation is processed. While this step is not 100% required, it provides insurance that recovery is possible should anything go wrong; this is especially important when dealing with workloads built by hand without accompanying automation. > The backup is an extra precaution we hope not to have to use. The procedure includes stopping ceph temporarily to backup a Ceph Mon; this is not a data backup. * Backup overcloud - [Official OSP13 documentation](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/undercloud_and_control_plane_back_up_and_restore/index). Collect all relavant information about the cloud workload using all available tenant **\$RC** files. ``` shell # Source tenant file source $RC # Pull container list openstack container list # Pull server list openstack server list ``` > openstack container list is run here to check if they have any swift containers which are backed by RGW in their case. Any important containers can be downloaded with the `swift` CLI tools. Steps for `cinder` or `glance` have not been included; however, a similar discovery, download/export process could be executed as needed. Once instances attached to the ceph datastore have been identified, it should be possible to snapshot running VMs to images. #### Modifying the *heat* Database Once the database is backed-up, begin with the restoration process. The first thing we need to do is update the Heat database to set the failed resources to *update*. To access the database, run the following commands. ``` shell # Escalate your privledges to root sudo su - # Access the mysql REPL mysql ``` Once on the SQL REPL, run the following update to set the failed and deleted resources to *update*. ``` sql USE heat UPDATE resource SET action='UPDATE' WHERE action='DELETE' AND status='FAILED'; UPDATE stack SET action='UPDATE' WHERE action='DELETE' AND status='FAILED'; FLUSH PRIVILEGES; EXIT ``` #### Redeployment With everything in place and all backups secured, we're now able to run the update deployment process. > The command used for the deployment is unique for every environment and should be well documented for the customer. Upon completion of the deployment process Director will have regained control of the stack.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.