Setup Ceph storage via cephadm

# Setup Ceph storage via cephadm ###### tags: `ceph` Example of Ceph storage cluster setup via cephadm. I setup Ceph storage for OpenStack to use as "shared" storage which allow "live-migration" of instance. OpenStack setup: https://hackmd.io/v2HFpS2AS8CVXiv_I2svZg (kolla-ansible) #### Environment ``` IP Address hostname Ceph Node ----------------------------------------- 192.168.1.23 nuc111 deploy & mon 192.168.1.24 nuc71 osd 192.168.1.25 nuc72 osd 192.168.1.26 nuc73 osd ``` The nuc111 will be deploy node, and initial ceph monitor host. (cephadm will auto add 2 addtional monitors automatically base on cluster size) The nuc71, nuc72 and nuc73 are OSD hosts, they have additional disk /dev/sdb (usb 3.0 drive) for OSD drive. #### Steps Create **cephadmin** user and set to sudoer ``` root@nuc111:~# useradd -m -s /bin/bash cephadmin root@nuc111:~# passwd cephadmin New password: Retype new password: passwd: password updated successfully root@nuc111:~# echo "cephadmin ALL=(ALL:ALL) NOPASSWD:ALL" >> /etc/sudoers.d/cephadmin root@nuc111:~# ls /etc/sudoers.d/ cephadmin README root@nuc111:~# chmod 0440 /etc/sudoers.d/cephadmin ``` Install **cephadm** tool ``` root@nuc111:~# apt install cephadm Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: libfwupdplugin1 libisns0 Use 'apt autoremove' to remove them. Recommended packages: docker.io The following NEW packages will be installed: cephadm 0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded. Need to get 58.5 kB of archives. After this operation, 307 kB of additional disk space will be used. Get:1 http://au.archive.ubuntu.com/ubuntu focal-updates/universe amd64 cephadm amd64 15.2.16-0ubuntu0.20.04.1 [58.5 kB] Fetched 58.5 kB in 0s (665 kB/s) Selecting previously unselected package cephadm. (Reading database ... 130835 files and directories currently installed.) Preparing to unpack .../cephadm_15.2.16-0ubuntu0.20.04.1_amd64.deb ... Unpacking cephadm (15.2.16-0ubuntu0.20.04.1) ... Setting up cephadm (15.2.16-0ubuntu0.20.04.1) ... Adding system user cephadm....done ``` **__NOTE__**: or install via curl to fetch most recent version. For example ``` root@node1:~# curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm root@node1:~# chmod +x ./cephadm ``` Switch to **cephadmin** user ``` root@nuc111:~# su cephadmin - cephadmin@nuc111:/root$ ``` Bootstrap a new cluster ``` cephadmin@nuc111:~$ sudo cephadm bootstrap --mon-ip 192.168.1.23 Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chrony.service is enabled and running Repeating the final host check... podman|docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Host looks OK Cluster fsid: 16e21856-6030-11ed-8a24-db02e0e54b22 Verifying IP 192.168.1.23 port 3300 ... Verifying IP 192.168.1.23 port 6789 ... Mon IP 192.168.1.23 is in CIDR network 192.168.1.0/24 Pulling container image quay.io/ceph/ceph:v15... Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network... Creating mgr... Verifying port 9283 ... Wrote keyring to /etc/ceph/ceph.client.admin.keyring Wrote config to /etc/ceph/ceph.conf Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/10)... mgr not available, waiting (2/10)... mgr not available, waiting (3/10)... mgr not available, waiting (4/10)... mgr not available, waiting (5/10)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for Mgr epoch 5... Mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to to /etc/ceph/ceph.pub Adding key to root@localhost's authorized_keys... Adding host nuc111... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Enabling mgr prometheus module... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling the dashboard module... Waiting for the mgr to restart... Waiting for Mgr epoch 13... Mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://NUC11ATKC41:8443/ User: admin Password: plur7o63uu You can access the Ceph CLI with: sudo /usr/sbin/cephadm shell --fsid 16e21856-6030-11ed-8a24-db02e0e54b22 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ Bootstrap complete. ``` **__NOTE__**: add `--skip-monitoring-stack` option to skip monitoring stack (Prometheus, Grafana, etc...) deploy. **__NOTE__**: add `--cluster-network` to use cluster network. (for example, 192.168.122.0/24) (for newer cephadm version??) https://github.com/ceph/ceph/pull/38911 When login dashbaoard first time, you will be asked to update password. Here I simply update the password. ``` username/password: admin/cephadmin ``` Check Ceph containers ``` cephadmin@nuc111:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 15225fbe7d8a quay.io/prometheus/alertmanager:v0.20.0 "/bin/alertmanager -…" 3 minutes ago Up 3 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-alertmanager.nuc111 23d37d214780 quay.io/ceph/ceph-grafana:6.7.4 "/bin/sh -c 'grafana…" 3 minutes ago Up 3 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-grafana.nuc111 a506c9016de5 quay.io/prometheus/prometheus:v2.18.1 "/bin/prometheus --c…" 4 minutes ago Up 4 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-prometheus.nuc111 795ec188ff2a quay.io/prometheus/node-exporter:v0.18.1 "/bin/node_exporter …" 4 minutes ago Up 4 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-node-exporter.nuc111 a497030e7c57 quay.io/ceph/ceph:v15 "/usr/bin/ceph-crash…" 5 minutes ago Up 5 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-crash.nuc111 19e955a4dadc quay.io/ceph/ceph:v15 "/usr/bin/ceph-mgr -…" 6 minutes ago Up 6 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-mgr.nuc111.oezajx 6120dd3bf195 quay.io/ceph/ceph:v15 "/usr/bin/ceph-mon -…" 6 minutes ago Up 6 minutes ceph-16e21856-6030-11ed-8a24-db02e0e54b22-mon.nuc111 ``` Check Ceph status ``` cephadmin@nuc111:~$ sudo /usr/sbin/cephadm shell --fsid 16e21856-6030-11ed-8a24-db02e0e54b22 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Using recent ceph image quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca root@nuc111:/# sudo ceph -s cluster: id: 16e21856-6030-11ed-8a24-db02e0e54b22 health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum nuc111 (age 8m) mgr: nuc111.oezajx(active, since 7m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: ``` Install Ceph-common package so we can use ceph command to check cluster directly ``` cephadmin@nuc111:~$ sudo cephadm version Using recent ceph image quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) cephadmin@nuc111:~$ sudo cephadm add-repo --release octopus Installing repo GPG key from https://download.ceph.com/keys/release.asc... Installing repo file at /etc/apt/sources.list.d/ceph.list... cephadmin@nuc111:~$ sudo cephadm install ceph-common Installing packages ['ceph-common']... ``` ``` cephadmin@nuc111:~$ sudo ceph -s cluster: id: 16e21856-6030-11ed-8a24-db02e0e54b22 health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum nuc111 (age 12m) mgr: nuc111.oezajx(active, since 12m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: ``` Add host label to nuc111 ``` cephadmin@nuc111:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS nuc111 nuc111 cephadmin@nuc111:~$ sudo ceph orch host label add nuc111 _admin Added label _admin to host nuc111 cephadmin@nuc111:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS nuc111 nuc111 _admin ``` Copy public key of Ceph to OSD hosts ``` cephadmin@nuc111:~$ ssh-copy-id -f -i /etc/ceph/ceph.pub root@nuc71 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub" The authenticity of host 'nuc71 (192.168.1.24)' can't be established. ECDSA key fingerprint is SHA256:jwfxrwrwnLwLblfkfLJC3Yr894+FGQznY8+ylYTNXR8. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes root@nuc71's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@nuc71'" and check to make sure that only the key(s) you wanted were added. cephadmin@nuc111:~$ ssh-copy-id -f -i /etc/ceph/ceph.pub root@nuc72 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub" The authenticity of host 'nuc72 (192.168.1.25)' can't be established. ECDSA key fingerprint is SHA256:9gXSkvu9mZsfNu56cQgMcSYnfEr33h5haoYs8CVxh6E. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes root@nuc72's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@nuc72'" and check to make sure that only the key(s) you wanted were added. cephadmin@nuc111:~$ ssh-copy-id -f -i /etc/ceph/ceph.pub root@nuc73 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub" The authenticity of host 'nuc73 (192.168.1.26)' can't be established. ECDSA key fingerprint is SHA256:4GMMsd+hO9Zh71ZrI/cHneRFoPO3kBuV/f5K7JeMVhY. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes root@nuc73's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@nuc73'" and check to make sure that only the key(s) you wanted were added. ``` Add OSD host to cluster and set label ``` cephadmin@nuc111:~$ sudo ceph orch host add nuc71 Added host 'nuc71' cephadmin@nuc111:~$ sudo ceph orch host add nuc72 Added host 'nuc72' cephadmin@nuc111:~$ sudo ceph orch host add nuc73 Added host 'nuc73' cephadmin@nuc111:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS nuc111 nuc111 _admin mon nuc71 nuc71 nuc72 nuc72 nuc73 nuc73 cephadmin@nuc111:~$ sudo ceph orch host label add nuc71 osd Added label osd to host nuc71 cephadmin@nuc111:~$ sudo ceph orch host label add nuc72 osd Added label osd to host nuc72 cephadmin@nuc111:~$ sudo ceph orch host label add nuc73 osd Added label osd to host nuc73 cephadmin@nuc111:~$ sudo ceph orch host ls HOST ADDR LABELS STATUS nuc111 nuc111 _admin mon nuc71 nuc71 osd nuc72 nuc72 osd nuc73 nuc73 osd ``` **__NOTE__**: you could tell cephadm to deploy daemons base on label. For example, to limit monitors run on host with "mon" label. ``` $ sudo ceph orch apply mon --placement="label:mon" ``` Add storage to cluster ``` cephadmin@nuc111:~$ sudo ceph orch daemon add osd nuc71:/dev/sdb Created osd(s) 0 on host 'nuc71' cephadmin@nuc111:~$ sudo ceph orch daemon add osd nuc72:/dev/sdb Created osd(s) 1 on host 'nuc72' cephadmin@nuc111:~$ sudo ceph orch daemon add osd nuc73:/dev/sdb Created osd(s) 2 on host 'nuc73' ``` Check cluster status ``` cephadmin@nuc111:~$ sudo ceph -s cluster: id: 16e21856-6030-11ed-8a24-db02e0e54b22 health: HEALTH_OK services: mon: 3 daemons, quorum nuc111,nuc71,nuc72 (age 119s) mgr: nuc111.oezajx(active, since 36m), standbys: nuc71.pmlfuu osd: 3 osds: 3 up (since 24s), 3 in (since 24s) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 3.0 GiB used, 692 GiB / 695 GiB avail pgs: 1 active+clean ``` Dashboard ![](https://i.imgur.com/WQQ1Gfi.png) #### Purging cluster ``` root@nuc111:~# ceph mgr module disable cephadm root@nuc111:~# ceph fsid 16e21856-6030-11ed-8a24-db02e0e54b22 cephadm rm-cluster --force --zap-osds --fsid 16e21856-6030-11ed-8a24-db02e0e54b22 ``` Ref.: https://docs.ceph.com/en/quincy/cephadm/operations/#purging-a-cluster https://www.flamingbytes.com/posts/uninstall-ceph/ #### Troubleshoot The cephadm deploy **Prometheus, Grafana** and related tools by default. It has the "node-exporter" container on each node. However, the container uses a lot of CPU resources in my environment (OSD node, 2 cores Celeron CPU) and affect command to ssh and df, lsblk, etc... commands are very slow (took couple minutes to connect or response). - https://www.mail-archive.com/prometheus-users@googlegroups.com/msg09479.html - https://github.com/prometheus/node_exporter#disabled-by-default To disable/remove the Prometheus rleated services (TBC) ``` $ ceph mgr module disable prometheus $ ceph orch rm node-exporter ``` With "1 stray daemon not managed by cephadm" warning message, restart ceph manager container. - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/N3MVYV7LBEXCBIANAFZAPQXU45OTIY3P/ - https://tracker.ceph.com/issues/50272 #### Reference - https://kifarunix.com/install-and-setup-ceph-storage-cluster-on-ubuntu/ - https://docs.ceph.com/en/latest/cephadm/install/ - https://docs.ceph.com/en/octopus/cephadm/monitoring/ - https://www.flamingbytes.com/posts/uninstall-ceph/ - https://docs.ceph.com/en/quincy/mgr/prometheus/#enabling-prometheus-output