---
# System prepended metadata

title: steps Performed

---

Description of problem (please be detailed as possible and provide log
snippests):
------------------------------------------------------------------------
Platform: AWS I3 LSO
Mode - Arbiter 

Crashcollector pods stuck in Init state, secret not found

steps Performed
=================
1. Install OCP 4.7 and LSO operator , with at least 3 zones, 2 zones with 2 Worker nodes, 3rd zone with atleast 1 W (UI doesn't support bringing up arbiter MON on Mater node yet)
2. Label the nodes with  "". 

In AWS, the nodes are already labelled with 
topology.kubernetes.io/zone=us-east-2a and failure-domain.beta.kubernetes.io/zone=us-east-2a, see additional info for more details.

Note: Since the current OCS build does not have the new features, edited the CSV to add the following:
oc edit csv ocs-operator.v4.7.0-222.ci

Edit the enabled features to the following:
features.ocs.openshift.io/enabled: '["kms", "arbiter", "flexible-scaling"]'


Install OCS operator 4.7.0-222.ci and click on create storage cluster
3. Select Internal -Attached mode 

Sub-Steps
3a Discover Disks: -> Select Nodes: --> Select 2W nodes, each in say zone-A and zone-B (to bring up OSDs)
3b. Create Storage Class -> Provide name for SC and PVs will be created on the LSO disks
3c. Storage and the nodes -> Click on the checkbox to Enable Arbiter, select the arbiter zone (here zone: us-east-2c) and select the storageclass created in above step. 
3d. Configure -> No change
3e. Review and create: review the selections and click create

Actual results
===================
The mon and mgr pods are up but the crashcollector pods are stuck in Init state with following error message:

 Warning  FailedMount  22s (x11 over 6m34s)  kubelet            MountVolume.SetUp failed for volume "rook-ceph-crash-collector-keyring" : secret "rook-ceph-crash-collector-keyring" not found

Expected Results:
================
The POD should be up


additional info
===============

$ oc get pods -o wide -n openshift-storage
NAME                                                        READY   STATUS     RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
csi-cephfsplugin-59r4s                                      3/3     Running    0          78m     10.0.193.245   ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-pm94f                                      3/3     Running    0          78m     10.0.137.196   ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5c5b96fb84-98299               6/6     Running    0          78m     10.130.2.32    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
csi-cephfsplugin-provisioner-5c5b96fb84-qg6xh               6/6     Running    0          78m     10.131.2.32    ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
csi-cephfsplugin-qcprd                                      3/3     Running    0          78m     10.0.176.85    ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
csi-cephfsplugin-vnz4l                                      3/3     Running    0          78m     10.0.163.48    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
csi-cephfsplugin-xqltm                                      3/3     Running    0          78m     10.0.133.0     ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
csi-rbdplugin-22ksn                                         3/3     Running    0          78m     10.0.137.196   ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-bxmrt                                         3/3     Running    0          78m     10.0.133.0     ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
csi-rbdplugin-lgh4w                                         3/3     Running    0          78m     10.0.163.48    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-lgxvw                                         3/3     Running    0          78m     10.0.176.85    ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-lznw2                                         3/3     Running    0          78m     10.0.193.245   ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-55c8b8c747-4rfvd                  6/6     Running    0          78m     10.129.2.17    ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-provisioner-55c8b8c747-bbll7                  6/6     Running    0          78m     10.131.0.64    ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
ip-10-0-133-0us-east-2computeinternal-debug                 1/1     Running    0          3m28s   10.0.133.0     ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
ip-10-0-137-196us-east-2computeinternal-debug               1/1     Running    0          3m28s   10.0.137.196   ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
ip-10-0-163-48us-east-2computeinternal-debug                1/1     Running    0          3m28s   10.0.163.48    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
ip-10-0-176-85us-east-2computeinternal-debug                1/1     Running    0          3m28s   10.0.176.85    ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
must-gather-xq6p7-helper                                    1/1     Running    0          24m     10.128.2.72    ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
noobaa-operator-7457969fbc-tzrnm                            1/1     Running    0          93m     10.131.0.61    ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
ocs-metrics-exporter-686f84867c-tlgmp                       1/1     Running    0          93m     10.131.2.30    ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
ocs-operator-657579557d-hdjt7                               1/1     Running    0          93m     10.131.0.62    ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
rook-ceph-crashcollector-ip-10-0-133-0-6484866c48-d4d44     0/1     Init:0/2   0          77m     <none>         ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
rook-ceph-crashcollector-ip-10-0-137-196-577fc4fd6d-grmpq   0/1     Init:0/2   0          77m     <none>         ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
rook-ceph-crashcollector-ip-10-0-163-48-66d678fd4f-hhgz9    0/1     Init:0/2   0          77m     <none>         ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
rook-ceph-crashcollector-ip-10-0-176-85-55588bbb98-kf4zn    0/1     Init:0/2   0          77m     <none>         ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
rook-ceph-crashcollector-ip-10-0-193-245-5fd7fd8848-sq7zk   0/1     Init:0/2   0          76m     <none>         ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
rook-ceph-mon-a-7c9c8859b6-lz8ms                            1/1     Running    0          78m     10.130.2.34    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
rook-ceph-mon-b-c694fccbf-z8j4s                             1/1     Running    0          77m     10.129.2.19    ip-10-0-176-85.us-east-2.compute.internal    <none>           <none>
rook-ceph-mon-c-64689fcb65-kjj2r                            1/1     Running    0          77m     10.131.2.34    ip-10-0-133-0.us-east-2.compute.internal     <none>           <none>
rook-ceph-mon-d-d9cf78c59-rtz54                             1/1     Running    0          77m     10.128.2.32    ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>
rook-ceph-mon-e-8758f55c4-54tb6                             1/1     Running    0          76m     10.131.0.66    ip-10-0-193-245.us-east-2.compute.internal   <none>           <none>
rook-ceph-operator-86b88bfbf7-f8k6z                         1/1     Running    0          93m     10.130.2.29    ip-10-0-163-48.us-east-2.compute.internal    <none>           <none>
rook-ceph-tools-8575486ffd-w275h                            1/1     Running    0          63m     10.0.137.196   ip-10-0-137-196.us-east-2.compute.internal   <none>           <none>


    nodeTopologies:
      labels:
        failure-domain.beta.kubernetes.io/region:
        - us-east-2
        failure-domain.beta.kubernetes.io/zone:
        - us-east-2b
        - us-east-2a
        kubernetes.io/hostname:
        - ip-10-0-163-48
        - ip-10-0-133-0
        - ip-10-0-137-196
        - ip-10-0-176-85