OCP 4.7.6 Post-Installation Configuration
===

###### tags: `IBM` `Infra` `OCP`

## Create Other Users with *cluster-admin* (ClusterRole)
:::danger
The secret key containing the users file for the --from-file argument must be named ```htpasswd```.
:::

htpasswd -c -B -b users.htpasswd <USER NAME> <PASSWORD>
oc create secret generic <SECRET NAME> --from-file=htpasswd=./users.htpasswd -n openshift-config

cat << EOF > other-users.yaml
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: my_htpasswd_provider
    mappingMethod: claim
    type: HTPasswd
    htpasswd:
      fileData:
        name: <SECRET NAME>
EOF

oc apply -f other-users.yaml
oc adm policy add-cluster-role-to-user cluster-admin <USER NAME>

刪除 kubeadmin user (可選)

## Enable *image-registry* (ClusterOperator)
預設不啟用 image-registry，必須設定好其背後的儲存空間，才可啟用。

建立 PV

vi image-registry-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: image-registry-pv
spec:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 10Gi
  nfs:
    path: /var/ocp/image-registry
    server: 172.16.30.39
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs01

oc create -f image-registry-pv.yaml

建立 PVC

vi image-registry-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: image-registry-pvc
  namespace: openshift-image-registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: nfs01
  volumeMode: Filesystem
  
oc create -f image-registry-pvc.yaml

:::info
PV 數量這麼多，該如何找到適合的呢？ 這時候當然是要由系統管理人員對 PV 進行分類，也就是說在建立 PV 同時必須賦予有意義的 storgeClassName 做為識別證，另外也需要規範回收策略，例如：

* Delete: 當用戶刪除 PersistentVolumeClaim 時，對應的 PersistentVolume 將被自動被刪除。
* Retain: 當用戶刪除 PersistentVolumeClaim 時，對應的 PersistentVolume 不會被刪除，它將變為 Released 狀態，表示所有的資料可以被手動恢復。
:::

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"pvc":{"claim":"image-registry-pvc"}}}}'

將 ManagementState 從 Removed 修改成 Managed

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
oc get configs.imageregistry.operator.openshift.io/cluster -o yaml | grep managementState

驗證 clusteroperator

oc get clusteroperator image-registry

確認出現 5000 port 的 service

oc get svc -n openshift-image-registry
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
image-registry            ClusterIP   10.246.187.88   <none>        5000/TCP    115s
image-registry-operator   ClusterIP   None            <none>        60000/TCP   3d12h

測試上傳 imagestream

oc debug nodes/<NODE ADDRESS>
sh-4.2# chroot /host
sh-4.2# oc login -u <USER> -p <PASSWORD> https://api.myocp.abbyland.org:6443
sh-4.2# oc policy add-role-to-user registry-viewer $(oc whoami)
sh-4.2# oc policy add-role-to-user registry-editor $(oc whoami)
sh-4.2# oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
sh-4.2# IMAGE_REGISTRY_HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
default-route-openshift-image-registry.dte-ocp46-qj1pxa-915b3b336cabec458a7c7ec2aa7c625f-0000.us-east.containers.appdomain.cloud
sh-4.2# podman login -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false $IMAGE_REGISTRY_HOST
sh-4.2# podman login -u [e-mail] -p <REDHAT_PASSWORD> --tls-verify=false registry.redhat.io
sh-4.2# podman pull registry.redhat.io/rhel8/support-tools
sh-4.2# podman tag registry.redhat.io/rhel8/support-tools $IMAGE_REGISTRY_HOST/rhel8/support-tools
sh-4.2# podman push $IMAGE_REGISTRY_HOST/rhel8/support-tools

## Configure Offline OperatorHub
### Disable the default remote OperatorHub sources for OLM

oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'   
oc get OperatorHub cluster -o yaml

### Create local mirrors of the OperatorHub content
| Catalog | Description |
| -------- | -------- |
| redhat-operators | Public catalog for Red Hat products packaged and shipped by Red Hat. Supported by Red Hat. |
| certified-operators | Public catalog for products from leading independent software vendors (ISVs). Red Hat partners with ISVs to package and ship. Supported by the ISV. |
| community-operators | Public catalog for software maintained by relevant representatives in the operator-framework/community-operators GitHub repository. No official support. |

1. 建立一 Operator catalog image
:::danger
The internal registry of the OpenShift Container Platform cluster cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process.

The Operator must:
* List any related images, or other container images that the Operator might require to perform their functions, in the relatedImages parameter of its ClusterServiceVersion (CSV) object.
* Reference all specified images by a digest (SHA) and not by a tag
:::

podman login registry.redhat.io
podman login reg.myocp.abbyland.org:5000

REG_CREDS=$HOME/pull-secret.json

oc adm catalog build \
    --appregistry-org redhat-operators \
    --from=registry.redhat.io/openshift4/ose-operator-registry:v4.7 \
    --filter-by-os="linux/ppc64le" \
    --to=reg.myocp.abbyland.org:5000/olm/redhat-operators:v4.7 \
    -a ${REG_CREDS} \
    --insecure

驗證

curl -u reguser:reguser1234 -k https://reg.myocp.abbyland.org:5000/v2/_catalog
{"repositories":["ocp4/openshift4", "olm/redhat-operators"]}

podman pull https://reg.myocp.abbyland.org:5000/olm/redhat-operators:v4.7
podman run -p 50051:50051 -it https://reg.myocp.abbyland.org:5000/olm/redhat-operators:v4.7

# grpcurl -plaintext localhost:50051 api.Registry/ListPackages
{
  "name": "3scale-operator"
}
{
  "name": "amq-broker"
}
{
  "name": "amq-online"
}

2. 解壓縮 Operator catalog image 以產生所有 operators 參考到的 images 清單，將清單所有內容推入 mirror registry

oc adm catalog mirror \
    reg.myocp.abbyland.org:5000/olm/redhat-operators:v4.7 \
    reg.myocp.abbyland.org:5000 \
    -a ${REG_CREDS} \
    --insecure \
    --filter-by-os='.*'

:::danger
If the --filter-by-os flag remains unset or set to any value other than .*, the command filters out different architectures, which changes the digest of the manifest list, also known as a multi-arch image. The incorrect digest causes deployments of those images and Operators on disconnected clusters to fail. For more information, see BZ#1890951.
:::
產生./redhat-operators-manifests目錄，底下有:
* imageContentSourcePolicy.yaml - 用來設定 nodes，使其關聯 Operator 清單參考到的 images 與對應的 mirror registry repositories
* mapping.txt (欲修改清單才會用到) - 給 ```oc image mirror``` 使用的清單，包含所有需要的 images，格式如

3. 建立 ImageContentSourcePolicy 物件

oc apply -f ./redhat-operators-manifests/imageContentSourcePolicy.yaml

4. 建立 CatalogSource 物件

vi catalogsource.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: my-operator-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: reg.myocp.abbyland.org:5000/olm/redhat-operators:v4.7
displayName: My Operator Catalog
publisher: Red Hat

oc create -f catalogsource.yaml

驗證

oc get pods -n openshift-marketplace

oc get catalogsource -n openshift-marketplace

oc get packagemanifest -n openshift-marketplace
NAME CATALOG AGE
etcd My Operator Catalog 34s

## Reference
- [乾貨！OpenShift離線環境OperatorHub和ImageStream配置實戰技巧](https://www.mdeditor.tw/pl/ggZY/zh-tw)
- [Using Operator Lifecycle Manager on restricted networks](https://docs.openshift.com/container-platform/4.5/operators/admin/olm-restricted-networks.html)
- [OpenShift 4 - 配置OpenShift集群日志环境EFK](https://blog.csdn.net/weixin_43902588/article/details/105586460)
- [Exposing image-registry](https://docs.openshift.com/container-platform/4.1/registry/securing-exposing-registry.html)