# pastebin

# repo
[ZTP-LAB](https://dev.azure.com/kskels/_git/ztp-lab)

# Issues

* Generating imageContent mappings is hacky
* Assisted service image mapping (using path from ImageSet)
* Metal3 alternate interface IP
* Metal3 Provisioning CIDR not working - had to bind it to one of the cluster node
* Azure DevOps not working, PAT based / username-password logins failing from argocd
* Dual Stack - Multi Node ZTP not working
* 

* 

## Links

[SNO install](https://docs.openshift.com/container-platform/4.10/installing/installing_sno/install-sno-installing-sno.html)
[rhcos-live.x86_64.iso ISO](https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.10/410.84.202205191234-0/x86_64/rhcos-410.84.202205191234-0-live.x86_64.iso)

apiVersion: v1
baseDomain: atx.kskels.com
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 1
metadata:
  name: ztp-hub-01
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: "2600:1700:4c0:800f::/64"
  clusterNetwork:
  - cidr: "fd01::/48"
    hostPrefix: 64
  serviceNetwork:
  - "fd02::/112"
platform:
  none: {}
bootstrapInPlace:
  installationDisk: /dev/vda
pullSecret: '{"auths":{"nexus.kskels.com:445":{"auth":"xyz"}}}'
sshKey: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGxLsmU7suROe520RAqaqE+Xtmx0iIGpG0IFLKcduA3t [e-mail]"

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  ...base-64-encoded, DER Certificate Authority cert...
  -----END CERTIFICATE-----

  -----BEGIN CERTIFICATE-----
  ...base-64-encoded, DER Certificate Authority cert...
  -----END CERTIFICATE-----

imageContentSources:
- mirrors:
  - nexus.kskels.com:445/openshift-release-dev/ocp-release
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - nexus.kskels.com:445/openshift-release-dev/ocp-v4.0-art-dev
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

alias coreos-installer='podman run --privileged --pull always --rm \
        -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data \
        -w /data quay.io/coreos/coreos-installer:release'

cp ocp/bootstrap-in-place-for-live-iso.ign iso.ign

coreos-installer iso ignition embed -fi iso.ign rhcos-live.x86_64.iso

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:bond0.100:none
bond=bond0:em1,em2:mode=active-backup
vlan=bond0.100:bond0

### IPv6

ip=[2600:1700:4c0:800f::3001]::[2600:1700:4c0:800f:208:a2ff:fe12:4257]:64:ztp-hub-01:enp9s0:off nameserver=[2600:1700:4c0:800f:208:a2ff:fe12:4257]

# Setup Operators

### Generate custom image sources

Execute this command on a node with `oc` and access to the `registry.redhat.io`. This command will only create manifests, see option `--manifests-only`.

oc adm catalog mirror registry.redhat.io/redhat/redhat-operator-index:v4.10 mycustomrepo.kskels.com/my/custom/path --manifests-only -a /home/kskels/nexus/pull-secret.json

This will generate `catalogSource.yaml`, `imageContentSourcePolicy.yaml`, `mapping.txt`.

Edit `catalogSource.yaml` name and update `redhat-operator-index` to `redhat-operators`. This is important to use default/standard name.

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operators
  namespace: openshift-marketplace
spec:
  image: mycustomrepo.kskels.com/my/custom-path-redhat-redhat-operator-index:v4.10
  sourceType: grpc

### Check Artifactory

`podman pull mycustomrepo.kskels.com/my/custom/path/odf4/odf-operator-bundle@sha256:192ef97f674aaaefe4519a09b0890216e956203750d4190a6d366e8aa5f1104f`

You can check more test images in the `mapping.txt`. There will be all images, just pick a few random to check that pulls are working.

### Remove default operator sources

Move to node with access to the OpenShift cluster.

Crate YAML file

$ cat disableAllDefaultSources.yaml 
apiVersion: config.openshift.io/v1
kind: OperatorHub
metadata:
  name: cluster
spec:
  disableAllDefaultSources: true

Apply `oc apply -f disableAllDefaultSources.yaml`

### Apply new image contents

Move `catalogSource.yaml`, `imageContentSourcePolicy.yaml` where you can access the OpenShift cluster.

`oc apply -f imageContentSourcePolicy.yaml`
`oc apply -f catalogSource.yaml`

# Install LVM Operator

Make sure the target disks are clean (no partitions, no LV groups).
One way to check is to login to the SNO server with SSH and wipe disks

Run this for all disks that will be used for storage (e.g. `sda` and `sdb`, since `sdc` is used for root OS)

`sudo sgdisk --zap-all /dev/sdb`

NOTE! Make sure to NOT zap the rood disk `sdc`:) That would destroy OpenShift installation.

* Navigate to `Operators -> OperatorHub`, search for `LVM.`. Install with default options.
* Once operator is installed, navigate `Operators -> Installed Operators`. Select `LVM -> LVMCluster`. Click `Create LVMCluster`. Use all default options.
* Check for storage class `oc get sc`
* Set it as default storage class `oc patch storageclass odf-lvm-vg1 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'`

# Install ACM

* Navigate to `Operators -> OperatorHub`, search for `Advanced` and select `Advanced Cluster Management for Kuberenetes`. Install with default options.
* Once finished, navigate to `Operators -> Installed Operators`, select ACM. Click on `Create MultiClusterHub` and install with default options.

# Configure ACM

This section enables Assisted Service and related configuration for ACM that allows Zero-Touch Provisioning deployments.

### Update Hive

oc patch hiveconfig hive --type merge -p '{"spec":{"targetNamespace":"hive","logLevel":"debug","featureGates":{"custom":{"enabled":["AlphaAgentInstallStrategy"]},"featureSet":"Custom"}}}'

Check the pods are restarting
`$ oc get pods -n hive`

### Setup Assisted Service

Log config

$ cat 01-assisted-service-config.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: assisted-service-config
  namespace: open-cluster-management
  labels:
    app: assisted-service
data:
  LOG_LEVEL: "debug"

`oc apply -f 01-assisted-service-config.yaml`

Registry config

$ cat 02-mirror-registry-config-map.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: mirror-registry-config-map
  namespace: multicluster-engine
  labels:
    app: assisted-service
data:
  registries.conf: |
    unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

    [[registry]]
    prefix = ""
    location = "quay.io/openshift-release-dev/ocp-release"
    #mirror-by-digest-only = false
    
    [[registry.mirror]]
    location = "nexus.kskels.com/ocp4/openshift4"
    insecure = false
    
    [[registry]]
    prefix = ""
    location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
    #mirror-by-digest-only = false
    
    [[registry.mirror]]
    location = "nexus.kskels.com/ocp4/openshift4"
    insecure = false

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/rhacm2"
    #mirror-by-digest-only = false
    
    [[registry.mirror]]
    location = "nexus.kskels.com/rhacm2"
    insecure = false

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/multicluster-engine"
    #mirror-by-digest-only = false
    
    [[registry.mirror]]
    location = "nexus.kskels.com/multicluster-engine"
    insecure = false

Agent Service

apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
  name: agent
  namespace: open-cluster-management
spec:
  databaseStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 15Gi
    # storageClassName: ocs-storagecluster-ceph-rbd
  filesystemStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 35Gi
    # storageClassName: ocs-storagecluster-ceph-rbd
  osImages:
  - openshiftVersion: "4.10"
    cpuArchitecture: x86_64
    rootFSUrl: https://nexus.kskels.com:8443/repository/images/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live-rootfs.x86_64.img
    url: https://nexus.kskels.com:8443/repository/images/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live.x86_64.iso
    version: 410.84.202201251210-0

  mirrorRegistryRef:
    name: mirror-registry-config-map

Check that service is up and running and healthy

$ oc get pods -n multicluster-engine |grep assisted
assisted-image-service-0                               1/1     Running   0          3m10s
assisted-service-6cd95cc7c9-f9ssf                      2/2     Running   0          3m10s

Metal3 config

$ cat 04-metal-provisioning.yaml 
apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  finalizers:
  - provisioning.metal3.io
  name: provisioning-configuration
spec:
  preProvisioningOSDownloadURLs: {}
  provisioningNetwork: Disabled
  watchAllNamespaces: true

Check metal3 nodes are up and running and healthy

$ oc get pods -A |grep metal
openshift-machine-api                              cluster-baremetal-operator-54b86b9d8c-nm4zf                       2/2     Running       0             16h
openshift-machine-api                              metal3-85cc54d944-httd6                                           7/7     Running       0             3m58s
openshift-machine-api                              metal3-image-cache-7sx8t                                          1/1     Running       0             3m58s
openshift-machine-api                              metal3-image-customization-6bb766f8f4-jjmmt                       1/1     Running       0             3m57s

We are ready to ZTP!

# ISOs

  - openshiftVersion: "4.10"
    cpuArchitecture: x86_64
    rootFSUrl: https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live-rootfs.x86_64.img
    url: https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.10/4.10.3/rhcos-4.10.3-x86_64-live.x86_64.iso
    version: 410.84.202201251210-0
    

# Secret for ZTP

apiVersion: v1
kind: Secret
metadata:
  name: pull-secret
  namespace: todd-atx-kskels-com
stringData:
  .dockerconfigjson: '{}'

# Live ISOs

`https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.10/4.10.16/`

# Ceph

oc edit storagecluster ocs-storagecluster

Add

  network:
    ipFamily: IPv6

# oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch  '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

# ceph -s
oc -n openshift-storage exec $(oc -n openshift-storage get pod  -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph -s

# oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

## clean disks

DISK="/dev/sdX"

# Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean)
sgdisk --zap-all $DISK

# Wipe a large portion of the beginning of the disk to remove more LVM metadata that may be present
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync

# SSDs may be better cleaned with blkdiscard instead of dd
blkdiscard $DISK

# Inform the OS of partition table changes
partprobe $DISK

## clean old Ceph

# This command hangs on some systems: with caution, 'dmsetup remove_all --force' can be used
ls /dev/mapper/ceph-* | xargs -I% -- dmsetup remove %

# ceph-volume setup can leave ceph-<UUID> directories in /dev and /dev/mapper (unnecessary clutter)
rm -rf /dev/ceph-*
rm -rf /dev/mapper/ceph--*

# Metal3

`oc get deploy -n openshift-machine-api metal3 -o yaml > metal3.backup`

`oc edit deploy -n openshift-machine-api metal3`

Remove `ownerReferences` section from metal3 deployment

  ownerReferences:
  - apiVersion: metal3.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Provisioning
    name: provisioning-configuration
    uid: 0a13348e-b17d-42f5-bb5f-156fa0a9708e

Set `nodeSelector` for a specific node

      nodeSelector:
        kubernetes.io/hostname: ocp01-node03

# Ingress Certs

Steps to change the default certs

__CA Bundle__

oc create \n
configmap custom-ca \n
--from-file=ca-bundle.crt=</path/to/example-ca.crt> \n
-n openshift-config

__Patch Cluster__

oc patch proxy/cluster \n
--type=merge  \n
--patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

__CA Secret__

oc create \n 
secret tls <secret> \n
--cert=</path/to/cert.crt> \n 
--key=</path/to/cert.key> -n openshift-ingress

__Patch Ingress Controller__

oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "<secret>"}}}' -n openshift-ingress-operator

# Console Customization

__Logo__

oc create configmap console-custom-logo \n 
--from-file /path/to/console-custom-logo.png \n 
-n openshift-config

apiVersion: operator.openshift.io/v1
kind: Console
metadata:
  name: cluster
spec:
  customization:
    customLogoFile:
      key: console-custom-logo.png
      name: console-custom-logo
    customProductName: My Console