---
# System prepended metadata

title: Deploy Tiny OKD4 VM Cluster
tags: [okd]

---

<style>.ui-infobar, #doc.markdown-body { max-width: 1200px; }</style>

# Deploy Tiny OKD4 VM Cluster

###### tags: `okd`

> 2022-08-06: Draft Version.
> 2022-09-30: Add ignition re-create script.

#### Overview
A tiny OKD4 (4.10) VM cluster (1 bastion + 1 bootstrap + 1 master + 1 worker) on a KVM Server (physical). 
The bootstrap could be deleted after cluster deploy completed.

The KVM server has 64GB RAM, 4 CPU Cores (8 Threads), running Ubuntu 20.04LTS.
There is client host for access web console.
```
[Client] | {Physical Network} | [KVM Server] | {Virtual Network} | [OKD4 VMs]
    .15  |   192.168.1.0/24   | .27      .1  | 192.168.122.0/24  | .50
         |                    |              |                   | .51
         |                    |              |                   | .52
         |                    |              |                   | .53
```

Base domain:  `okd.homelab.com`
Cluster name: `okd4`

Tool and Image Version:
- openshift-install version **4.10.0-0.okd-2022-07-09-073606**
- openshift-client (oc, kubectl) version **4.10.0-0.okd-2022-07-09-073606**
- Fedora CoreOS metal raw image version **36.20220703.3.1**
- Fedora CoreOS ISO image version **fedora-coreos-36.20220703.3.1-live.x86_64.iso**

#### VM Specification
| VM Name            | OS               | vCPU | RAM  | Storage | IP Address     |
| ------------------ | -----------------| ---- | ---- | ------- | -------------- |
| ssd-okd4-services  | CentOS 7.9       | 2    | 2GB  | 32GB    | 192.168.122.50 |
| ssd-okd4-bootstrap | Fedora CoreOS 36 | 2    | 8GB  | 32GB    | 192.168.122.51 |
| ssd-okd4-master    | Fedora CoreOS 36 | 8    | 16GB | 64GB    | 192.168.122.52 |
| ssd-okd4-worker-1  | Fedora CoreOS 36 | 2    | 8GB  | 64GB    | 192.168.122.53 |

#### Environment Description
- Cluster connects libvirt "default" virtual network 192.168.122.0/24 (NAT and DHCP enabled), Gateway 192.168.122.1.
- If "virtIO" is used for storage bus driver, disk device name will be /dev/vda.
- Use "virsh net-update" command to bind IP address with MAC to the VMs, for example.
    ```
    virsh net-update default add ip-dhcp-host \
    "<host mac='52:54:00:a6:1b:66' name='ssd-okd4-services' ip='192.168.122.50'/>" \
    --live --config
    ```
- The ssd-okd4-services VM is known as bastion node.
- For bootstrap, master and worker VMs, IP address are configured directly instead of using DHCP. If IP configured via DHCP, Network Manager on Fedore CoreOS adds gateway IP to first nameserver (to be confirmed).
- For bootstrap, master and worker VMs, make boot order to "VirtIO Disk" first and "CDROM" second. (mount Fedora CoreOS ISO image to CDROM).
- The bootstrap node VM must have minimum 8GB RAM in order to complete installation. Otherwise container stuck with error that run out of space in "/run".
- The master node VM must have minimum 16GB RAM and 8 vCPU (maybe 4vCPU works, but not tested yet). Otherwise some containers might stuck in pending status due to lack of resources.


#### 1. Setup OKD4 Services Node

##### 1.1 Install DNS (named) server
Install DNS packages
```
[root@ssd-okd4-services ~]# yum install -y vim bind bind-utils
```
Configure named configuration: **/etc/named.conf**
```
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
// See the BIND Administrator's Reference Manual (ARM) for details about the
// configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html

options {
	listen-on port 53 { 127.0.0.1; 192.168.122.50; };
	#listen-on-v6 port 53 { ::1; };
	directory 	"/var/named";
	dump-file 	"/var/named/data/cache_dump.db";
	statistics-file "/var/named/data/named_stats.txt";
	memstatistics-file "/var/named/data/named_mem_stats.txt";
	recursing-file  "/var/named/data/named.recursing";
	secroots-file   "/var/named/data/named.secroots";
	allow-query     { localhost; 192.168.122.0/24; };

	/* 
	 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
	 - If you are building a RECURSIVE (caching) DNS server, you need to enable 
	   recursion. 
	 - If your recursive DNS server has a public IP address, you MUST enable access 
	   control to limit queries to your legitimate users. Failing to do so will
	   cause your server to become part of large scale DNS amplification 
	   attacks. Implementing BCP38 within your network would greatly
	   reduce such attack surface 
	*/
	recursion yes;

    forwarders {
        8.8.8.8;
        8.8.4.4;
    };
    
	dnssec-enable yes;
	dnssec-validation yes;

	/* Path to ISC DLV key */
	bindkeys-file "/etc/named.root.key";

	managed-keys-directory "/var/named/dynamic";

	pid-file "/run/named/named.pid";
	session-keyfile "/run/named/session.key";
};

logging {
        channel default_debug {
                file "data/named.run";  // fullpath: /var/named/data/named.run
                severity dynamic;
        };
};

zone "." IN {
	type hint;
	file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

// zone files
zone "okd.homelab.com" IN {
    type master;
    file "forward.okd.homelab.com";
};
zone "122.168.192.in-addr.arpa" IN {
    type master;
    file "reverse.okd.homelab.com";
};
```
Create forward zone file: **/var/named/forward.okd.homelab.com**
```
$TTL	86400  ; 24 hours could have been written as 24h or 1D

@ IN SOA okd4-services.okd.homelab.com. (
        admin.okd.homelab.com. ; responsible party
        2022072001  ; serial
        3H  ; refresh
        15  ; retry
        1w  ; expire
        3h  ; minimum
)

; name servers - NS records
  IN NS okd4-services

; name server - A records
okd4-services.okd.homelab.com. IN A 192.168.122.50

; okd4 container platform cluster - A records
okd4-bootstrap.okd4.okd.homelab.com. IN A 192.168.122.51
okd4-master.okd4.okd.homelab.com.    IN A 192.168.122.52
okd4-worker-1.okd4.okd.homelab.com.  IN A 192.168.122.53

; okd4 internal cluster IPs - A records
api.okd4.okd.homelab.com.     IN A 192.168.122.50
api-int.okd4.okd.homelab.com. IN A 192.168.122.50
*.apps.okd4.okd.homelab.com.  IN A 192.168.122.50
etcd-0.okd4.okd.homelab.com.  IN A 192.168.122.52
console-openshift-console.apps.okd4.okd.homelab.com. IN A 192.168.122.50
oauth-openshift.apps.okd4.okd.homelab.com. IN A 192.168.122.50

; okd4 internal cluster IPs - SRV records
_etcd-server-ssl._tcp.okd4.okd.homelab.com. 86400 IN SRV 0 10 2380 etcd-0.okd4
```
Configure reverse zone file: **/var/named/reverse.okd.homelab.com**
```
$TTL 604800
@ IN SOA okd4-services.okd.homelab.com. (
        admin.okd.homelab.com.  ; responsible party 
        2022072001  ; serial
        3H  ; refresh
        15  ; retry
        1w  ; expire
        3h  ; minimum
)

; name servers - NS records
@ IN NS okd4-services.okd.homelab.com.

; name servers - PTR records
50 IN PTR okd4-services.okd.homelab.com.

; OpenShift Container Platform Cluster - PTR records
51 IN PTR okd4-bootstrap.okd4.okd.homelab.com.
52 IN PTR okd4-master.okd4.okd.homelab.com.
53 IN PTR okd4-worker-1.okd4.okd.homelab.com.
50 IN PTR api.okd4.okd.homelab.com.
50 IN PTR api-int.okd4.okd.homelab.com.
```
Validate configure and start named service
```
[root@ssd-okd4-services ~]# named-checkconf /etc/named.conf

[root@ssd-okd4-services named]# named-checkzone okd.homelab.com /var/named/forward.okd.homelab.com
zone okd.homelab.com/IN: loaded serial 2022072001
OK
[root@ssd-okd4-services named]# named-checkzone 122.168.192.in-addr.arpa /var/named/reverse.okd.homelab.com
zone 122.168.192.in-addr.arpa/IN: loaded serial 2022072001
OK

[root@ssd-okd4-services ~]# systemctl enable named
[root@ssd-okd4-services ~]# systemctl start named
[root@ssd-okd4-services ~]# systemctl status named
[root@ssd-okd4-services ~]# sudo firewall-cmd --permanent --add-port=53/udp
success
[root@ssd-okd4-services ~]# sudo firewall-cmd --reload
success
```
Disable Network Manager to update **/etc/resolv.conf**
Adding `dns=none` and `rc-manager=unmanaged` to `main` section in **/etc/NetworkManager/NetworkManager.conf**
```
[root@ssd-okd4-services ~]# cat /etc/NetworkManager/NetworkManager.conf | grep -A 4 "\[main\]"
[main]
#plugins=ifcfg-rh,ibft
dns=none
rc-manager=unmanaged

[root@ssd-okd4-services ~]# sudo systemctl restart NetworkManager
```
Edit **resolv.conf** to use localhost DNS server.
```[root@ssd-okd4-services ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 127.0.0.1
```
Test DNS domain resolution.
```
[root@ssd-okd4-services ~]# dig +short okd4-services.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short okd4-bootstrap.okd4.okd.homelab.com 
192.168.122.51
[root@ssd-okd4-services ~]# dig +short okd4-master.okd4.okd.homelab.com 
192.168.122.52
[root@ssd-okd4-services ~]# dig +short okd4-worker-1.okd4.okd.homelab.com 
192.168.122.53
[root@ssd-okd4-services ~]# dig +short api.okd4.okd.homelab.com 
192.168.122.50
[root@ssd-okd4-services ~]# dig +short api-int.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short *.apps.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short etcd-0.okd4.okd.homelab.com
192.168.122.52
[root@ssd-okd4-services ~]# dig +short console-openshift-consoe.apps.okd4.okd.homelab.com 
192.168.122.50
[root@ssd-okd4-services ~]# dig +short oauth-openshift.apps.okd4.okd.homelab.com
192.168.122.50

[root@ssd-okd4-services ~]# dig -x 192.168.122.50 +short
api-int.okd4.okd.homelab.com.
api.okd4.okd.homelab.com.
okd4-services.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.51 +short
okd4-bootstrap.okd4.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.52 +short
okd4-master.okd4.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.53 +short
okd4-worker-1.okd4.okd.homelab.com.
```
**_NOTE:_** Or use other dig options "dig +noall +answer" and specify DNS IP, for example:
```
[root@ssd-okd4-services ~]# dig +noall +answer @192.168.122.50 okd4-services.okd.homelab.com
okd4-services.okd.homelab.com. 86400 IN	A	192.168.122.50
[root@ssd-okd4-services ~]# dig +noall +answer @192.168.122.50 -x 192.168.122.51
51.122.168.192.in-addr.arpa. 604800 IN	PTR	okd4-bootstrap.okd4.okd.homelab.com.
```

##### 1.2 Install Haproxy Server
Install haproxy package
```
[root@ssd-okd4-services ~]# yum install haproxy -y
```
Configure haproxy **/etc/haproxy/haproxy.cfg**
```
#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen stats
    bind :9000
    mode http
    stats enable
    stats uri /

# API
frontend okd4_api_frontend
    bind :6443
    default_backend okd4_api_backend
    mode tcp
    option tcplog
backend okd4_api_backend
    balance source
    mode tcp
    server okd4-bootstrap 192.168.122.51:6443 check
    server okd4-master    192.168.122.52:6443 check

# Machine Config Server
frontend okd4_machine_config_server_frontend
    bind :22623
    default_backend okd4_machine_config_server_backend
    mode tcp
    option tcplog
backend okd4_machine_config_server_backend
    balance source
    mode tcp
    server okd4-bootstrap 192.168.122.51:22623 check
    server okd4-master    192.168.122.52:22623 check

# HTTP Ingress
frontend okd4_http_ingress_frontend
    bind :80
    default_backend okd4_http_ingress_backend
    mode tcp
    option tcplog
backend okd4_http_ingress_backend
    balance source
    mode tcp
    server okd4-worker-1    192.168.122.53:80 check

# HTTPS Ingress
frontend okd4_http_ingress_frontend
    bind *:443
    default_backend okd4_https_ingress_backend
    mode tcp
    option tcplog
backend okd4_https_ingress_backend
    balance source
    mode tcp
    server okd4-worker-1    192.168.122.53:443 check
```
Start HAProxy server
```
[root@ssd-okd4-services ~]# setsebool -P haproxy_connect_any 1
[root@ssd-okd4-services ~]# systemctl enable haproxy
Created symlink from /etc/systemd/system/multi-user.target.wants/haproxy.service to /usr/lib/systemd/system/haproxy.service.
[root@ssd-okd4-services ~]# systemctl start haproxy
[root@ssd-okd4-services ~]# systemctl status haproxy

[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=6443/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=22623/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-service=http
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-service=https
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success
```

##### 1.3 Install Apache Server
Install apache package
```
[root@ssd-okd4-services ~]# yum install httpd -y
```
Set listening port to 8080 in **/etc/httpd/conf/httpd.conf**
```
#Listen 12.34.56.78:80
Listen 8080
```
Start server
```
[root@ssd-okd4-services ~]# setsebool -P httpd_read_user_content 1
[root@ssd-okd4-services ~]# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
[root@ssd-okd4-services ~]# systemctl start httpd
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=8080/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success

[root@ssd-okd4-services ~]# curl -s http://192.168.122.50:8080 | tail
               the sources of Red Hat Enterprise Linux (RHEL).<p>
            
            <p>Additionally to being a popular choice for web hosting, CentOS also provides a rich platform for open source communities to build upon. For more information
               please visit the <a href="http://www.centos.org/">CentOS website</a>.</p>
          </div>
        </div>
		  </div>
    </div>
  </div>
</body></html>

```

##### 1.4 Install openshift-installer and oc client
Download openshift oc client and openshift-install form **https://github.com/openshift/okd/releases**
```
[root@ssd-okd4-services ~]# mkdir openshift-install
[root@ssd-okd4-services ~]# cd openshift-install

[root@ssd-okd4-services okd4-install]# wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
[root@ssd-okd4-services okd4-install]# wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz

[root@ssd-okd4-services okd4-install]# tar -xvf ./openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz 
README.md
oc
kubectl
[root@ssd-okd4-services okd4-install]# tar -xvf ./openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz 
README.md
openshift-install

[root@ssd-okd4-services okd4-install]# ls -l
total 1020792
-rwxr-xr-x. 1 root root 122829960 Jun 22 04:12 kubectl
-rwxr-xr-x. 1 root root 122829960 Jun 22 04:12 oc
-rw-rw-r--. 1 root root  49524984 Jul 11 20:42 openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
-rwxr-xr-x. 1 root root 619614208 Jun 22 01:44 openshift-install
-rw-rw-r--. 1 root root 130480916 Jul 11 20:36 openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
-rw-r--r--. 1 root root       706 Jun 22 01:44 README.md

[root@ssd-okd4-services okd4-install]# ./oc version
Client Version: 4.10.0-0.okd-2022-07-09-073606

[root@ssd-okd4-services okd4-install]# ./openshift-install version
./openshift-install 4.10.0-0.okd-2022-07-09-073606
built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
release image quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d
release architecture amd64
```
Generate ssh key pair if you dont have one.
```
[root@ssd-okd4-services okd4-install]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:4ZpNqumApn8ec7R0EUqGX7rWxiLodOSeoXQcNstQiRI root@ssd-okd4-services
The key's randomart image is:
+---[RSA 2048]----+
|E. . oo .        |
|. . +o ...       |
| . . ..oo        |
|  . = o. o       |
|   O +o+S        |
| .+ Xo+O+        |
|.=.=o=*o.        |
|o o.+*           |
|...+=            |
+----[SHA256]-----+
```
Create **install-config.yaml**
```
[root@ssd-okd4-services okd4-install]# cat install-config.yaml
apiVersion: v1
baseDomain: okd.homelab.com
metadata:
  name: okd4

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 1

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 1

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork: 
  - 172.30.0.0/16

platform:
  none: {}

fips: false

pullSecret: '{"auths":{"fake":{"auth": "bar"}}}' 
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG9D6Tj8CFJNI/FP2nZcU7n4IgjaOZJ4xH5ljoaKu/C1ppzSG8g/zgUTJmr/o+NpWz4kRYFG+k7GK6jALJp+WY24MC9nOzdn4znITRgNzSAjUUL+qq05NWpRqHVhudtIFNLG2zcp6ct4pzZlxKMtincKlI6EdpLgMYTBxflJRe7aP5xByUCQGr1uqrirLTInFwzyqzHrhUOX9l+pwvAqTFhkSrXz/t7pgE7FJ/p7Glb7npwOaCFYkOiw4u/M3yzO8qVof/rL145yJBQJCEA8KrChLr/l/QMGfD0YNZbNRTjl0Td+5L/MXHb05g+5LpPMZZg8H8S+oDDpKoCjvC1/gd root@ssd-okd4-services'
```
* Copy the public key in /root/.ssh/id_rsa.pub to the "sshKey" value.
* The OKD cluster domain will be "metadata name" + "baseDomain", which is "okd4.okd.homelab.com"
* set platform to "none: {}" for user-provisioned cluster.

Copy **install-config.yaml** to **install_dir** directory
```
[root@ssd-okd4-services okd4-install]# mkdir install_dir
[root@ssd-okd4-services okd4-install]# cp ./install-config.yaml ./install_dir/
```
Create manifests in **install_dir** directory
```
[root@ssd-okd4-services okd4-install]# ./openshift-install create manifests --dir=install_dir
INFO Consuming Install Config from target directory 
INFO Manifests created in: install_dir/manifests and install_dir/openshift 
[root@ssd-okd4-services okd4-install]# ls -l ./install_dir/
total 8
drwxr-x---. 2 root root 4096 Jul 24 22:15 manifests
drwxr-x---. 2 root root 4096 Jul 24 22:15 openshift
```

Create ignition configs in **install_dir** directory (manifests will be cleared)
```
[root@ssd-okd4-services okd4-install]# ./openshift-install create ignition-configs --dir=install_dir
INFO Consuming Common Manifests from target directory 
INFO Consuming Master Machines from target directory 
INFO Consuming OpenShift Install (Manifests) from target directory 
INFO Consuming Openshift Manifests from target directory 
INFO Consuming Worker Machines from target directory 
INFO Ignition-Configs created in: install_dir and install_dir/auth 
[root@ssd-okd4-services okd4-install]# ls -l ./install_dir/
total 276
drwxr-x---. 2 root root     50 Jul 24 22:16 auth
-rw-r-----. 1 root root 269438 Jul 24 22:16 bootstrap.ign
-rw-r-----. 1 root root   1722 Jul 24 22:16 master.ign
-rw-r-----. 1 root root     96 Jul 24 22:16 metadata.json
-rw-r-----. 1 root root   1722 Jul 24 22:16 worker.ign
```
Host `ignition` configs and `Fedore CoreOS bare-metal image` and `sig` files on Apache server.
```
[root@ssd-okd4-services okd4-install]# mkdir /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# cp -afpR install_dir/* /var/www/html/okd4/

# update file owner, permission and security context
[root@ssd-okd4-services okd4-install]# chown -R apache: /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chmod -R 755 /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chcon -R -t httpd_sys_content_t /var/www/html/

# test access toignition files
[root@ssd-okd4-services okd4-install]# curl localhost:8080/okd4/
```
```
[root@ssd-okd4-services okd4-install]# wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/36.20220703.3.1/x86_64/fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz

[root@ssd-okd4-services okd4-install]# wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/36.20220703.3.1/x86_64/fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz.sig

[root@ssd-okd4-services okd4-install]# mv ./fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz fcos36.raw.xz
[root@ssd-okd4-services okd4-install]# mv ./fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz.sig fcos36.raw.xz.sig
[root@ssd-okd4-services okd4-install]# mv ./fcos36.raw.xz ./fcos36.raw.xz.sig /var/www/html/okd4/

# update file owner, permission and security context
[root@ssd-okd4-services okd4-install]# chown -R apache: /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chmod -R 755 /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chcon -R -t httpd_sys_content_t /var/www/html/
```

##### 1.5 add OKD nodes to **/etc/hosts**
```
[root@ssd-okd4-services ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.122.51 bootstrap
192.168.122.52 master
192.168.122.53 worker-1
```

#### 2. Deploy OKD4 Bootstrap Node

##### 2.1 Power on Bootstrap VM and press "tab" key to edit kernel boot options .
![](https://i.imgur.com/YidiWYg.png)

##### 2.2 Append extra kernel boot options.
``` 
ip=192.168.122.51::192.168.122.1:255.255.255.0:::none 
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda 
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz 
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/bootstrap.ign
```
![](https://i.imgur.com/EwmsezR.png)
**_NOTE:_** Take snapshot after entered kernel boot options (do not press enter). This is for case if install fail and resume to entering the boot options.

Press enter to install, you should see image and signature downloading.
![](https://i.imgur.com/0YsJmRB.png)

After system installation completed, it will be rebooted and boot from disk.
![](https://i.imgur.com/tlYsICs.png)

Wait the bootstrap node runs an minimal Kubernetes for deploy master node.

##### 2.3 SSH to bootstrap node
```
[root@ssd-okd4-services ~]# ssh bootstrap -l core
The authenticity of host 'bootstrap (192.168.122.51)' can't be established.
ECDSA key fingerprint is SHA256:Blrhzptpx3fjA6lj97r78eZPcZkBAMuB4CKY4Wm8b2o.
ECDSA key fingerprint is MD5:ac:1a:5f:f1:e9:10:28:7e:60:c3:43:12:ae:8a:63:97.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bootstrap,192.168.122.51' (ECDSA) to the list of known hosts.
This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

  journalctl -b -f -u release-image.service -u bootkube.service
Fedora CoreOS 36.20220703.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[core@ssd-okd4-bootstrap ~]$ 
```
**_NOTE:_** To check release-image and bootkube services on the bootstrap node, excute `journalctl -b -f -u release-image.service -u bootkube.service`
**_NOTE:_** Monitor bootstrap status with `openshift-install` tool.
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606 
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da 
INFO Waiting up to 20m0s (until 12:15AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443... 
```
**_NOTE:_** The bootstrap node will be rebooted again while bootstrapping.
**_NOTE:_** When the boostrap node successfully running an minimum kubernetes, you should see "waiting for bootstrapping to complete" message.
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606 
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da 
INFO Waiting up to 20m0s (until 12:42AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443... 
INFO API v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty up 
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Networking...                      
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG Using Install Config loaded from state file  
INFO Waiting up to 30m0s (until 12:52AM) for bootstrapping to complete...
```
**_NOTE:_** In bootstrap node, check the minimum kuberntes PODs are running.
```
[root@okd4-bootstrap ~]# crictl pods
POD ID              CREATED             STATE               NAME                                                                     NAMESPACE                             ATTEMPT             RUNTIME
e4c0d1c3d80a4       4 minutes ago       Ready               bootstrap-kube-scheduler-okd4-bootstrap.okd4.okd.homelab.com             kube-system                           0                   (default)
23c60759fae12       4 minutes ago       Ready               bootstrap-kube-apiserver-okd4-bootstrap.okd4.okd.homelab.com             openshift-kube-apiserver              0                   (default)
788ebe7e5cd0b       4 minutes ago       Ready               bootstrap-kube-controller-manager-okd4-bootstrap.okd4.okd.homelab.com    kube-system                           0                   (default)
e984eb128192f       4 minutes ago       Ready               cloud-credential-operator-okd4-bootstrap.okd4.okd.homelab.com            openshift-cloud-credential-operator   0                   (default)
b3ad5da52a9ba       4 minutes ago       Ready               bootstrap-cluster-version-operator-okd4-bootstrap.okd4.okd.homelab.com   openshift-cluster-version             0                   (default)
59a6e3da5a49e       5 minutes ago       Ready               etcd-bootstrap-member-okd4-bootstrap.okd4.okd.homelab.com                openshift-etcd                        0                   (default)
589cb66783f63       5 minutes ago       Ready               bootstrap-machine-config-operator-okd4-bootstrap.okd4.okd.homelab.com    default                               0                   (default)
```

#### 3. Deploy OKD4 Master Node

##### 3.1 Power on Master VM and press "tab" key to edit kernel boot options
##### 3.2 Append extra options below and press enter to start installation
```
ip=192.168.122.52::192.168.122.1:255.255.255.0:::none
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda 
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz 
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/master.ign
```
![](https://i.imgur.com/nEuAB81.png)

**_NOTE:_** Check master node kubelet service.
```
[root@ssd-okd4-services ~]# ssh master -l core
The authenticity of host 'master (192.168.122.52)' can't be established.
ECDSA key fingerprint is SHA256:YBKgKgqmNgCY5noB2LhZx8fExr974Uejz5OyDhApDZg.
ECDSA key fingerprint is MD5:09:1f:3b:6b:0c:80:c0:a2:92:f6:35:38:91:64:4a:c2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.122.52' (ECDSA) to the list of known hosts.
Fedora CoreOS 35
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[core@okd4-master ~]$ 
[core@okd4-master ~]$ sudo su -
[root@okd4-master ~]# journalctl -u kubelet -f
```
**_NOTE:_** Check bootstrapping completed if master node deployed successfully. You should see `DEBUG Bootstrap status: complete`..
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606 
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da 
INFO Waiting up to 20m0s (until 1:20AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443... 
INFO API v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty up 
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Networking...                      
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG Using Install Config loaded from state file  
INFO Waiting up to 30m0s (until 1:30AM) for bootstrapping to complete... 
DEBUG Bootstrap status: complete                   
INFO It is now safe to remove the bootstrap resources 
INFO Time elapsed: 0s 
```

##### 3.3 Set the oc command on service node.
```
[root@ssd-okd4-services okd4-install]# cp ./oc /usr/bin/
[root@ssd-okd4-services okd4-install]# which oc
/usr/bin/oc
[root@ssd-okd4-services okd4-install]# mkdir ~/.kube/
[root@ssd-okd4-services okd4-install]# cp ./install_dir/auth/kubeconfig ~/.kube/config
[root@ssd-okd4-services okd4-install]# ls -l ~/.kube/config 
-rw-r-----. 1 root root 8946 Aug  6 00:09 /root/.kube/config
```
**_NOTE_:** Or export "KUBECONFIG" instead of ".kube/config" : `export KUBECONFIG=/root/okd4-install/install_dir/auth/kubeconfig` 


##### 3.4 Check pods status on master node
Check node status
```
[root@ssd-okd4-services okd4-install]# oc whoami
system:admin
[root@ssd-okd4-services okd4-install]# oc get node
NAME                               STATUS   ROLES    AGE   VERSION
okd4-master.okd4.okd.homelab.com   Ready    master   27m   v1.23.5+3afdacb
```

Check clusteroperators
```
[root@ssd-okd4-services okd4-install]# oc get clusteroperators
NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.okd-2022-07-09-073606   False       False         True       18m     OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.122.29:443/healthz": dial tcp 172.30.122.29:443: connect: connection refused...
baremetal                                  4.10.0-0.okd-2022-07-09-073606   True        False         False      17m     
cloud-controller-manager                   4.10.0-0.okd-2022-07-09-073606   True        False         False      24m     
cloud-credential                           4.10.0-0.okd-2022-07-09-073606   True        False         False      45m     
cluster-autoscaler                         4.10.0-0.okd-2022-07-09-073606   True        False         False      17m     
config-operator                            4.10.0-0.okd-2022-07-09-073606   True        False         False      19m     
console                                    4.10.0-0.okd-2022-07-09-073606   False       False         True       8m27s   RouteHealthAvailable: console route is not admitted
csi-snapshot-controller                    4.10.0-0.okd-2022-07-09-073606   True        False         False      18m     
dns                                        4.10.0-0.okd-2022-07-09-073606   True        False         False      15m     
etcd                                       4.10.0-0.okd-2022-07-09-073606   True        False         False      15m     
image-registry                             4.10.0-0.okd-2022-07-09-073606   True        False         False      9m8s    
ingress                                    4.10.0-0.okd-2022-07-09-073606   True        False         True       16m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-8ff64bc5b-z4fnl" cannot be scheduled: 0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Make sure you have sufficient worker nodes.)
insights                                   4.10.0-0.okd-2022-07-09-073606   True        False         False      49s     
kube-apiserver                             4.10.0-0.okd-2022-07-09-073606   True        False         False      11m     
kube-controller-manager                    4.10.0-0.okd-2022-07-09-073606   True        False         False      11m     
kube-scheduler                             4.10.0-0.okd-2022-07-09-073606   True        False         False      11m     
kube-storage-version-migrator              4.10.0-0.okd-2022-07-09-073606   True        False         False      10m     
machine-api                                4.10.0-0.okd-2022-07-09-073606   True        False         False      10m     
machine-approver                           4.10.0-0.okd-2022-07-09-073606   True        False         False      17m     
machine-config                                                              True        True          True       7m27s   Unable to apply 4.10.0-0.okd-2022-07-09-073606: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1)
marketplace                                4.10.0-0.okd-2022-07-09-073606   True        False         False      16m     
monitoring                                                                  Unknown     True          Unknown    17m     Rolling out the stack.
network                                    4.10.0-0.okd-2022-07-09-073606   True        True          False      20m     Deployment "openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
node-tuning                                4.10.0-0.okd-2022-07-09-073606   True        False         False      15m     
openshift-apiserver                        4.10.0-0.okd-2022-07-09-073606   True        False         False      9m29s   
openshift-controller-manager               4.10.0-0.okd-2022-07-09-073606   True        False         False      10m     
openshift-samples                          4.10.0-0.okd-2022-07-09-073606   True        False         False      8m20s   
operator-lifecycle-manager                 4.10.0-0.okd-2022-07-09-073606   True        False         False      14m     
operator-lifecycle-manager-catalog         4.10.0-0.okd-2022-07-09-073606   True        False         False      11m     
operator-lifecycle-manager-packageserver   4.10.0-0.okd-2022-07-09-073606   True        False         False      11m     
service-ca                                 4.10.0-0.okd-2022-07-09-073606   True        False         False      18m     
storage                                    4.10.0-0.okd-2022-07-09-073606   True        False         False      19m 
```

Check all PODs, You should see PODs in Running/Pending/Completed status.
```
[root@ssd-okd4-services okd4-install]# oc get pods -A
NAMESPACE                                          NAME                                                         READY   STATUS      RESTARTS        AGE
openshift-apiserver-operator                       openshift-apiserver-operator-7b57fbbb4b-t4tjb                1/1     Running     1 (19m ago)     27m
openshift-apiserver                                apiserver-7744bdd5f5-6hfs4                                   2/2     Running     3 (14m ago)     17m
openshift-authentication-operator                  authentication-operator-7c67cf9d6d-px2rs                     1/1     Running     1 (20m ago)     27m
openshift-cloud-controller-manager-operator        cluster-cloud-controller-manager-operator-5b855fd95c-bhvl6   2/2     Running     0               27m
openshift-cloud-credential-operator                cloud-credential-operator-5448765698-h2xbr                   2/2     Running     0               27m
openshift-cluster-machine-approver                 machine-approver-6fb56d9777-72cn8                            2/2     Running     0               27m
openshift-cluster-node-tuning-operator             cluster-node-tuning-operator-5fb447984f-h7v7f                1/1     Running     0               27m
openshift-cluster-node-tuning-operator             tuned-cpbqs                                                  1/1     Running     0               17m
openshift-cluster-samples-operator                 cluster-samples-operator-84ddc465c4-rjrj9                    2/2     Running     0               11m
openshift-cluster-storage-operator                 cluster-storage-operator-5bc758cdc5-scv8h                    1/1     Running     1 (20m ago)     27m
openshift-cluster-storage-operator                 csi-snapshot-controller-6b56df796f-brpns                     1/1     Running     2 (14m ago)     21m
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-7c9b645b4f-v5r7r            1/1     Running     0               27m
openshift-cluster-storage-operator                 csi-snapshot-webhook-556df58965-d8hh5                        1/1     Running     0               21m
openshift-cluster-version                          cluster-version-operator-7d9c76888b-rvk78                    1/1     Running     0               27m
openshift-config-operator                          openshift-config-operator-669b58b97f-l2zz6                   1/1     Running     1 (20m ago)     27m
openshift-console-operator                         console-operator-9f59cb585-xftr5                             1/1     Running     0               11m
openshift-console                                  downloads-7945dd564-2bwnr                                    1/1     Running     0               10m
openshift-controller-manager-operator              openshift-controller-manager-operator-d6c8f59b8-qfwr6        1/1     Running     1 (19m ago)     27m
openshift-controller-manager                       controller-manager-6w9pr                                     1/1     Running     0               11m
openshift-dns-operator                             dns-operator-5499565b9d-9gwtr                                2/2     Running     0               27m
openshift-dns                                      dns-default-jjw7r                                            2/2     Running     0               19m
openshift-dns                                      node-resolver-g8gd6                                          1/1     Running     0               19m
openshift-etcd-operator                            etcd-operator-f9d78946f-drs2g                                1/1     Running     1 (20m ago)     27m
openshift-etcd                                     etcd-okd4-master.okd4.okd.homelab.com                        4/4     Running     0               14m
openshift-etcd                                     installer-2-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               20m
openshift-etcd                                     installer-4-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               17m
openshift-image-registry                           cluster-image-registry-operator-7dc6c985f8-nr9jh             1/1     Running     0               27m
openshift-image-registry                           node-ca-bfwb2                                                1/1     Running     0               114s
openshift-ingress-operator                         ingress-operator-69b895ccb8-292mh                            2/2     Running     3 (14m ago)     27m
openshift-ingress                                  router-default-8ff64bc5b-z4fnl                               0/1     Pending     0               19m
openshift-insights                                 insights-operator-7fbc56b87f-kw5f7                           1/1     Running     2 (14m ago)     27m
openshift-kube-apiserver-operator                  kube-apiserver-operator-5ffc5f559b-cr4c5                     1/1     Running     1 (20m ago)     27m
openshift-kube-apiserver                           installer-2-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               19m
openshift-kube-apiserver                           installer-3-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               17m
openshift-kube-apiserver                           installer-4-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               13m
openshift-kube-apiserver                           kube-apiserver-okd4-master.okd4.okd.homelab.com              5/5     Running     0               12m
openshift-kube-controller-manager-operator         kube-controller-manager-operator-578fb4ffb8-q9whf            1/1     Running     1 (20m ago)     27m
openshift-kube-controller-manager                  installer-5-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               19m
openshift-kube-controller-manager                  installer-6-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               13m
openshift-kube-controller-manager                  installer-7-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               8m29s
openshift-kube-controller-manager                  kube-controller-manager-okd4-master.okd4.okd.homelab.com     4/4     Running     1 (6m44s ago)   7m39s
openshift-kube-controller-manager                  revision-pruner-6-okd4-master.okd4.okd.homelab.com           0/1     Completed   0               11m
openshift-kube-controller-manager                  revision-pruner-7-okd4-master.okd4.okd.homelab.com           0/1     Completed   0               8m31s
openshift-kube-scheduler-operator                  openshift-kube-scheduler-operator-85c6464c44-kpx5l           1/1     Running     1 (20m ago)     27m
openshift-kube-scheduler                           installer-3-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               20m
openshift-kube-scheduler                           installer-4-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               19m
openshift-kube-scheduler                           installer-5-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               19m
openshift-kube-scheduler                           installer-6-okd4-master.okd4.okd.homelab.com                 0/1     Completed   0               13m
openshift-kube-scheduler                           openshift-kube-scheduler-okd4-master.okd4.okd.homelab.com    3/3     Running     0               12m
openshift-kube-scheduler                           revision-pruner-6-okd4-master.okd4.okd.homelab.com           0/1     Completed   0               11m
openshift-kube-storage-version-migrator-operator   kube-storage-version-migrator-operator-69d8d886cb-d8627      1/1     Running     1 (19m ago)     27m
openshift-kube-storage-version-migrator            migrator-85976b4574-qvgnl                                    1/1     Running     0               20m
openshift-machine-api                              cluster-autoscaler-operator-58cb8cc768-gbsrs                 2/2     Running     3 (14m ago)     27m
openshift-machine-api                              cluster-baremetal-operator-697f959f47-p8gh8                  2/2     Running     0               27m
openshift-machine-api                              machine-api-operator-7649c6f59-xwwc4                         2/2     Running     0               27m
openshift-machine-config-operator                  machine-config-controller-54d6bc656f-j8rjg                   1/1     Running     0               19m
openshift-machine-config-operator                  machine-config-daemon-wngv7                                  2/2     Running     0               22m
openshift-machine-config-operator                  machine-config-operator-59c476b849-kpvn6                     1/1     Running     0               27m
openshift-machine-config-operator                  machine-config-server-fkjmv                                  1/1     Running     0               19m
openshift-marketplace                              community-operators-jb8h9                                    1/1     Running     0               14m
openshift-marketplace                              marketplace-operator-5985df74f9-n5j6m                        1/1     Running     4 (14m ago)     27m
openshift-monitoring                               cluster-monitoring-operator-c48fd7c76-nwpbt                  2/2     Running     0               27m
openshift-monitoring                               kube-state-metrics-849c6bb5d5-b5x7x                          0/3     Pending     0               17m
openshift-monitoring                               node-exporter-5h8s9                                          2/2     Running     0               17m
openshift-monitoring                               openshift-state-metrics-7896d98645-4p9ms                     0/3     Pending     0               17m
openshift-monitoring                               prometheus-adapter-595764bbc-7vfbh                           0/1     Pending     0               11m
openshift-monitoring                               prometheus-operator-78bd5cc486-9jcxw                         2/2     Running     0               20m
openshift-multus                                   multus-9mhkw                                                 1/1     Running     0               26m
openshift-multus                                   multus-additional-cni-plugins-z6n4q                          1/1     Running     0               26m
openshift-multus                                   multus-admission-controller-wnspz                            2/2     Running     0               22m
openshift-multus                                   network-metrics-daemon-2nqqc                                 2/2     Running     0               26m
openshift-network-diagnostics                      network-check-source-6fdcb969d6-6nf7d                        0/1     Pending     0               26m
openshift-network-diagnostics                      network-check-target-jds26                                   1/1     Running     0               26m
openshift-network-operator                         network-operator-546db9df5d-d98n7                            1/1     Running     0               27m
openshift-oauth-apiserver                          apiserver-788c59c55b-m6lc2                                   1/1     Running     1 (14m ago)     17m
openshift-operator-lifecycle-manager               catalog-operator-55956f4487-dz542                            1/1     Running     0               27m
openshift-operator-lifecycle-manager               collect-profiles-27661860-6nsvt                              0/1     Pending     0               8m32s
openshift-operator-lifecycle-manager               olm-operator-775bfb89b7-l4bpq                                1/1     Running     0               27m
openshift-operator-lifecycle-manager               package-server-manager-6db8bb5585-7rfqb                      1/1     Running     1 (15m ago)     27m
openshift-operator-lifecycle-manager               packageserver-7fc675f65d-hhnkx                               1/1     Running     0               17m
openshift-ovn-kubernetes                           ovnkube-master-4sfzx                                         6/6     Running     0               26m
openshift-ovn-kubernetes                           ovnkube-node-4sgxs                                           5/5     Running     0               26m
openshift-service-ca-operator                      service-ca-operator-86bcf476fb-g4bgs                         1/1     Running     1 (19m ago)     27m
openshift-service-ca                               service-ca-54b4cf6549-kf8fn                                  1/1     Running     0               21m
```
**_Note_:** "installer" PODs might in error status which is fine because they are for configuring nodes. The pods will be re-run again if they have Error.

Check CSR status
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME                                             AGE   SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-5pjzn                                        35m   kubernetes.io/kubelet-serving                 system:node:okd4-master.okd4.okd.homelab.com                                      <none>              Approved,Issued
csr-fs7z4                                        36m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-kmw8k                                        36m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
system:openshift:openshift-authenticator-2pxvw   25m   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
system:openshift:openshift-monitoring-764wv      24m   kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-monitoring:cluster-monitoring-operator            <none>              Approved,Issued
```

#### 4. Deploy OKD4 Worker Node

##### 4.1 Power on Worker-1 VM and add boot options below to install.
```
ip=192.168.122.53::192.168.122.1:255.255.255.0:::none
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda 
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz 
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/worker.ign
```

##### 4.2 Check kubelet service
After installation completed and reboot, ssh to the worker node
```
[root@ssd-okd4-services okd4-install]# ssh worker-1 -l core
The authenticity of host 'worker-1 (192.168.122.53)' can't be established.
ECDSA key fingerprint is SHA256:uesRavGkWYbZmuTvwItQ4/SEljIxVWbkrUO9ATONJh0.
ECDSA key fingerprint is MD5:38:7a:f6:91:4e:69:36:b0:3d:bc:54:02:a8:9e:bc:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'worker-1,192.168.122.53' (ECDSA) to the list of known hosts.
Fedora CoreOS 35
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos


[core@okd4-worker-1 ~]$ sudo su -
Last login: Thu Aug  4 14:42:03 UTC 2022 on pts/0

[root@okd4-worker-1 ~]# systemctl status kubelet
[root@okd4-worker-1 ~]# journalctl -u kubelet -f
```

**_Note_:** form the systemctl status and journalctl command, you will some error logging. For example:
```
[root@okd4-worker-1 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-mco-default-env.conf, 10-mco-default-madv.conf, 20-logging.conf, 20-nodenet.conf
     Active: active (running) since Fri 2022-08-05 15:20:29 UTC; 1min 9s ago
    Process: 1799 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
    Process: 1800 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
    Process: 1801 ExecStartPre=/bin/rm -f /var/lib/kubelet/memory_manager_state (code=exited, status=0/SUCCESS)
   Main PID: 1802 (kubelet)
      Tasks: 15 (limit: 9419)
     Memory: 28.0M
        CPU: 1.541s
     CGroup: /system.slice/kubelet.service
             └─ 1802 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.k>

Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.783959    1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.884946    1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: W0805 15:21:38.945933    1802 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage>
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.946190    1802 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "cs>
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.985410    1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
```

##### 4.3 Approve CSR
Check CSR status
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME                                             AGE     SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-5pjzn                                        48m     kubernetes.io/kubelet-serving                 system:node:okd4-master.okd4.okd.homelab.com                                      <none>              Approved,Issued
csr-bqdts                                        4m58s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending
csr-fs7z4                                        48m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-kmw8k                                        48m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-wvx5m                                        5m14s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending
system:openshift:openshift-authenticator-2pxvw   37m     kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
system:openshift:openshift-monitoring-764wv      37m     kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-monitoring:cluster-monitoring-operator            <none>              Approved,Issued
```
Approve the CSR in pending status
```
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-bqdts
certificatesigningrequest.certificates.k8s.io/csr-bqdts approved
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-wvx5m
certificatesigningrequest.certificates.k8s.io/csr-wvx5m approved
```
Check cluster node status
```
[root@ssd-okd4-services okd4-install]# oc get node
NAME                                 STATUS     ROLES    AGE   VERSION
okd4-master.okd4.okd.homelab.com     Ready      master   50m   v1.23.5+3afdacb
okd4-worker-1.okd4.okd.homelab.com   NotReady   worker   79s   v1.23.5+3afdacb
```
Check CSR status again
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME                                             AGE     SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-5pjzn                                        51m     kubernetes.io/kubelet-serving                 system:node:okd4-master.okd4.okd.homelab.com                                      <none>              Approved,Issued
csr-bqdts                                        7m52s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-fs7z4                                        51m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-kmw8k                                        51m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-smlvw                                        90s     kubernetes.io/kubelet-serving                 system:node:okd4-worker-1.okd4.okd.homelab.com                                    <none>              Pending
csr-wvx5m                                        8m8s    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
system:openshift:openshift-authenticator-2pxvw   40m     kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-authentication-operator:authentication-operator   <none>              Approved,Issued
system:openshift:openshift-monitoring-764wv      40m     kubernetes.io/kube-apiserver-client           system:serviceaccount:openshift-monitoring:cluster-monitoring-operator            <none>              Approved,Issued
```
Approve the CSR for the worker node.
```
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-smlvw
certificatesigningrequest.certificates.k8s.io/csr-smlvw approved
```
Check node status, it would take several minutes to get worker node to ready status
```
[root@ssd-okd4-services okd4-install]# oc get node
NAME                                 STATUS   ROLES    AGE     VERSION
okd4-master.okd4.okd.homelab.com     Ready    master   56m     v1.23.5+3afdacb
okd4-worker-1.okd4.okd.homelab.com   Ready    worker   6m40s   v1.23.5+3afdacb
```

##### 4.3 Check PODs status on worker node.
It would take several minutes to run PODs on worker node.
In the end, it should have PODs in running status on worker node. For example:
```
[root@ssd-okd4-services okd4-install]# oc get pods -A --field-selector spec.nodeName=okd4-worker-1.okd4.okd.homelab.com
NAMESPACE                                NAME                                       READY   STATUS      RESTARTS       AGE
openshift-cluster-node-tuning-operator   tuned-nxwdd                                1/1     Running     0              12m
openshift-dns                            dns-default-sfd27                          2/2     Running     0              6m5s
openshift-dns                            node-resolver-wswnm                        1/1     Running     0              12m
openshift-image-registry                 node-ca-pgvj5                              1/1     Running     0              12m
openshift-ingress-canary                 ingress-canary-cj2sn                       1/1     Running     0              6m5s
openshift-ingress                        router-default-8ff64bc5b-z4fnl             1/1     Running     0              49m
openshift-machine-config-operator        machine-config-daemon-5nt46                2/2     Running     0              12m
openshift-monitoring                     alertmanager-main-0                        6/6     Running     0              2m47s
openshift-monitoring                     grafana-65b68c96d-dz2ll                    3/3     Running     0              5m15s
openshift-monitoring                     kube-state-metrics-849c6bb5d5-b5x7x        3/3     Running     0              47m
openshift-monitoring                     node-exporter-wfptq                        2/2     Running     0              12m
openshift-monitoring                     openshift-state-metrics-7896d98645-4p9ms   3/3     Running     0              47m
openshift-monitoring                     prometheus-adapter-595764bbc-7vfbh         1/1     Running     2 (4m5s ago)   42m
openshift-monitoring                     prometheus-k8s-0                           6/6     Running     0              5m11s
openshift-monitoring                     thanos-querier-c6bcb64b9-kq6g8             6/6     Running     0              5m13s
openshift-multus                         multus-additional-cni-plugins-w4lr8        1/1     Running     0              12m
openshift-multus                         multus-z7t87                               1/1     Running     0              12m
openshift-multus                         network-metrics-daemon-8n2g2               2/2     Running     0              12m
openshift-network-diagnostics            network-check-source-6fdcb969d6-6nf7d      1/1     Running     0              57m
openshift-network-diagnostics            network-check-target-dlnkq                 1/1     Running     0              12m
openshift-operator-lifecycle-manager     collect-profiles-27661890-jh28h            0/1     Completed   0              8m59s
openshift-ovn-kubernetes                 ovnkube-node-8sj29                         5/5     Running     0              12m
```

#### 5. Check Cluster Status

##### 5.1 Update Haproxy to remove bootstrap node.
```
[root@ssd-okd4-services okd4-install]# sed -i '/ okd4-bootstrap /s/^./#/' /etc/haproxy/haproxy.cfg
[root@ssd-okd4-services okd4-install]# cat /etc/haproxy/haproxy.cfg | grep bootstrap
#    server okd4-bootstrap 192.168.122.51:6443 check
#    server okd4-bootstrap 192.168.122.51:22623 check
[root@ssd-okd4-services okd4-install]# systemctl reload haproxy
```

*__Note__*: to remove #
```sed -i '/ okd4-bootstrap /s/^./ /' /etc/haproxy/haproxy.cfg```

##### 5.2 Power off the bootstrap VM.

##### 5.3 Access web console.
Get admin password
```
[root@ssd-okd4-services okd4-install]# cat install_dir/auth/kubeadmin-password 
FxZcu-qHP3E-KsT4T-YZzC2[root@ssd-okd4-services okd4-install]# 
```
Ensure the KVM Server has IP forwarding enabled
```
$ cat /proc/sys/net/ipv4/ip_forward
1
```
Check iptables on KVM Server, remove rules which prevent traffic to "default" network (virbr0). For example:
```
$ iptables -v --line-numbers -L

Chain LIBVIRT_FWI (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1      49M   71G ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
2        1    60 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1      26M 2676M ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
2        0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
```
Delete rule via the line number.
```
$ iptables -D LIBVIRT_FWI 2
$ iptables -D LIBVIRT_FWO 2
```
Add hosts to hosts file on the cleint node. For example:
```
192.168.122.50 console-openshift-console.apps.okd4.okd.homelab.com
192.168.122.50 oauth-openshift.apps.okd4.okd.homelab.com
```
Add static route for 192.168.122.0 via IP of KVM server on the client.
```
$ ip route add 192.168.122.0/24 via 192.168.1.15 dev eno1
$ ip route show
default via 192.168.1.1 dev eno1 proto dhcp src 192.168.1.27 metric 100 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.27 metric 100 
192.168.122.0/24 via 192.168.1.15 dev eno1
```
Ensure ping success via the domain name
```
$ ping console-openshift-console.apps.okd4.okd.homelab.com -c 3
PING console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50) 56(84) bytes of data.
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=1 ttl=63 time=0.908 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=2 ttl=63 time=0.726 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=3 ttl=63 time=0.598 ms

$ ping oauth-openshift.apps.okd4.okd.homelab.com -c 3
PING oauth-openshift.apps.okd4.okd.homelab.com (192.168.122.50) 56(84) bytes of data.
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=1 ttl=63 time=0.616 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=2 ttl=63 time=0.403 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=3 ttl=63 time=0.449 ms
```
Open web browser to access **console-openshift-console.apps.okd4.okd.homelab.com**
![](https://i.imgur.com/xymCZZX.png)

Redirected to oauth url for login.
![](https://i.imgur.com/OG7frdZ.png)

Enter username **kubeadmin** and password from previous step.
![](https://i.imgur.com/YigWxkR.png)

OKD4 Web console login successfully.
![](https://i.imgur.com/GdaGk1p.png)


##### 5.4 Check Haproxy stats
Check Haproxy stats enabled in `/etc/haproxy/haproxy.cfg`.
```
listen stats
    bind :9000
    mode http
    stats enable
    stats uri /
```
Allow TCP port 9000
```
[root@ssd-okd4-services ~]# sudo firewall-cmd --permanent --add-port=9000/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success
```
Open Haproxy stats page
![](https://i.imgur.com/88iPc3b.png)

##### 5.5 Check routes
```
[root@ssd-okd4-services okd4-install]# oc get route -A
NAMESPACE                  NAME                HOST/PORT                                                          PATH   SERVICES            PORT    TERMINATION            WILDCARD
openshift-authentication   oauth-openshift     oauth-openshift.apps.okd4.okd.homelab.com                                 oauth-openshift     6443    passthrough/Redirect   None
openshift-console          console             console-openshift-console.apps.okd4.okd.homelab.com                       console             https   reencrypt/Redirect     None
openshift-console          downloads           downloads-openshift-console.apps.okd4.okd.homelab.com                     downloads           http    edge/Redirect          None
openshift-ingress-canary   canary              canary-openshift-ingress-canary.apps.okd4.okd.homelab.com                 ingress-canary      8080    edge/Redirect          None
openshift-monitoring       alertmanager-main   alertmanager-main-openshift-monitoring.apps.okd4.okd.homelab.com   /api   alertmanager-main   web     reencrypt/Redirect     None
openshift-monitoring       grafana             grafana-openshift-monitoring.apps.okd4.okd.homelab.com                    grafana             https   reencrypt/Redirect     None
openshift-monitoring       prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.okd4.okd.homelab.com             prometheus-k8s      web     reencrypt/Redirect     None
openshift-monitoring       thanos-querier      thanos-querier-openshift-monitoring.apps.okd4.okd.homelab.com      /api   thanos-querier      web     reencrypt/Redirect     None

```

#### Troubleshoot & Tips:
- To delete all non-running containers in bootstrap node.
    `podman container prune`

- no space left in /var/run
https://github.com/openshift/okd/issues/649
    ```
    $ rm -rf /run/mco-machine-os-content
    $ ln -s /tmp /run/mco-machine-os-content
    ```

- /etc/resolv.conf has name server 192.168.122.1 (Gateway IP) instead of 192.168.122.50
In the Fedora CoreOS 36, `/etc/resolv.conf` is a symbolic link to `/var/run/systemd/resolve/resolv.conf`. This is managemed by systemd-resolved service. Ensure you specified `ip` and `nameserver` options in kernel boot options.

- `oc get clusteroperators` command shows **machine-config** clusteroperators has error.
    ```
    machine-config                                                              True        True          True       40h     Unable to apply 4.10.0-0.okd-2022-07-09-073606: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1)
    ```
    ```
    [root@ssd-okd4-services okd4-install]# oc get mcp
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master                                                      False     True       True       1              0                   0                     1                      40h
    worker   rendered-worker-c76b9c8896ffcb19ef85c1c5d8cacd39   True      False      False      1              1                   1                     0                      40h
    ```
    Still unsure root cause. related posts:
    https://githubhot.com/repo/openshift/okd/issues/1289
    https://stackoverflow.com/questions/73101073/newly-install-okd4-cluster-getting-machine-config-errors

- script to recreate ignition configs. 
    ```
    #!/bin/bash

    rm -rf ./install_dir
    mkdir ./install_dir

    cp install-config.yaml ./install_dir

    ./openshift-install create manifests --dir=install_dir
    ./openshift-install create ignition-configs --dir=install_dir

    cp -afpR ./install_dir/* /var/www/html/okd4/

    chown -R apache: /var/www/html/okd4
    chmod -R 755 /var/www/html/okd4
    chcon -R -t httpd_sys_content_t /var/www/html/

    mkdir -p ~/.kube/
    cp -afpR ./install_dir/auth/kubeconfig ~/.kube/config

    echo > ~/.ssh/known_hosts

    ls -l /var/www/html/okd4/

    # reverse haporxy config if changed by sed -i '/ okd4-bootstrap /s/^/#/' /etc/haproxy/haproxy.cfg
    sed -i '/ okd4-bootstrap /s/^.//' /etc/haproxy/haproxy.cfg
    ```

#### Todo: 
1. use DHCP with DNS server to the ssd-okd4-services. 
https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/
https://unix.stackexchange.com/questions/346967/obtain-ip-from-dhcp-sever-but-set-dns-servers-statically-on-debian
**/etc/NetworkManager/system-connections**

2. auto enter kernel options.

#### Reference:

Installation:
https://www.okd.io/installation/
https://medium.com/swlh/guide-okd-4-5-single-node-cluster-832693cb752b
https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee
https://upstreamwithoutapaddle.com/home-lab/prepare-kvm-okd-install/
https://craig-robinson.medium.com/openshift-4-4-okd-bare-metal-install-on-vmware-home-lab-6841ce2d37eb
https://blog.csdn.net/networken/article/details/123953256
https://blog.maumene.org/2020/11/18/OKD-or-OpenShit-in-one-box.html
http://www.gbca.cn/node/132

Other:
https://developers.redhat.com/blog/2020/03/12/how-to-customize-fedora-coreos-for-dedicated-workloads-with-ostree#
https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/
https://github.com/coreos/fedora-coreos-docs/issues/89
https://github.com/openshift/okd/blob/master/FAQ.md#which-fedora-coreos-should-i-use

https://computingforgeeks.com/how-to-install-okd-openshift-cluster-on-openstack/
http://www.yylin.io/2019/08/12/openshift-ansible-v3-11/#%E5%BE%8C%E7%BA%8C%E8%A8%AD%E5%AE%9A