<style>.ui-infobar, #doc.markdown-body { max-width: 1200px; }</style>
# Deploy Tiny OKD4 VM Cluster with Harbor registry (WIP)
This installation includes a "Harbor" registry setup to speed up cluster deployment.
###### tags: `okd`
> 2022-08-06: Draft Version
> 2022-09-18: Add Harbor registry steps
> 2022-09-19: Refine step "Setup OKD4 Services Node"
#### Overview
A tiny OKD4 (4.10) VM cluster (1 bastion + 1 bootstrap + 1 master + 1 worker) on a KVM Server (physical).
The bootstrap could be deleted after cluster deploy completed.
The KVM server has 64GB RAM, 4 CPU Cores (8 Threads), running Ubuntu 20.04LTS.
There is client host for access web console.
```
[Client] | {Physical Network} | [KVM Server] | {Virtual Network} | [OKD4 VMs]
.15 | 192.168.1.0/24 | .27 .1 | 192.168.122.0/24 | .50 (services)
| | | | .51 (master)
| | | | .52 (worker-1)
| | | | .53 (worker-2)
```
Base domain: `okd.homelab.com`
Cluster name: `okd4`
Tool and Image Version:
- openshift-install version **4.10.0-0.okd-2022-07-09-073606**
- openshift-client (oc, kubectl) version **4.10.0-0.okd-2022-07-09-073606**
- Fedora CoreOS metal raw image version **36.20220703.3.1**
- Fedora CoreOS ISO image version **fedora-coreos-36.20220703.3.1-live.x86_64.iso**
#### VM Specification
| VM Name | OS | vCPU | RAM | Storage | IP Address |
| ------------------ | -----------------| ---- | ---- | ------- | -------------- |
| ssd-okd4-services | CentOS 7.9 | 2 | 4GB | 32GB | 192.168.122.50 |
| ssd-okd4-bootstrap | Fedora CoreOS 36 | 2 | 8GB | 32GB | 192.168.122.51 |
| ssd-okd4-master | Fedora CoreOS 36 | 8 | 16GB | 64GB | 192.168.122.52 |
| ssd-okd4-worker-1 | Fedora CoreOS 36 | 2 | 8GB | 64GB | 192.168.122.53 |
*__Note__:* suggest minimium disk size: bootstrap(16G), master(32G), worker(32G)
Disk usage after deployment.
```
[root@ssd-okd4-services ~]# ssh bootstrap -l core df -h | grep vda
/dev/vda4 32G 6.5G 25G 21% /sysroot
/dev/vda3 350M 196M 132M 60% /boot
[root@ssd-okd4-services ~]# ssh master -l core df -h | grep vda
/dev/vda4 64G 15G 50G 23% /sysroot
/dev/vda3 350M 196M 132M 60% /boot
[root@ssd-okd4-services ~]# ssh worker-1 -l core df -h | grep vda
/dev/vda4 32G 12G 20G 37% /sysroot
/dev/vda3 350M 196M 132M 60% /boot
```
#### Environment Description
- Cluster connects libvirt "default" virtual network 192.168.122.0/24 (NAT and DHCP enabled), Gateway 192.168.122.1.
- If "virtIO" is used for storage bus driver, disk device name will be /dev/vda.
- Use "virsh net-update" command to bind IP address with MAC to the VMs, for example.
```
virsh net-update default add ip-dhcp-host \
"<host mac='52:54:00:a6:1b:66' name='ssd-okd4-services' ip='192.168.122.50'/>" \
--live --config
```
- The ssd-okd4-services VM is known as bastion node.
- For bootstrap, master and worker VMs, IP address are configured directly instead of using DHCP. If IP configured via DHCP, Network Manager on Fedore CoreOS adds gateway IP to first nameserver (to be confirmed).
- For bootstrap, master and worker VMs, make boot order to "VirtIO Disk" first and "CDROM" second. (mount Fedora CoreOS ISO image to CDROM).
- The bootstrap node VM must have minimum 8GB RAM in order to complete installation. Otherwise container stuck with error that run out of space in "/run".
- The master node VM must have minimum 16GB RAM and 8 vCPU (maybe 4vCPU works, but not tested yet). Otherwise some containers might stuck in pending status due to lack of resources.
- The services node hosts Harbor registry on port 9443. Using the local registry to speed up OKD4 deploy.
#### 1. Setup OKD4 Services Node
##### 1.1 Install DNS (named) server
Install DNS packages
```
[root@ssd-okd4-services ~]# yum install -y vim bind bind-utils
```
Configure named configuration: **/etc/named.conf**
```
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
// See the BIND Administrator's Reference Manual (ARM) for details about the
// configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html
options {
listen-on port 53 { 127.0.0.1; 192.168.122.50; };
#listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
recursing-file "/var/named/data/named.recursing";
secroots-file "/var/named/data/named.secroots";
allow-query { localhost; 192.168.122.0/24; };
/*
- If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
- If you are building a RECURSIVE (caching) DNS server, you need to enable
recursion.
- If your recursive DNS server has a public IP address, you MUST enable access
control to limit queries to your legitimate users. Failing to do so will
cause your server to become part of large scale DNS amplification
attacks. Implementing BCP38 within your network would greatly
reduce such attack surface
*/
recursion yes;
forwarders {
8.8.8.8;
8.8.4.4;
};
dnssec-enable yes;
dnssec-validation yes;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.root.key";
managed-keys-directory "/var/named/dynamic";
pid-file "/run/named/named.pid";
session-keyfile "/run/named/session.key";
};
logging {
channel default_debug {
file "data/named.run"; // fullpath: /var/named/data/named.run
severity dynamic;
};
};
zone "." IN {
type hint;
file "named.ca";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
// zone files
zone "okd.homelab.com" IN {
type master;
file "forward.okd.homelab.com";
};
zone "122.168.192.in-addr.arpa" IN {
type master;
file "reverse.okd.homelab.com";
};
```
Create forward zone file: **/var/named/forward.okd.homelab.com**
```
$TTL 86400 ; 24 hours could have been written as 24h or 1D
@ IN SOA okd4-services.okd.homelab.com. (
admin.okd.homelab.com. ; responsible party
2022072001 ; serial
3H ; refresh
15 ; retry
1w ; expire
3h ; minimum
)
; name servers - NS records
@ IN NS okd4-services.okd.homelab.com.
; name server - A records
okd4-services.okd.homelab.com. IN A 192.168.122.50
; image registry (Harbor) - A records
registry.okd4.okd.homelab.com. IN A 192.168.122.50
; okd4 container platform cluster - A records
okd4-bootstrap.okd4.okd.homelab.com. IN A 192.168.122.51
okd4-master.okd4.okd.homelab.com. IN A 192.168.122.52
okd4-worker-1.okd4.okd.homelab.com. IN A 192.168.122.53
; okd4 internal cluster IPs - A records
api.okd4.okd.homelab.com. IN A 192.168.122.50
api-int.okd4.okd.homelab.com. IN A 192.168.122.50
*.apps.okd4.okd.homelab.com. IN A 192.168.122.50
etcd-0.okd4.okd.homelab.com. IN A 192.168.122.52
console-openshift-console.apps.okd4.okd.homelab.com. IN A 192.168.122.50
oauth-openshift.apps.okd4.okd.homelab.com. IN A 192.168.122.50
; okd4 internal cluster IPs - SRV records
_etcd-server-ssl._tcp.okd4.okd.homelab.com. 86400 IN SRV 0 10 2380 etcd-0.okd4
```
Configure reverse zone file: **/var/named/reverse.okd.homelab.com**
```
$TTL 604800
@ IN SOA okd4-services.okd.homelab.com. (
admin.okd.homelab.com. ; responsible party
2022072001 ; serial
3H ; refresh
15 ; retry
1w ; expire
3h ; minimum
)
; name servers - NS records
@ IN NS okd4-services.okd.homelab.com.
; name servers - PTR records
50 IN PTR okd4-services.okd.homelab.com.
; OpenShift Container Platform Cluster - PTR records
51 IN PTR okd4-bootstrap.okd4.okd.homelab.com.
52 IN PTR okd4-master.okd4.okd.homelab.com.
53 IN PTR okd4-worker-1.okd4.okd.homelab.com.
50 IN PTR api.okd4.okd.homelab.com.
50 IN PTR api-int.okd4.okd.homelab.com.
50 IN PTR registry.okd4.okd.homelab.com.
```
Validate configure and start named service
```
[root@ssd-okd4-services ~]# named-checkconf /etc/named.conf
[root@ssd-okd4-services named]# named-checkzone okd.homelab.com /var/named/forward.okd.homelab.com
zone okd.homelab.com/IN: loaded serial 2022072001
OK
[root@ssd-okd4-services named]# named-checkzone 122.168.192.in-addr.arpa /var/named/reverse.okd.homelab.com
zone 122.168.192.in-addr.arpa/IN: loaded serial 2022072001
OK
[root@ssd-okd4-services ~]# systemctl enable named
[root@ssd-okd4-services ~]# systemctl start named
[root@ssd-okd4-services ~]# systemctl status named
[root@ssd-okd4-services ~]# sudo firewall-cmd --permanent --add-port=53/udp
success
[root@ssd-okd4-services ~]# sudo firewall-cmd --reload
success
```
Disable Network Manager to update **/etc/resolv.conf**
Adding `dns=none` and `rc-manager=unmanaged` to `main` section in **/etc/NetworkManager/NetworkManager.conf**
```
[root@ssd-okd4-services ~]# cat /etc/NetworkManager/NetworkManager.conf | grep -A 4 "\[main\]"
[main]
#plugins=ifcfg-rh,ibft
dns=none
rc-manager=unmanaged
[root@ssd-okd4-services ~]# sudo systemctl restart NetworkManager
```
Edit **resolv.conf** to use localhost DNS server.
```[root@ssd-okd4-services ~]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1
```
Test DNS domain resolution.
```
[root@ssd-okd4-services ~]# dig +short okd4-services.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short okd4-bootstrap.okd4.okd.homelab.com
192.168.122.51
[root@ssd-okd4-services ~]# dig +short okd4-master.okd4.okd.homelab.com
192.168.122.52
[root@ssd-okd4-services ~]# dig +short okd4-worker-1.okd4.okd.homelab.com
192.168.122.53
[root@ssd-okd4-services ~]# dig +short api.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short api-int.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short *.apps.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short etcd-0.okd4.okd.homelab.com
192.168.122.52
[root@ssd-okd4-services ~]# dig +short console-openshift-consoe.apps.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short oauth-openshift.apps.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig +short registry.okd4.okd.homelab.com
192.168.122.50
[root@ssd-okd4-services ~]# dig -x 192.168.122.50 +short
api.okd4.okd.homelab.com.
api-int.okd4.okd.homelab.com.
registry.okd4.okd.homelab.com.
okd4-services.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.51 +short
okd4-bootstrap.okd4.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.52 +short
okd4-master.okd4.okd.homelab.com.
[root@ssd-okd4-services ~]# dig -x 192.168.122.53 +short
okd4-worker-1.okd4.okd.homelab.com.
```
**_NOTE:_** Or use other dig options "dig +noall +answer" and specify DNS IP, for example:
```
[root@ssd-okd4-services ~]# dig +noall +answer @192.168.122.50 okd4-services.okd.homelab.com
okd4-services.okd.homelab.com. 86400 IN A 192.168.122.50
[root@ssd-okd4-services ~]# dig +noall +answer @192.168.122.50 -x 192.168.122.51
51.122.168.192.in-addr.arpa. 604800 IN PTR okd4-bootstrap.okd4.okd.homelab.com.
```
##### 1.2 Install Haproxy Server
Install haproxy package
```
[root@ssd-okd4-services ~]# yum install haproxy -y
```
Configure haproxy **/etc/haproxy/haproxy.cfg**
```
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats
bind :9000
mode http
stats enable
stats uri /
# API
frontend okd4_api_frontend
bind :6443
default_backend okd4_api_backend
mode tcp
option tcplog
backend okd4_api_backend
balance source
mode tcp
server okd4-bootstrap 192.168.122.51:6443 check
server okd4-master 192.168.122.52:6443 check
# Machine Config Server
frontend okd4_machine_config_server_frontend
bind :22623
default_backend okd4_machine_config_server_backend
mode tcp
option tcplog
backend okd4_machine_config_server_backend
balance source
mode tcp
server okd4-bootstrap 192.168.122.51:22623 check
server okd4-master 192.168.122.52:22623 check
# HTTP Ingress
frontend okd4_http_ingress_frontend
bind :80
default_backend okd4_http_ingress_backend
mode tcp
option tcplog
backend okd4_http_ingress_backend
balance source
mode tcp
server okd4-worker-1 192.168.122.53:80 check
# HTTPS Ingress
frontend okd4_http_ingress_frontend
bind *:443
default_backend okd4_https_ingress_backend
mode tcp
option tcplog
backend okd4_https_ingress_backend
balance source
mode tcp
server okd4-worker-1 192.168.122.53:443 check
```
Start HAProxy server
```
[root@ssd-okd4-services ~]# setsebool -P haproxy_connect_any 1
[root@ssd-okd4-services ~]# systemctl enable haproxy
Created symlink from /etc/systemd/system/multi-user.target.wants/haproxy.service to /usr/lib/systemd/system/haproxy.service.
[root@ssd-okd4-services ~]# systemctl start haproxy
[root@ssd-okd4-services ~]# systemctl status haproxy
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=6443/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=22623/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-service=http
success
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-service=https
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success
```
##### 1.3 Install Apache Server
Install apache package
```
[root@ssd-okd4-services ~]# yum install httpd -y
```
Set listening port to 8080 in **/etc/httpd/conf/httpd.conf**
```
#Listen 12.34.56.78:80
Listen 8080
```
Start server
```
[root@ssd-okd4-services ~]# setsebool -P httpd_read_user_content 1
[root@ssd-okd4-services ~]# systemctl enable httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
[root@ssd-okd4-services ~]# systemctl start httpd
[root@ssd-okd4-services ~]# firewall-cmd --permanent --add-port=8080/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success
[root@ssd-okd4-services ~]# curl -s http://192.168.122.50:8080 | tail
the sources of Red Hat Enterprise Linux (RHEL).<p>
<p>Additionally to being a popular choice for web hosting, CentOS also provides a rich platform for open source communities to build upon. For more information
please visit the <a href="http://www.centos.org/">CentOS website</a>.</p>
</div>
</div>
</div>
</div>
</div>
</body></html>
```
##### 1.4 Install openshift-installer and oc client
Download openshift oc client and openshift-install form **https://github.com/openshift/okd/releases**
```
[root@ssd-okd4-services ~]# mkdir openshift-install
[root@ssd-okd4-services ~]# cd openshift-install
[root@ssd-okd4-services okd4-install]# wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
[root@ssd-okd4-services okd4-install]# wget https://github.com/openshift/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
[root@ssd-okd4-services okd4-install]# tar -xvf ./openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
README.md
oc
kubectl
[root@ssd-okd4-services okd4-install]# tar -xvf ./openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
README.md
openshift-install
[root@ssd-okd4-services okd4-install]# ls -l
total 1020792
-rwxr-xr-x. 1 root root 122829960 Jun 22 04:12 kubectl
-rwxr-xr-x. 1 root root 122829960 Jun 22 04:12 oc
-rw-rw-r--. 1 root root 49524984 Jul 11 20:42 openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
-rwxr-xr-x. 1 root root 619614208 Jun 22 01:44 openshift-install
-rw-rw-r--. 1 root root 130480916 Jul 11 20:36 openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
-rw-r--r--. 1 root root 706 Jun 22 01:44 README.md
[root@ssd-okd4-services okd4-install]# ./oc version
Client Version: 4.10.0-0.okd-2022-07-09-073606
[root@ssd-okd4-services okd4-install]# ./openshift-install version
./openshift-install 4.10.0-0.okd-2022-07-09-073606
built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
release image quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d
release architecture amd64
```
##### 1.5 add OKD nodes to **/etc/hosts**
```
[root@ssd-okd4-services ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.51 bootstrap
192.168.122.52 master
192.168.122.53 worker-1
```
##### 1.6 Install Harbor for image registry
Install latest docker (Remove docker if exist)
```
[root@ssd-okd4-services ~]# rpm -qa | grep docker
docker-client-1.13.1-209.git7d71120.el7.centos.x86_64
docker-compose-plugin-2.10.2-3.el7.x86_64
docker-common-1.13.1-209.git7d71120.el7.centos.x86_64
docker-1.13.1-209.git7d71120.el7.centos.x86_64
[root@ssd-okd4-services ~]# yum remove docker-client-1.13.1-209.git7d71120.el7.centos.x86_64 docker-common-1.13.1-209.git7d71120.el7.centos.x86_64 docker-1.13.1-209.git7d71120.el7.centos.x86_64 docker-compose-plugin-2.10.2-3.el7.x86_64 -y
```
```
[root@ssd-okd4-services ~]# curl -fsSL https://get.docker.com/ | sh
[root@ssd-okd4-services ~]# systemctl restart docker
[root@ssd-okd4-services ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@ssd-okd4-services ~]# docker -v
Docker version 20.10.18, build b40c2f6
[root@ssd-okd4-services ~]# docker version
Client: Docker Engine - Community
Version: 20.10.18
API version: 1.41
Go version: go1.18.6
Git commit: b40c2f6
Built: Thu Sep 8 23:14:08 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.18
API version: 1.41 (minimum version 1.12)
Go version: go1.18.6
Git commit: e42327a
Built: Thu Sep 8 23:12:21 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.8
GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
Install docker compose
```
[root@ssd-okd4-services ~]# curl -L https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 42.4M 100 42.4M 0 0 5929k 0 0:00:07 0:00:07 --:--:-- 6512k
[root@ssd-okd4-services ~]# chmod +x /usr/local/bin/docker-compose
[root@ssd-okd4-services ~]# docker-compose version
Docker Compose version v2.11.0
```
Download and install Harbor package
```
[root@ssd-okd4-services ~]# curl -L https://github.com/goharbor/harbor/releases/download/v2.6.0/harbor-offline-installer-v2.6.0.tgz -o harbor-offline-installer-v2.6.0.tgz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 721M 100 721M 0 0 3405k 0 0:03:36 0:03:36 --:--:-- 4024k
[root@ssd-okd4-services ~]# tar -zxf ./harbor-offline-installer-v2.6.0.tgz -C /opt/
```
##### 1.7 Create HTTPS certificate for Harbor
_*NOTE*_: Two certificates to be created; CA certificate and server certificate.
Create folder for harbor certificate
```
[root@ssd-okd4-services ~]# mkdir /opt/harbor/cert
[root@ssd-okd4-services ~]# cd /opt/harbor/cert/
```
Generate private key for CA certificate
```
[root@ssd-okd4-services cert]# openssl genrsa -out homelab.com.key 4096
Generating RSA private key, 4096 bit long modulus
.................................++
....................................++
e is 65537 (0x10001)
```
Generate CA certificate
```
[root@ssd-okd4-services cert]# openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=AU/ST=NSW/L=Sydney/O=homelab/OU=okd/CN=okd.homelab.com" -key homelab.com.key -out homelab.com.crt
[root@ssd-okd4-services cert]# ls -l
total 8
-rw-r--r--. 1 root root 2009 Sep 17 17:03 homelab.com.crt
-rw-r--r--. 1 root root 3243 Sep 17 17:03 homelab.com.key
```
Generate private key for server certificate
```
[root@ssd-okd4-services cert]# openssl genrsa -out registry.okd4.okd.homelab.com.key 4096
Generating RSA private key, 4096 bit long modulus
......................++
......................++
e is 65537 (0x10001)
```
Create CSR for server certificate
```
[root@ssd-okd4-services cert]# openssl req -sha512 -new -subj "/C=CN/ST=NSW/L=Sydney/O=homelab.com/OU=okd/CN=registery.okd4.okd.homelab.com" -key registry.okd4.okd.homelab.com.key -out registry.okd4.okd.homelab.com.csr
```
Create x509 v3 extension file, `registry.okd4.okd.homelab.com.x509-v3.conf`
```
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=registry.okd4.okd.homelab.com
DNS.2=harbor.okd4.okd.homelab.com
DNS.3=registry
DNS.4=harbor
```
```
[root@ssd-okd4-services cert]# ls -l
total 20
-rw-r--r--. 1 root root 2009 Sep 17 17:03 homelab.com.crt
-rw-r--r--. 1 root root 3243 Sep 17 17:03 homelab.com.key
-rw-r--r--. 1 root root 1720 Sep 17 17:04 registry.okd4.okd.homelab.com.csr
-rw-r--r--. 1 root root 3243 Sep 17 17:04 registry.okd4.okd.homelab.com.key
-rw-r--r--. 1 root root 310 Sep 17 17:05 registry.okd4.okd.homelab.com.x509-v3.conf
```
Generate certificate with the x509 v3 file
```
[root@ssd-okd4-services cert]# openssl x509 -req -sha512 -days 30650 -extfile registry.okd4.okd.homelab.com.x509-v3.conf -CA homelab.com.crt -CAkey homelab.com.key -CAcreateserial -in registry.okd4.okd.homelab.com.csr -out registry.okd4.okd.homelab.com.crt
Signature ok
subject=/C=CN/ST=NSW/L=Sydney/O=homelab.com/OU=okd/CN=registery.okd4.okd.homelab.com
Getting CA Private Key
[root@ssd-okd4-services cert]# ls -l
total 28
-rw-r--r--. 1 root root 2009 Sep 17 17:03 homelab.com.crt
-rw-r--r--. 1 root root 3243 Sep 17 17:03 homelab.com.key
-rw-r--r--. 1 root root 17 Sep 17 17:06 homelab.srl
-rw-r--r--. 1 root root 2163 Sep 17 17:06 registry.okd4.okd.homelab.com.crt
-rw-r--r--. 1 root root 1720 Sep 17 17:04 registry.okd4.okd.homelab.com.csr
-rw-r--r--. 1 root root 3243 Sep 17 17:04 registry.okd4.okd.homelab.com.key
-rw-r--r--. 1 root root 310 Sep 17 17:05 registry.okd4.okd.homelab.com.x509-v3.conf
```
Copy certificate to System and update ca trust.
__*Note*__: ``/etc/pki/ca-trust/source/anchors/`` for CentOS.
__*Note*__: ``/usr/local/share/ca-certificates/`` for Ubuntu and then run "update-ca-certificates".
```
[root@ssd-okd4-services cert]# cp homelab.com.crt /etc/pki/ca-trust/source/anchors/
[root@ssd-okd4-services cert]# cp registry.okd4.okd.homelab.com.crt /etc/pki/ca-trust/source/anchors/
[root@ssd-okd4-services cert]# ls -l /etc/pki/ca-trust/source/anchors/
total 8
-rw-r--r--. 1 root root 2009 Sep 18 22:49 homelab.com.crt
-rw-r--r--. 1 root root 2163 Sep 17 17:16 registry.okd4.okd.homelab.com.crt
[root@ssd-okd4-services cert]# update-ca-trust
```
Copy certificate to Harbor running directory
```
[root@ssd-okd4-services cert]# mkdir -p /data/cert
[root@ssd-okd4-services cert]# cp registry.okd4.okd.homelab.com.crt /data/cert/
[root@ssd-okd4-services cert]# cp registry.okd4.okd.homelab.com.key /data/cert/
```
Provide certificate to Docker
```
[root@ssd-okd4-services cert]# openssl x509 -inform PEM -in registry.okd4.okd.homelab.com.crt -out registry.okd4.okd.homelab.com.cert
[root@ssd-okd4-services cert]# cp registry.okd4.okd.homelab.com.cert /etc/docker/certs.d/registry.okd4.okd.homelab.com\:9443/
[root@ssd-okd4-services cert]# cp registry.okd4.okd.homelab.com.key /etc/docker/certs.d/registry.okd4.okd.homelab.com\:9443/
[root@ssd-okd4-services cert]# cp homelab.com.crt /etc/docker/certs.d/registry.okd4.okd.homelab.com\:9443/
[root@ssd-okd4-services cert]# ls -l /etc/docker/certs.d/registry.okd4.okd.homelab.com\:9443/
total 12
-rw-r--r--. 1 root root 2009 Sep 17 22:23 homelab.com.crt
-rw-r--r--. 1 root root 2163 Sep 17 22:23 registry.okd4.okd.homelab.com.cert
-rw-r--r--. 1 root root 3243 Sep 17 22:23 registry.okd4.okd.homelab.com.key
[root@ssd-okd4-services cert]# systemctl restart docker
```
Update harbor config `/opt/harbor/harbor.yml`
```
[root@ssd-okd4-services cert]# cd /opt/harbor/
[root@ssd-okd4-services harbor]# cp ./harbor.yml.tmpl harbor.yml
[root@ssd-okd4-services harbor]# vim harbor.yml
```
```
3 # The IP address or hostname to access admin UI and registry service.
4 # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
5 hostname: registry.okd4.okd.homelab.com
8 http:
9 # port for http, default is 80. If https enabled, this port will redirect to https port
10 port: 9090
12 # https related config
13 https:
14 # https port for harbor, default is 443
15 port: 9443
16 # The path of cert and key files for nginx
17 certificate: /data/cert/registry.okd4.okd.homelab.com.crt
18 private_key: /data/cert/registry.okd4.okd.homelab.com.key
```
##### 1.8 Setup and start Harbor
Start Harbor
```
[root@ssd-okd4-services harbor]# /opt/harbor/install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 20.10.18
[Step 1]: checking docker-compose is installed ...
Note: Docker Compose version v2.10.2
[Step 2]: loading Harbor images ...
output omit...
[Step 5]: starting Harbor ...
[+] Running 10/10
⠿ Network harbor_harbor Created 0.2s
⠿ Container harbor-log Started 0.6s
⠿ Container registryctl Started 1.6s
⠿ Container registry Started 2.0s
⠿ Container harbor-db Started 2.0s
⠿ Container harbor-portal Started 1.2s
⠿ Container redis Started 1.9s
⠿ Container harbor-core Started 2.3s
⠿ Container harbor-jobservice Started 3.3s
⠿ Container nginx Started 3.3s
✔ ----Harbor has been installed and started successfully.----
```
Setup Harbor service auto start. `/etc/systemd/system/harbor.service`
```
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/goharbor/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /opt/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
```
```
[root@ssd-okd4-services harbor]# systemctl enable harbor
Created symlink from /etc/systemd/system/multi-user.target.wants/harbor.service to /etc/systemd/system/harbor.service.
```
Check Harbor process status
```
[root@ssd-okd4-services harbor]# docker-compose ps
NAME COMMAND SERVICE STATUS PORTS
harbor-core "/harbor/entrypoint.…" core running (healthy)
harbor-db "/docker-entrypoint.…" postgresql running (healthy)
harbor-jobservice "/harbor/entrypoint.…" jobservice running (healthy)
harbor-log "/bin/sh -c /usr/loc…" log running (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal "nginx -g 'daemon of…" portal running (healthy)
nginx "nginx -g 'daemon of…" proxy running (healthy) 0.0.0.0:9090->8080/tcp, :::9090->8080/tcp, 0.0.0.0:9443->8443/tcp, :::9443->8443/tcp
redis "redis-server /etc/r…" redis running (healthy)
registry "/home/harbor/entryp…" registry running (healthy)
registryctl "/home/harbor/start.…" registryctl running (healthy)
```
Update /etc/hosts with harbor host (If domain is not configured in DNS server)
```
192.168.122.50 registry.okd4.okd.homelab.com
```
Verify docker login to Harbor, default username/password: admin/Harbor12345
```
[root@ssd-okd4-services harbor]# docker login registry.okd4.okd.homelab.com:9443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
```
Allow tcp port 9090, 9443
```
[root@ssd-okd4-services harbor]# firewall-cmd --permanent --add-port=9443/tcp
success
[root@ssd-okd4-services harbor]# firewall-cmd --permanent --add-port=9090/tcp
success
[root@ssd-okd4-services harbor]# firewall-cmd --reload
success
```
Check iptables on KVM Server, remove rules which prevent traffic to "default" network (virbr0)
```
$ iptables -D LIBVIRT_FWI 2
$ iptables -D LIBVIRT_FWO 2
```
Update /etc/hosts on client
```
192.168.122.50 registry.okd4.okd.homelab.com
```
On client host, add route to reach KVM if not exist
```
$ sudo ip route add 192.168.122.0/24 via 192.168.1.15 dev eno1
```
Open web browser to access Harbor from client
Login with admin/Harbor12345
Create project "openshift"
__*Note*__: You might import the CA into browser to trust the CA.
For example in Chormium, Settings -> Privacy and security -> Security -> Manage certificates -> Authorities -> Click Import to add CA (homelab.com.crt)
##### 1.9 Synchorize OKD images to Harbor registry
Get pull secret from "https://console.redhat.com/openshift/install/pull-secret"
```
[root@ssd-okd4-services ~]# mkdir -p /opt/okd-install/4.10.0
[root@ssd-okd4-services ~]# cd /opt/okd-install/4.10.0/
[root@ssd-okd4-services 4.10.0]# cat pull-secret .txt
{"auths":{"cloud.openshift.com":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfZDMxZjk2NWQwMmI2NDhhNmI0NWFlYzliNzk3MzhjZmQ6RUhEWjNZUk5EVzVPRTI1RklBMV
output omit...
```
Install jq if not available
```
$ yum install epel-release -y
$ yum update -y
$ yum install jq -y
```
Convert pull-secret.txt to json format
```
[root@ssd-okd4-services 4.10.0]# cat ./pull-secret.txt | jq . > pull-secret.json
[root@ssd-okd4-services 4.10.0]# ll
total 8
-rw-r--r--. 1 root root 2967 Sep 18 14:33 pull-secret.json
-rw-r--r--. 1 root root 2851 Sep 18 14:25 pull-secret.txt
```
Generate base64 credential for Harbor image registry
```
[root@ssd-okd4-services 4.10.0]# echo -n 'admin:Harbor12345' | base64 -w0
YWRtaW46SGFyYm9yMTIzNDU=
```
Create Harbor image registry authentication file
```
[root@ssd-okd4-services 4.10.0]# cat >pull-secret-local.json<<EOF
{
"auths":{
"registry.okd4.okd.homelab.com:9443": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU=",
"email": ""
}
}
}
EOF
```
Append Harbor image registry authentication file with the pull-secret.json
```
[root@ssd-okd4-services 4.10.0]# jq -s '.[0] * .[1]' pull-secret.json pull-secret-local.json > pull-secret-all.json
```
```
[root@ssd-okd4-services 4.10.0]# cat pull-secret-all.json | tail -n 7
},
"registry.okd4.okd.homelab.com:9443": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU=",
"email": ""
}
}
}
```
Check oc client version
```
[root@ssd-okd4-services 4.10.0]# oc version
Client Version: 4.10.0-0.okd-2022-07-09-073606
```
Export following environment variables
```
export OKD_RELEASE="4.10.0-0.okd-2022-07-09-073606"
export LOCAL_REGISTRY="registry.okd4.okd.homelab.com:9443"
export LOCAL_REPOSITORY="openshift/okd"
export PRODUCT_REPO="openshift"
export LOCAL_SECRET_JSON="/opt/okd-install/4.10.0/pull-secret-all.json"
export RELEASE_NAME="okd"
```
```
[root@ssd-okd4-services 4.10.0]# export OKD_RELEASE="4.10.0-0.okd-2022-07-09-073606"
[root@ssd-okd4-services 4.10.0]# export LOCAL_REGISTRY="registry.okd4.okd.homelab.com:9443"
[root@ssd-okd4-services 4.10.0]# export LOCAL_REPOSITORY="openshift/okd"
[root@ssd-okd4-services 4.10.0]# export PRODUCT_REPO="openshift"
[root@ssd-okd4-services 4.10.0]# export LOCAL_SECRET_JSON="/opt/okd-install/4.10.0/pull-secret-all.json"
[root@ssd-okd4-services 4.10.0]# export RELEASE_NAME="okd"
```
Pull OKD image from quay.io to local Harbor registry
_**NOTE**_: if you see ``connection reset by peer`` message, run command again.
```
oc adm release mirror -a ${LOCAL_SECRET_JSON} \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OKD_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OKD_RELEASE}
```
```
[root@ssd-okd4-services 4.10.0]# oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OKD_RELEASE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OKD_RELEASE}
info: Mirroring 170 images to registry.okd4.okd.homelab.com:9443/openshift/okd ...
omit output...
Success
Update image: registry.okd4.okd.homelab.com:9443/openshift/okd:4.10.0-0.okd-2022-07-09-073606
Mirror prefix: registry.okd4.okd.homelab.com:9443/openshift/okd
Mirror prefix: registry.okd4.okd.homelab.com:9443/openshift/okd:4.10.0-0.okd-2022-07-09-073606
To use the new mirrored repository to install, add the following section to the install-config.yaml:
imageContentSources:
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd-content
To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd-content
```
Check repositories on web console
##### 1.10 Create installation configuration file
Generate ssh key pair if you dont have one.
```
[root@ssd-okd4-services ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:4ZpNqumApn8ec7R0EUqGX7rWxiLodOSeoXQcNstQiRI root@ssd-okd4-services
The key's randomart image is:
+---[RSA 2048]----+
|E. . oo . |
|. . +o ... |
| . . ..oo |
| . = o. o |
| O +o+S |
| .+ Xo+O+ |
|.=.=o=*o. |
|o o.+* |
|...+= |
+----[SHA256]-----+
```
Get public key and Harbor CA certificate
```
[root@ssd-okd4-services ~]# cat /opt/harbor/cert/homelab.com.crt
[root@ssd-okd4-services ~]# cat /opt/okd-install/4.10.0/pull-secret-all.json | jq -c
```
Create **install-config.yaml** in the **openshift-install**
```
[root@ssd-okd4-services ~]# cd ~/openshift-install
[root@ssd-okd4-services okd4-install]# cat install-config.yaml
apiVersion: v1
baseDomain: okd.homelab.com
metadata:
name: okd4
compute:
- hyperthreading: Enabled
name: worker
replicas: 1
controlPlane:
hyperthreading: Enabled
name: master
replicas: 1
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfZDMxZjk2NWQwMmI2NDhhNmI0NWFlYzliNzk3MzhjZmQ6RUhEWjNZUk5EVzVPRTI1RklBMVc3ODlNWlgxVlo1WEVJTkY5WkIzQ1czSVRRRVdHMkNXQ0QwOUE4VlVUS0dOVg==", output omit... '
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG9D6Tj8CFJNI/FP2nZcU7n4IgjaOZJ4xH5ljoaKu/C1ppzSG8g/zgUTJmr/o+NpWz4kRYFG+k7GK6jALJp+WY24MC9nOzdn4znITRgNzSAjUUL+qq05NWpRqHVhudtIFNLG2zcp6ct4pzZlxKMtincKlI6EdpLgMYTBxflJRe7aP5xByUCQGr1uqrirLTInFwzyqzHrhUOX9l+pwvAqTFhkSrXz/t7pgE7FJ/p7Glb7npwOaCFYkOiw4u/M3yzO8qVof/rL145yJBQJCEA8KrChLr/l/QMGfD0YNZbNRTjl0Td+5L/MXHb05g+5LpPMZZg8H8S+oDDpKoCjvC1/gd root@ssd-okd4-services'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIFnzCCA4egAwIBAgIJAPzctvCfvl/DMA0GCSqGSIb3DQEBDQUAMGYxCzAJBgNV
BAYTAkFVMQwwCgYDVQQIDANOU1cxDzANBgNVBAcMBlN5ZG5leTEQMA4GA1UECgwH
output omit...
imageContentSources:
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd
- mirrors:
- registry.okd4.okd.homelab.com:9443/openshift/okd
source: quay.io/openshift/okd-content
```
* Copy the public key in /root/.ssh/id_rsa.pub for "sshKey" value.
* Copy output of "cat /opt/okd-install/4.10.0/pull-secret-all.json | jq -c" for "pullSecret" value
* Copy "/opt/harbor/cert/homelab.com.crt" for "additionalTrustBundle" value
* The OKD cluster domain will be "metadata name" + "baseDomain", which is "okd4.okd.homelab.com"
* Set "imageContentSources" to use the Harbor registry
* Set platform to "none: {}" for user-provisioned cluster.
Copy **install-config.yaml** to **install_dir** directory
```
[root@ssd-okd4-services okd4-install]# mkdir install_dir
[root@ssd-okd4-services okd4-install]# cp ./install-config.yaml ./install_dir/
```
Create manifests in **install_dir** directory
```
[root@ssd-okd4-services okd4-install]# ./openshift-install create manifests --dir=install_dir
INFO Consuming Install Config from target directory
INFO Manifests created in: install_dir/manifests and install_dir/openshift
[root@ssd-okd4-services okd4-install]# ls -l ./install_dir/
total 8
drwxr-x---. 2 root root 4096 Jul 24 22:15 manifests
drwxr-x---. 2 root root 4096 Jul 24 22:15 openshift
```
Create ignition configs in **install_dir** directory (manifests will be cleared)
```
[root@ssd-okd4-services okd4-install]# ./openshift-install create ignition-configs --dir=install_dir
INFO Consuming Common Manifests from target directory
INFO Consuming Master Machines from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Openshift Manifests from target directory
INFO Consuming Worker Machines from target directory
INFO Ignition-Configs created in: install_dir and install_dir/auth
[root@ssd-okd4-services okd4-install]# ls -l ./install_dir/
total 276
drwxr-x---. 2 root root 50 Jul 24 22:16 auth
-rw-r-----. 1 root root 269438 Jul 24 22:16 bootstrap.ign
-rw-r-----. 1 root root 1722 Jul 24 22:16 master.ign
-rw-r-----. 1 root root 96 Jul 24 22:16 metadata.json
-rw-r-----. 1 root root 1722 Jul 24 22:16 worker.ign
```
Host `ignition` configs and `Fedore CoreOS bare-metal image` and `sig` files on Apache server.
```
[root@ssd-okd4-services okd4-install]# mkdir /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# cp -afpR install_dir/* /var/www/html/okd4/
# update file owner, permission and security context
[root@ssd-okd4-services okd4-install]# chown -R apache: /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chmod -R 755 /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chcon -R -t httpd_sys_content_t /var/www/html/
# test access toignition files
[root@ssd-okd4-services okd4-install]# curl localhost:8080/okd4/
```
```
[root@ssd-okd4-services okd4-install]# wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/36.20220703.3.1/x86_64/fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz
[root@ssd-okd4-services okd4-install]# wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/36.20220703.3.1/x86_64/fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz.sig
[root@ssd-okd4-services okd4-install]# mv ./fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz fcos36.raw.xz
[root@ssd-okd4-services okd4-install]# mv ./fedora-coreos-36.20220703.3.1-metal.x86_64.raw.xz.sig fcos36.raw.xz.sig
[root@ssd-okd4-services okd4-install]# mv ./fcos36.raw.xz ./fcos36.raw.xz.sig /var/www/html/okd4/
# update file owner, permission and security context
[root@ssd-okd4-services okd4-install]# chown -R apache: /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chmod -R 755 /var/www/html/okd4
[root@ssd-okd4-services okd4-install]# chcon -R -t httpd_sys_content_t /var/www/html/
```
#### 2. Deploy OKD4 Bootstrap Node
##### 2.1 Power on Bootstrap VM and press "tab" key to edit kernel boot options .
##### 2.2 Append extra kernel boot options.
```
ip=192.168.122.51::192.168.122.1:255.255.255.0:::none
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/bootstrap.ign
```
**_NOTE:_** Take snapshot after entered kernel boot options (do not press enter). This is for case if install fail and resume to entering the boot options.
Press enter to install, you should see image and signature downloading.
After system installation completed, it will be rebooted and boot from disk.
Wait the bootstrap node runs an minimal Kubernetes for deploy master node.
##### 2.3 SSH to bootstrap node
```
[root@ssd-okd4-services ~]# ssh bootstrap -l core
The authenticity of host 'bootstrap (192.168.122.51)' can't be established.
ECDSA key fingerprint is SHA256:Blrhzptpx3fjA6lj97r78eZPcZkBAMuB4CKY4Wm8b2o.
ECDSA key fingerprint is MD5:ac:1a:5f:f1:e9:10:28:7e:60:c3:43:12:ae:8a:63:97.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bootstrap,192.168.122.51' (ECDSA) to the list of known hosts.
This is the bootstrap node; it will be destroyed when the master is fully up.
The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.
journalctl -b -f -u release-image.service -u bootkube.service
Fedora CoreOS 36.20220703.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[core@ssd-okd4-bootstrap ~]$
```
**_NOTE:_** To check release-image and bootkube services on the bootstrap node, excute `journalctl -b -f -u release-image.service -u bootkube.service`
**_NOTE:_** Monitor bootstrap status with `openshift-install` tool.
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
INFO Waiting up to 20m0s (until 12:15AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443...
```
**_NOTE:_** The bootstrap node will be rebooted again while bootstrapping.
**_NOTE:_** When the boostrap node successfully running an minimum kubernetes, you should see "waiting for bootstrapping to complete" message.
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
INFO Waiting up to 20m0s (until 12:42AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443...
INFO API v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty up
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Networking...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
INFO Waiting up to 30m0s (until 12:52AM) for bootstrapping to complete...
```
**_NOTE:_** In bootstrap node, check the minimum kuberntes PODs are running.
```
[root@okd4-bootstrap ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
e4c0d1c3d80a4 4 minutes ago Ready bootstrap-kube-scheduler-okd4-bootstrap.okd4.okd.homelab.com kube-system 0 (default)
23c60759fae12 4 minutes ago Ready bootstrap-kube-apiserver-okd4-bootstrap.okd4.okd.homelab.com openshift-kube-apiserver 0 (default)
788ebe7e5cd0b 4 minutes ago Ready bootstrap-kube-controller-manager-okd4-bootstrap.okd4.okd.homelab.com kube-system 0 (default)
e984eb128192f 4 minutes ago Ready cloud-credential-operator-okd4-bootstrap.okd4.okd.homelab.com openshift-cloud-credential-operator 0 (default)
b3ad5da52a9ba 4 minutes ago Ready bootstrap-cluster-version-operator-okd4-bootstrap.okd4.okd.homelab.com openshift-cluster-version 0 (default)
59a6e3da5a49e 5 minutes ago Ready etcd-bootstrap-member-okd4-bootstrap.okd4.okd.homelab.com openshift-etcd 0 (default)
589cb66783f63 5 minutes ago Ready bootstrap-machine-config-operator-okd4-bootstrap.okd4.okd.homelab.com default 0 (default)
```
#### 3. Deploy OKD4 Master Node
##### 3.1 Power on Master VM and press "tab" key to edit kernel boot options
##### 3.2 Append extra options below and press enter to start installation
```
ip=192.168.122.52::192.168.122.1:255.255.255.0:::none
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/master.ign
```
**_NOTE:_** Check master node kubelet service.
```
[root@ssd-okd4-services ~]# ssh master -l core
The authenticity of host 'master (192.168.122.52)' can't be established.
ECDSA key fingerprint is SHA256:YBKgKgqmNgCY5noB2LhZx8fExr974Uejz5OyDhApDZg.
ECDSA key fingerprint is MD5:09:1f:3b:6b:0c:80:c0:a2:92:f6:35:38:91:64:4a:c2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.122.52' (ECDSA) to the list of known hosts.
Fedora CoreOS 35
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[core@okd4-master ~]$
[core@okd4-master ~]$ sudo su -
[root@okd4-master ~]# journalctl -u kubelet -f
```
**_NOTE:_** Check bootstrapping completed if master node deployed successfully. You should see `DEBUG Bootstrap status: complete`..
```
[root@ssd-okd4-services okd4-install]# ./openshift-install --dir=install_dir/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.10.0-0.okd-2022-07-09-073606
DEBUG Built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
INFO Waiting up to 20m0s (until 1:20AM) for the Kubernetes API at https://api.okd4.okd.homelab.com:6443...
INFO API v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty up
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Networking...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
INFO Waiting up to 30m0s (until 1:30AM) for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
INFO Time elapsed: 0s
```
##### 3.3 Set the oc command on service node.
```
[root@ssd-okd4-services okd4-install]# cp ./oc /usr/bin/
[root@ssd-okd4-services okd4-install]# which oc
/usr/bin/oc
[root@ssd-okd4-services okd4-install]# mkdir ~/.kube/
[root@ssd-okd4-services okd4-install]# cp ./install_dir/auth/kubeconfig ~/.kube/config
[root@ssd-okd4-services okd4-install]# ls -l ~/.kube/config
-rw-r-----. 1 root root 8946 Aug 6 00:09 /root/.kube/config
```
**_NOTE_:** Or export "KUBECONFIG" instead of ".kube/config" : `export KUBECONFIG=/root/okd4-install/install_dir/auth/kubeconfig`
##### 3.4 Check pods status on master node
Check node status
```
[root@ssd-okd4-services okd4-install]# oc whoami
system:admin
[root@ssd-okd4-services okd4-install]# oc get node
NAME STATUS ROLES AGE VERSION
okd4-master.okd4.okd.homelab.com Ready master 27m v1.23.5+3afdacb
```
Check clusteroperators
```
[root@ssd-okd4-services okd4-install]# oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.10.0-0.okd-2022-07-09-073606 False False True 18m OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.122.29:443/healthz": dial tcp 172.30.122.29:443: connect: connection refused...
baremetal 4.10.0-0.okd-2022-07-09-073606 True False False 17m
cloud-controller-manager 4.10.0-0.okd-2022-07-09-073606 True False False 24m
cloud-credential 4.10.0-0.okd-2022-07-09-073606 True False False 45m
cluster-autoscaler 4.10.0-0.okd-2022-07-09-073606 True False False 17m
config-operator 4.10.0-0.okd-2022-07-09-073606 True False False 19m
console 4.10.0-0.okd-2022-07-09-073606 False False True 8m27s RouteHealthAvailable: console route is not admitted
csi-snapshot-controller 4.10.0-0.okd-2022-07-09-073606 True False False 18m
dns 4.10.0-0.okd-2022-07-09-073606 True False False 15m
etcd 4.10.0-0.okd-2022-07-09-073606 True False False 15m
image-registry 4.10.0-0.okd-2022-07-09-073606 True False False 9m8s
ingress 4.10.0-0.okd-2022-07-09-073606 True False True 16m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-8ff64bc5b-z4fnl" cannot be scheduled: 0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. Make sure you have sufficient worker nodes.)
insights 4.10.0-0.okd-2022-07-09-073606 True False False 49s
kube-apiserver 4.10.0-0.okd-2022-07-09-073606 True False False 11m
kube-controller-manager 4.10.0-0.okd-2022-07-09-073606 True False False 11m
kube-scheduler 4.10.0-0.okd-2022-07-09-073606 True False False 11m
kube-storage-version-migrator 4.10.0-0.okd-2022-07-09-073606 True False False 10m
machine-api 4.10.0-0.okd-2022-07-09-073606 True False False 10m
machine-approver 4.10.0-0.okd-2022-07-09-073606 True False False 17m
machine-config True True True 7m27s Unable to apply 4.10.0-0.okd-2022-07-09-073606: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1)
marketplace 4.10.0-0.okd-2022-07-09-073606 True False False 16m
monitoring Unknown True Unknown 17m Rolling out the stack.
network 4.10.0-0.okd-2022-07-09-073606 True True False 20m Deployment "openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
node-tuning 4.10.0-0.okd-2022-07-09-073606 True False False 15m
openshift-apiserver 4.10.0-0.okd-2022-07-09-073606 True False False 9m29s
openshift-controller-manager 4.10.0-0.okd-2022-07-09-073606 True False False 10m
openshift-samples 4.10.0-0.okd-2022-07-09-073606 True False False 8m20s
operator-lifecycle-manager 4.10.0-0.okd-2022-07-09-073606 True False False 14m
operator-lifecycle-manager-catalog 4.10.0-0.okd-2022-07-09-073606 True False False 11m
operator-lifecycle-manager-packageserver 4.10.0-0.okd-2022-07-09-073606 True False False 11m
service-ca 4.10.0-0.okd-2022-07-09-073606 True False False 18m
storage 4.10.0-0.okd-2022-07-09-073606 True False False 19m
```
Check all PODs, You should see PODs in Running/Pending/Completed status.
```
[root@ssd-okd4-services okd4-install]# oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator openshift-apiserver-operator-7b57fbbb4b-t4tjb 1/1 Running 1 (19m ago) 27m
openshift-apiserver apiserver-7744bdd5f5-6hfs4 2/2 Running 3 (14m ago) 17m
openshift-authentication-operator authentication-operator-7c67cf9d6d-px2rs 1/1 Running 1 (20m ago) 27m
openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-5b855fd95c-bhvl6 2/2 Running 0 27m
openshift-cloud-credential-operator cloud-credential-operator-5448765698-h2xbr 2/2 Running 0 27m
openshift-cluster-machine-approver machine-approver-6fb56d9777-72cn8 2/2 Running 0 27m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5fb447984f-h7v7f 1/1 Running 0 27m
openshift-cluster-node-tuning-operator tuned-cpbqs 1/1 Running 0 17m
openshift-cluster-samples-operator cluster-samples-operator-84ddc465c4-rjrj9 2/2 Running 0 11m
openshift-cluster-storage-operator cluster-storage-operator-5bc758cdc5-scv8h 1/1 Running 1 (20m ago) 27m
openshift-cluster-storage-operator csi-snapshot-controller-6b56df796f-brpns 1/1 Running 2 (14m ago) 21m
openshift-cluster-storage-operator csi-snapshot-controller-operator-7c9b645b4f-v5r7r 1/1 Running 0 27m
openshift-cluster-storage-operator csi-snapshot-webhook-556df58965-d8hh5 1/1 Running 0 21m
openshift-cluster-version cluster-version-operator-7d9c76888b-rvk78 1/1 Running 0 27m
openshift-config-operator openshift-config-operator-669b58b97f-l2zz6 1/1 Running 1 (20m ago) 27m
openshift-console-operator console-operator-9f59cb585-xftr5 1/1 Running 0 11m
openshift-console downloads-7945dd564-2bwnr 1/1 Running 0 10m
openshift-controller-manager-operator openshift-controller-manager-operator-d6c8f59b8-qfwr6 1/1 Running 1 (19m ago) 27m
openshift-controller-manager controller-manager-6w9pr 1/1 Running 0 11m
openshift-dns-operator dns-operator-5499565b9d-9gwtr 2/2 Running 0 27m
openshift-dns dns-default-jjw7r 2/2 Running 0 19m
openshift-dns node-resolver-g8gd6 1/1 Running 0 19m
openshift-etcd-operator etcd-operator-f9d78946f-drs2g 1/1 Running 1 (20m ago) 27m
openshift-etcd etcd-okd4-master.okd4.okd.homelab.com 4/4 Running 0 14m
openshift-etcd installer-2-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 20m
openshift-etcd installer-4-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 17m
openshift-image-registry cluster-image-registry-operator-7dc6c985f8-nr9jh 1/1 Running 0 27m
openshift-image-registry node-ca-bfwb2 1/1 Running 0 114s
openshift-ingress-operator ingress-operator-69b895ccb8-292mh 2/2 Running 3 (14m ago) 27m
openshift-ingress router-default-8ff64bc5b-z4fnl 0/1 Pending 0 19m
openshift-insights insights-operator-7fbc56b87f-kw5f7 1/1 Running 2 (14m ago) 27m
openshift-kube-apiserver-operator kube-apiserver-operator-5ffc5f559b-cr4c5 1/1 Running 1 (20m ago) 27m
openshift-kube-apiserver installer-2-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 19m
openshift-kube-apiserver installer-3-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 17m
openshift-kube-apiserver installer-4-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 13m
openshift-kube-apiserver kube-apiserver-okd4-master.okd4.okd.homelab.com 5/5 Running 0 12m
openshift-kube-controller-manager-operator kube-controller-manager-operator-578fb4ffb8-q9whf 1/1 Running 1 (20m ago) 27m
openshift-kube-controller-manager installer-5-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 19m
openshift-kube-controller-manager installer-6-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 13m
openshift-kube-controller-manager installer-7-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 8m29s
openshift-kube-controller-manager kube-controller-manager-okd4-master.okd4.okd.homelab.com 4/4 Running 1 (6m44s ago) 7m39s
openshift-kube-controller-manager revision-pruner-6-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 11m
openshift-kube-controller-manager revision-pruner-7-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 8m31s
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-85c6464c44-kpx5l 1/1 Running 1 (20m ago) 27m
openshift-kube-scheduler installer-3-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 20m
openshift-kube-scheduler installer-4-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 19m
openshift-kube-scheduler installer-5-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 19m
openshift-kube-scheduler installer-6-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 13m
openshift-kube-scheduler openshift-kube-scheduler-okd4-master.okd4.okd.homelab.com 3/3 Running 0 12m
openshift-kube-scheduler revision-pruner-6-okd4-master.okd4.okd.homelab.com 0/1 Completed 0 11m
openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-69d8d886cb-d8627 1/1 Running 1 (19m ago) 27m
openshift-kube-storage-version-migrator migrator-85976b4574-qvgnl 1/1 Running 0 20m
openshift-machine-api cluster-autoscaler-operator-58cb8cc768-gbsrs 2/2 Running 3 (14m ago) 27m
openshift-machine-api cluster-baremetal-operator-697f959f47-p8gh8 2/2 Running 0 27m
openshift-machine-api machine-api-operator-7649c6f59-xwwc4 2/2 Running 0 27m
openshift-machine-config-operator machine-config-controller-54d6bc656f-j8rjg 1/1 Running 0 19m
openshift-machine-config-operator machine-config-daemon-wngv7 2/2 Running 0 22m
openshift-machine-config-operator machine-config-operator-59c476b849-kpvn6 1/1 Running 0 27m
openshift-machine-config-operator machine-config-server-fkjmv 1/1 Running 0 19m
openshift-marketplace community-operators-jb8h9 1/1 Running 0 14m
openshift-marketplace marketplace-operator-5985df74f9-n5j6m 1/1 Running 4 (14m ago) 27m
openshift-monitoring cluster-monitoring-operator-c48fd7c76-nwpbt 2/2 Running 0 27m
openshift-monitoring kube-state-metrics-849c6bb5d5-b5x7x 0/3 Pending 0 17m
openshift-monitoring node-exporter-5h8s9 2/2 Running 0 17m
openshift-monitoring openshift-state-metrics-7896d98645-4p9ms 0/3 Pending 0 17m
openshift-monitoring prometheus-adapter-595764bbc-7vfbh 0/1 Pending 0 11m
openshift-monitoring prometheus-operator-78bd5cc486-9jcxw 2/2 Running 0 20m
openshift-multus multus-9mhkw 1/1 Running 0 26m
openshift-multus multus-additional-cni-plugins-z6n4q 1/1 Running 0 26m
openshift-multus multus-admission-controller-wnspz 2/2 Running 0 22m
openshift-multus network-metrics-daemon-2nqqc 2/2 Running 0 26m
openshift-network-diagnostics network-check-source-6fdcb969d6-6nf7d 0/1 Pending 0 26m
openshift-network-diagnostics network-check-target-jds26 1/1 Running 0 26m
openshift-network-operator network-operator-546db9df5d-d98n7 1/1 Running 0 27m
openshift-oauth-apiserver apiserver-788c59c55b-m6lc2 1/1 Running 1 (14m ago) 17m
openshift-operator-lifecycle-manager catalog-operator-55956f4487-dz542 1/1 Running 0 27m
openshift-operator-lifecycle-manager collect-profiles-27661860-6nsvt 0/1 Pending 0 8m32s
openshift-operator-lifecycle-manager olm-operator-775bfb89b7-l4bpq 1/1 Running 0 27m
openshift-operator-lifecycle-manager package-server-manager-6db8bb5585-7rfqb 1/1 Running 1 (15m ago) 27m
openshift-operator-lifecycle-manager packageserver-7fc675f65d-hhnkx 1/1 Running 0 17m
openshift-ovn-kubernetes ovnkube-master-4sfzx 6/6 Running 0 26m
openshift-ovn-kubernetes ovnkube-node-4sgxs 5/5 Running 0 26m
openshift-service-ca-operator service-ca-operator-86bcf476fb-g4bgs 1/1 Running 1 (19m ago) 27m
openshift-service-ca service-ca-54b4cf6549-kf8fn 1/1 Running 0 21m
```
**_Note_:** "installer" PODs might in error status which is fine because they are for configuring nodes. The pods will be re-run again if they have Error.
Check CSR status
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-5pjzn 35m kubernetes.io/kubelet-serving system:node:okd4-master.okd4.okd.homelab.com <none> Approved,Issued
csr-fs7z4 36m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-kmw8k 36m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
system:openshift:openshift-authenticator-2pxvw 25m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued
system:openshift:openshift-monitoring-764wv 24m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-monitoring:cluster-monitoring-operator <none> Approved,Issued
```
#### 4. Deploy OKD4 Worker Node
##### 4.1 Power on Worker-1 VM and add boot options below to install.
```
ip=192.168.122.53::192.168.122.1:255.255.255.0:::none
nameserver=192.168.122.50 coreos.inst.install_dev=/dev/vda
coreos.inst.image_url=http://192.168.122.50:8080/okd4/fcos36.raw.xz
coreos.inst.ignition_url=http://192.168.122.50:8080/okd4/worker.ign
```
##### 4.2 Check kubelet service
After installation completed and reboot, ssh to the worker node
```
[root@ssd-okd4-services okd4-install]# ssh worker-1 -l core
The authenticity of host 'worker-1 (192.168.122.53)' can't be established.
ECDSA key fingerprint is SHA256:uesRavGkWYbZmuTvwItQ4/SEljIxVWbkrUO9ATONJh0.
ECDSA key fingerprint is MD5:38:7a:f6:91:4e:69:36:b0:3d:bc:54:02:a8:9e:bc:7b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'worker-1,192.168.122.53' (ECDSA) to the list of known hosts.
Fedora CoreOS 35
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[core@okd4-worker-1 ~]$ sudo su -
Last login: Thu Aug 4 14:42:03 UTC 2022 on pts/0
[root@okd4-worker-1 ~]# systemctl status kubelet
[root@okd4-worker-1 ~]# journalctl -u kubelet -f
```
**_Note_:** form the systemctl status and journalctl command, you will some error logging. For example:
```
[root@okd4-worker-1 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-mco-default-env.conf, 10-mco-default-madv.conf, 20-logging.conf, 20-nodenet.conf
Active: active (running) since Fri 2022-08-05 15:20:29 UTC; 1min 9s ago
Process: 1799 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Process: 1800 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
Process: 1801 ExecStartPre=/bin/rm -f /var/lib/kubelet/memory_manager_state (code=exited, status=0/SUCCESS)
Main PID: 1802 (kubelet)
Tasks: 15 (limit: 9419)
Memory: 28.0M
CPU: 1.541s
CGroup: /system.slice/kubelet.service
└─ 1802 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.k>
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.783959 1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.884946 1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: W0805 15:21:38.945933 1802 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage>
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.946190 1802 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "cs>
Aug 05 15:21:38 okd4-worker-1.okd4.okd.homelab.com hyperkube[1802]: E0805 15:21:38.985410 1802 kubelet.go:2484] "Error getting node" err="node \"okd4-worker-1.okd4.okd.homelab.com\" not found"
```
##### 4.3 Approve CSR
Check CSR status
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-5pjzn 48m kubernetes.io/kubelet-serving system:node:okd4-master.okd4.okd.homelab.com <none> Approved,Issued
csr-bqdts 4m58s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-fs7z4 48m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-kmw8k 48m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-wvx5m 5m14s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
system:openshift:openshift-authenticator-2pxvw 37m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued
system:openshift:openshift-monitoring-764wv 37m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-monitoring:cluster-monitoring-operator <none> Approved,Issued
```
Approve the CSR in pending status
```
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-bqdts
certificatesigningrequest.certificates.k8s.io/csr-bqdts approved
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-wvx5m
certificatesigningrequest.certificates.k8s.io/csr-wvx5m approved
```
Check cluster node status
```
[root@ssd-okd4-services okd4-install]# oc get node
NAME STATUS ROLES AGE VERSION
okd4-master.okd4.okd.homelab.com Ready master 50m v1.23.5+3afdacb
okd4-worker-1.okd4.okd.homelab.com NotReady worker 79s v1.23.5+3afdacb
```
Check CSR status again
```
[root@ssd-okd4-services okd4-install]# oc get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-5pjzn 51m kubernetes.io/kubelet-serving system:node:okd4-master.okd4.okd.homelab.com <none> Approved,Issued
csr-bqdts 7m52s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-fs7z4 51m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-kmw8k 51m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
csr-smlvw 90s kubernetes.io/kubelet-serving system:node:okd4-worker-1.okd4.okd.homelab.com <none> Pending
csr-wvx5m 8m8s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Approved,Issued
system:openshift:openshift-authenticator-2pxvw 40m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-authentication-operator:authentication-operator <none> Approved,Issued
system:openshift:openshift-monitoring-764wv 40m kubernetes.io/kube-apiserver-client system:serviceaccount:openshift-monitoring:cluster-monitoring-operator <none> Approved,Issued
```
Approve the CSR for the worker node.
```
[root@ssd-okd4-services okd4-install]# oc adm certificate approve csr-smlvw
certificatesigningrequest.certificates.k8s.io/csr-smlvw approved
```
Check node status, it would take several minutes to get worker node to ready status
```
[root@ssd-okd4-services okd4-install]# oc get node
NAME STATUS ROLES AGE VERSION
okd4-master.okd4.okd.homelab.com Ready master 56m v1.23.5+3afdacb
okd4-worker-1.okd4.okd.homelab.com Ready worker 6m40s v1.23.5+3afdacb
```
##### 4.3 Check PODs status on worker node.
It would take several minutes to run PODs on worker node.
In the end, it should have PODs in running status on worker node. For example:
```
[root@ssd-okd4-services okd4-install]# oc get pods -A --field-selector spec.nodeName=okd4-worker-1.okd4.okd.homelab.com
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-cluster-node-tuning-operator tuned-nxwdd 1/1 Running 0 12m
openshift-dns dns-default-sfd27 2/2 Running 0 6m5s
openshift-dns node-resolver-wswnm 1/1 Running 0 12m
openshift-image-registry node-ca-pgvj5 1/1 Running 0 12m
openshift-ingress-canary ingress-canary-cj2sn 1/1 Running 0 6m5s
openshift-ingress router-default-8ff64bc5b-z4fnl 1/1 Running 0 49m
openshift-machine-config-operator machine-config-daemon-5nt46 2/2 Running 0 12m
openshift-monitoring alertmanager-main-0 6/6 Running 0 2m47s
openshift-monitoring grafana-65b68c96d-dz2ll 3/3 Running 0 5m15s
openshift-monitoring kube-state-metrics-849c6bb5d5-b5x7x 3/3 Running 0 47m
openshift-monitoring node-exporter-wfptq 2/2 Running 0 12m
openshift-monitoring openshift-state-metrics-7896d98645-4p9ms 3/3 Running 0 47m
openshift-monitoring prometheus-adapter-595764bbc-7vfbh 1/1 Running 2 (4m5s ago) 42m
openshift-monitoring prometheus-k8s-0 6/6 Running 0 5m11s
openshift-monitoring thanos-querier-c6bcb64b9-kq6g8 6/6 Running 0 5m13s
openshift-multus multus-additional-cni-plugins-w4lr8 1/1 Running 0 12m
openshift-multus multus-z7t87 1/1 Running 0 12m
openshift-multus network-metrics-daemon-8n2g2 2/2 Running 0 12m
openshift-network-diagnostics network-check-source-6fdcb969d6-6nf7d 1/1 Running 0 57m
openshift-network-diagnostics network-check-target-dlnkq 1/1 Running 0 12m
openshift-operator-lifecycle-manager collect-profiles-27661890-jh28h 0/1 Completed 0 8m59s
openshift-ovn-kubernetes ovnkube-node-8sj29 5/5 Running 0 12m
```
#### 5. Check Cluster Status
##### 5.1 Update Haproxy to remove bootstrap node.
```
[root@ssd-okd4-services okd4-install]# sed -i '/ okd4-bootstrap /s/^/#/' /etc/haproxy/haproxy.cfg
[root@ssd-okd4-services okd4-install]# cat /etc/haproxy/haproxy.cfg | grep bootstrap
# server okd4-bootstrap 192.168.122.51:6443 check
# server okd4-bootstrap 192.168.122.51:22623 check
[root@ssd-okd4-services okd4-install]# systemctl reload haproxy
```
*__Note__*: to remove #
```sed -i '/ okd4-bootstrap /s/^./ /' /etc/haproxy/haproxy.cfg```
##### 5.2 Power off the bootstrap VM.
##### 5.3 Access web console.
Get admin password
```
[root@ssd-okd4-services okd4-install]# cat install_dir/auth/kubeadmin-password
FxZcu-qHP3E-KsT4T-YZzC2[root@ssd-okd4-services okd4-install]#
```
Ensure the KVM Server has IP forwarding enabled
```
$ cat /proc/sys/net/ipv4/ip_forward
1
```
Check iptables on KVM Server, remove rules which prevent traffic to "default" network (virbr0). For example:
```
$ iptables -v --line-numbers -L
Chain LIBVIRT_FWI (1 references)
num pkts bytes target prot opt in out source destination
1 49M 71G ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
2 1 60 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWO (1 references)
num pkts bytes target prot opt in out source destination
1 26M 2676M ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
2 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
```
Delete rule via the line number.
```
$ iptables -D LIBVIRT_FWI 2
$ iptables -D LIBVIRT_FWO 2
```
Add hosts to hosts file on the cleint node. For example:
```
192.168.122.50 console-openshift-console.apps.okd4.okd.homelab.com
192.168.122.50 oauth-openshift.apps.okd4.okd.homelab.com
```
Add static route for 192.168.122.0 via IP of KVM server on the client.
```
$ ip route add 192.168.122.0/24 via 192.168.1.15 dev eno1
$ ip route show
default via 192.168.1.1 dev eno1 proto dhcp src 192.168.1.27 metric 100
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.27 metric 100
192.168.122.0/24 via 192.168.1.15 dev eno1
```
Ensure ping success via the domain name
```
$ ping console-openshift-console.apps.okd4.okd.homelab.com -c 3
PING console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50) 56(84) bytes of data.
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=1 ttl=63 time=0.908 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=2 ttl=63 time=0.726 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=3 ttl=63 time=0.598 ms
$ ping oauth-openshift.apps.okd4.okd.homelab.com -c 3
PING oauth-openshift.apps.okd4.okd.homelab.com (192.168.122.50) 56(84) bytes of data.
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=1 ttl=63 time=0.616 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=2 ttl=63 time=0.403 ms
64 bytes from console-openshift-console.apps.okd4.okd.homelab.com (192.168.122.50): icmp_seq=3 ttl=63 time=0.449 ms
```
Open web browser to access **console-openshift-console.apps.okd4.okd.homelab.com**
Redirected to oauth url for login.
Enter username **kubeadmin** and password from previous step.
OKD4 Web console login successfully.
##### 5.4 Check Haproxy stats
Check Haproxy stats enabled in `/etc/haproxy/haproxy.cfg`.
```
listen stats
bind :9000
mode http
stats enable
stats uri /
```
Allow TCP port 9000
```
[root@ssd-okd4-services ~]# sudo firewall-cmd --permanent --add-port=9000/tcp
success
[root@ssd-okd4-services ~]# firewall-cmd --reload
success
```
Open Haproxy stats page
##### 5.5 Check routes
```
[root@ssd-okd4-services okd4-install]# oc get route -A
NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
openshift-authentication oauth-openshift oauth-openshift.apps.okd4.okd.homelab.com oauth-openshift 6443 passthrough/Redirect None
openshift-console console console-openshift-console.apps.okd4.okd.homelab.com console https reencrypt/Redirect None
openshift-console downloads downloads-openshift-console.apps.okd4.okd.homelab.com downloads http edge/Redirect None
openshift-ingress-canary canary canary-openshift-ingress-canary.apps.okd4.okd.homelab.com ingress-canary 8080 edge/Redirect None
openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.okd4.okd.homelab.com /api alertmanager-main web reencrypt/Redirect None
openshift-monitoring grafana grafana-openshift-monitoring.apps.okd4.okd.homelab.com grafana https reencrypt/Redirect None
openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.okd4.okd.homelab.com prometheus-k8s web reencrypt/Redirect None
openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.okd4.okd.homelab.com /api thanos-querier web reencrypt/Redirect None
```
#### Troubleshoot & Tips:
- To delete all non-running containers in bootstrap node.
`podman container prune`
- no space left in /var/run
https://github.com/openshift/okd/issues/649
```
$ rm -rf /run/mco-machine-os-content
$ ln -s /tmp /run/mco-machine-os-content
```
- /etc/resolv.conf has name server 192.168.122.1 (Gateway IP) instead of 192.168.122.50
In the Fedora CoreOS 36, `/etc/resolv.conf` is a symbolic link to `/var/run/systemd/resolve/resolv.conf`. This is managemed by systemd-resolved service. Ensure you specified `ip` and `nameserver` options in kernel boot options.
- `oc get clusteroperators` command shows **machine-config** clusteroperators has error.
```
machine-config True True True 40h Unable to apply 4.10.0-0.okd-2022-07-09-073606: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1)
```
```
[root@ssd-okd4-services okd4-install]# oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master False True True 1 0 0 1 40h
worker rendered-worker-c76b9c8896ffcb19ef85c1c5d8cacd39 True False False 1 1 1 0 40h
```
Still unsure root cause. related posts:
https://githubhot.com/repo/openshift/okd/issues/1289
https://stackoverflow.com/questions/73101073/newly-install-okd4-cluster-getting-machine-config-errors
#### Todo:
1. use DHCP with DNS server to the ssd-okd4-services.
https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/
https://unix.stackexchange.com/questions/346967/obtain-ip-from-dhcp-sever-but-set-dns-servers-statically-on-debian
**/etc/NetworkManager/system-connections**
2. auto enter kernel options.
#### Reference:
Installation:
https://www.okd.io/installation/
https://medium.com/swlh/guide-okd-4-5-single-node-cluster-832693cb752b
https://itnext.io/guide-installing-an-okd-4-5-cluster-508a2631cbee
https://upstreamwithoutapaddle.com/home-lab/prepare-kvm-okd-install/
https://craig-robinson.medium.com/openshift-4-4-okd-bare-metal-install-on-vmware-home-lab-6841ce2d37eb
https://blog.csdn.net/networken/article/details/123953256
https://blog.maumene.org/2020/11/18/OKD-or-OpenShit-in-one-box.html
http://www.gbca.cn/node/132
Other:
https://developers.redhat.com/blog/2020/03/12/how-to-customize-fedora-coreos-for-dedicated-workloads-with-ostree#
https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/
https://github.com/coreos/fedora-coreos-docs/issues/89
https://github.com/openshift/okd/blob/master/FAQ.md#which-fedora-coreos-should-i-use
https://computingforgeeks.com/how-to-install-okd-openshift-cluster-on-openstack/
http://www.yylin.io/2019/08/12/openshift-ansible-v3-11/#%E5%BE%8C%E7%BA%8C%E8%A8%AD%E5%AE%9A
Harbor:
https://goharbor.io
https://goharbor.io/docs/2.4.0/install-config/configure-https/
https://goharbor.io/docs/2.4.0/install-config/troubleshoot-installation/#https
https://www.digitalocean.com/community/tutorials/how-to-set-up-and-configure-a-certificate-authority-ca-on-ubuntu-20-04
Sign in with Wallet
Connect another wallet