---
# System prepended metadata

title: Openshift 4.11 UPI安装部署

---

# Openshift 4.11 UPI安装部署



#OpenShift5个

#容器5个

本篇用于记录和梳理Openshift 4.11 UPI(User Provisioned Infrastructure) bare metal的安装部署过程和所需的配置文件。

底层环境采用esxi7.0虚拟化环境。具体架构信息如下：

![Image](https://mmbiz.qpic.cn/mmbiz_png/YiazHKG87rmADODdTQK957cZdwZ2lMtwJsA0FkAXlfZ2N0N2ntHGBzhCb2FpqhhUibgj1ct6RdXlmIXqdkrymUHw/640?wx_fmt=png&wxfrom=5&wx_lazy=1&wx_co=1)

**1 准备工作：**

1.  下载 CentOS 8 x86_64 image以及准备对应yum源
    
2.  登陆RedHat OpenShift Cluster Manager https://console.redhat.com/openshift
    
3.  选择cluster ->  create cluster -> datacenter -> bare metal(x86_64) -> User-provisioned infrastructure
    
4.  下载相关文件:
    

-   Openshift Installer for Linux
    
-   Pull secret
    
-   Command Line tools
    
-   Red Hat Enterprise Linux CoreOS (RHCOS) ISO
    

**2 准备Registry Node**

Registry node将从quay.io下载openshift安装所需的镜像包到本地环境，作为本地镜像仓库提高安装部署的效率和速度。      

安装harbor需要的docker-ce以及docker-compose  

#curl -o docker.repo https://download.docker.com/linux/centos/docker-ce.repo    

#yum install docker-ce docker-compose-plugin 

#systemctl enable docker && systemctl start docker 

下载harbor离线安装包

#wget https://github.com/goharbor/harbor/releases/download/v2.6.0/harbor-offline-installer-v2.6.0.tgz

#tar zxvf harbor-offline-installer-v2.6.0.tgz  

harbor镜像仓库采用https加密方式，因此需要使用到自签名证书进行加密。

生产根证书

#openssl genrsa -out ca.key 4096

#openssl req -x509 -new -nodes -sha512 -days 3650   -subj "/C=CN/ST=Shanghai/L=Shanghai/O=is/OU=Personal/CN=is.com"  -key ca.key -out ca.crt

生成签名请求

#openssl genrsa -out is.com.key 4096

#openssl req -sha512 -new  -subj "/C=CN/ST=Shanghai/L=Shanghai/O=is/OU=Personal/CN=is.com" -key is.com.key -out is.com.csr

#vi v3.ext

authorityKeyIdentifier=keyid,issuer

basicConstraints=CA:FALSE

keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[alt_names]

DNS.1=is.com

DNS.2=is

DNS.3=registry.ocp41.is.com

使用根证书进行签名  

#openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in is.com.csr -out is.com.crt

将crt文件转换成cert

#openssl x509 -inform PEM -in is.com.crt -out is.com.cert

将证书拷贝至docker目录

#mkdir -p /etc/docker/certs.d/registry.ocp411.is.com/

#cp is.com.cert /etc/docker/certs.d/registry.ocp411.is.com/

#cp is.com.key /etc/docker/certs.d/registry.ocp411.is.com/

#cp ca.crt  /etc/docker/certs.d/registry.ocp411.is.com/

修改harbor配置模版

#cp harbor.yml.tmpl harbor.yml

编辑安装配置文件

#vi harbor.yml 

hostname: registry.ocp4.is.com

http:

  port: 5080

https:

  port: 5088

  certificate: /etc/docker/certs.d/registry.ocp4.is.com/is.com.cert

  private_key: /etc/docker/certs.d/registry.ocp4.is.com/is.com.key

harbor_admin_password: Harbor12345

database:

  password: root123

data_volume: /data

执行准备文件，检查安装文件完整性

#./prepare 

执行安装脚本，并且安装notray，trivy，chartmuseum

# ./install.sh --with-notary --with-trivy --with-chartmuseum

安装完成后测试登录

#docker login -u admin https://registry.ocp4.is.com

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

创建新的项目ocp4用于存放openshift 4.11的镜像文件

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

解压软件包

#tar zxvf oc-4.11.8-linux.tar.gz

将文件移至对应目录

#mv oc /usr/local/sbin/ 

#mv kubectl /usr/local/sbin/

将pull-secert.txt转换为json文件

#cat pull-secert.txt | jq . > /root/pull-secert.json

将harbor密码进行转换

#echo -n 'admin:Harbor12345' | base64 -w0

将获得的结果添加入pull-secert.json文件

添加后的pull-secert.json文件如下：

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

定义变量

#OCP_RELEASE=4.11.8 #对应openshift版本

#LOCAL_REGISTRY='registry.ocp4.is.com' #本地registry

#LOCAL_REPOSITORY=ocp4/openshift4.11 #本地repo

#PRODUCT_REPO='openshift-release-dev' #指定版本

#LOCAL_SECRET_JSON='/pull-secert.json'  #密码文件

#RELEASE_NAME="ocp-release" #指定名称

#ARCHITECTURE=x86_64 #架构

#REMOVABLE_MEDIA_PATH=/home/images #可移动介质目录，用于移动拷贝

#GODEBUG=x509ignoreCN=0

将harbor镜像仓库所使用的自签证书ca.crt拷贝至 /etc/pki/ca-trust/source/anchors/ 并执行证书更新

#cp ca.crt  /etc/pki/ca-trust/source/anchors/

#update-ca-trust extract 

#trust list |head -n 10 #可以查看到当前服务器安装的证书状态

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

开始镜像文件

#oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} 

镜像完成后提示成功，保存imageContentSources字段用于后续安装。

imageContentSources:

- mirrors:

  - registry.ocp4.is.com/ocp4/openshift4.11

  source: quay.io/openshift-release-dev/ocp-release

- mirrors:

  - registry.ocp4.is.com/ocp4/openshift4.11

  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

**3 准备bastion Node**

安装chrony NTP软件

 #yum install chrony -y

修改chrony配置

# vim /etc/chrony.conf

server  ntp1.aliyun.com  iburst #以server开头指定NTP服务器，可添加多台。

driftfile /var/lib/chrony/drift # 根据实际时间计算出服务器增减时间的比率，然后记录到一个文件中，在系统重启后为系统做出最佳时间补偿调整。

makestep 1.0 3 # 如果系统时钟的偏移量大于1秒，则允许系统时钟在前三次更新中步进。

rtcsync # 启用实时时钟（RTC）的内核同步。

allow 192.168.3.0/24 # 指定 NTP 客户端地址，以允许或拒绝连接到扮演时钟服务器的机器

keyfile /etc/chrony.keys # 指定包含 NTP 身份验证密钥的文件。

logdir /var/log/chrony # 指定日志文件的目录。

启动服务并加入系统自启动

#systemctl start chronyd && systemctl enable chronyd 

开启网络时间同步

#timedatectl set-ntp true

查看当前NTP服务器状态

#chronyc sources -v

  

安装bind DNS服务

#yum install bind bind-utils  

#vi /etc/named.conf

options {

        listen-on port 53 { any; };

        listen-on-v6 port 53 { ::1; };

        directory "/var/named";

        dump-file "/var/named/data/cache_dump.db";

        statistics-file "/var/named/data/named_stats.txt";

        memstatistics-file "/var/named/data/named_mem_stats.txt";

        secroots-file"/var/named/data/named.secroots";

        recursing-file"/var/named/data/named.recursing";

        allow-query     { any; };

        recursion yes;

        dnssec-enable yes;

        dnssec-validation yes;

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";

        session-keyfile "/run/named/session.key";

logging {

        channel default_debug {

                file "data/named.run";

                severity dynamic;

        };

};

zone "." IN {

    type hint;

    file "named.ca";

};

zone "ocp4.is.com" IN {

 type master;

 file "ocp4.is.com.zone";

};

zone "3.168.192.in-addr.arpa" IN {

 type master;

 file "3.168.192.in-addr.arpa.zone";

};

include "/etc/named.rfc1912.zones";

include "/etc/named.root.key";

添加A记录，部分域名将在后续使用。

#vi /var/named/ocp4.is.com.zone

$TTL 1D

@   IN SOA  @ ns1.ocp4.is.com. (

                    2021032700   ; serial

                    3H           ; refresh

                    30M          ; retry

                    2W           ; expire

                    1W )         ; minimum

@   NS  ns1.ocp4.is.com.

ns1        IN  A   192.168.3.60

dns        IN  A   192.168.3.60

registry    IN  A   192.168.3.60

bastion     IN  A   192.168.3.61

nfs        IN  A   192.168.3.61

lb         IN  A   192.168.3.61

api        IN  A   192.168.3.61

api-int    IN  A   192.168.3.61

bootstrap  IN  A   192.168.3.62

master1    IN  A   192.168.3.63

master2    IN  A   192.168.3.64

master3    IN  A   192.168.3.65

worker1    IN  A   192.168.3.66

worker2    IN  A   192.168.3.67

worker3    IN  A   192.168.3.68

*.apps       IN A   192.168.3.61

添加PTR记录

#vi /var/named/3.168.192.in-addr.arpa.zone

$TTL 1D

@   IN SOA  @ ns1.ocp4.is.com. (

                    2021032700   ; serial

                    3H           ; refresh

                    30M          ; retry

                    2W           ; expire

                    1W )         ; minimum

@   NS  ns1.ocp4.is.com.

60   IN  PTR ns1.ocp4.is.com.

60   IN  PTR dns.ocp4.is.com.

61   IN  PTR nfs.ocp4.is.com.

61   IN  PTR bastion.ocp4.is.com.

60   IN  PTR registry.ocp4.is.com.

61   IN  PTR lb.ocp4.is.com.

61   IN  PTR api.ocp4.is.com.

61   IN  PTR api-int.ocp4.is.com.

62   IN  PTR bootstrap.ocp4.is.com.

63   IN  PTR master1.ocp4.is.com.

64   IN  PTR master2.ocp4.is.com.

65   IN  PTR master3.ocp4.is.com.

66   IN  PTR worker1.ocp4.is.com.

67   IN  PTR worker2.ocp4.is.com.

68   IN  PTR worker3.ocp4.is.com.

启动named服务并加入系统自启动

#systemctl start named && systemctl enable named

编辑/etc/resolv.conf

#echo "nameserver 192.168.3.61" > /etc/resolv.conf

通过dig命令确认DNS是否生效

# dig registry.ocp4.is.com

#dig -x 192.168.3.60

  

安装NFS服务

#yum install nfs-utils -y

创建NFS共享目录

#mkdir /shares

修改属主

#chown -R nobody:nobody /shares/

修改权限

#chmod -R 777 /shares/

添加共享权限文件

#echo "/shares   192.168.3.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)" > /etc/exports

使共享目录生效

# exportfs -rv

启用NFS服务并加入开机启动

# systemctl enable nfs-server rpcbind &&  systemctl start nfs-server rpcbind nfs-mountd

  

  

安装HAproxy负载均衡

#yum install haproxy  

#vi /etc/haproxy/haproxy.cfg

# Global settings  

#---------------------------------------------------------------------

global

    maxconn     20000

    log         /dev/log local0 info

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    user        haproxy

    group       haproxy

    daemon

  

    # turn on stats unix socket

    stats socket /var/lib/haproxy/stats

  

#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

    log                     global

    mode                    http

    option                  httplog

    option                  dontlognull

    option http-server-close

    option redispatch

    option forwardfor       except 127.0.0.0/8

    retries                 3

    maxconn                 20000

    timeout http-request    10000ms

    timeout http-keep-alive 10000ms

    timeout check           10000ms

    timeout connect         40000ms

    timeout client          300000ms

    timeout server          300000ms

    timeout queue           50000ms

  

# Enable HAProxy stats

listen stats

    bind :9000

    stats uri /stats

    stats refresh 10000ms

  

# Kube API Server

frontend k8s_api_frontend

    bind :6443

    default_backend k8s_api_backend

    mode tcp

  

backend k8s_api_backend

    mode tcp

    balance source

    server      bootstrap 192.168.3.62:6443 check

    server      master1 192.168.3.63:6443 check

    server      master2 192.168.3.64:6443 check

    server      master3 192.168.3.65:6443 check

  

# OCP Machine Config Server

frontend ocp_machine_config_server_frontend

    mode tcp

    bind :22623

    default_backend ocp_machine_config_server_backend

  

backend ocp_machine_config_server_backend

    mode tcp

    balance source

    server      bootstrap 192.168.3.62:22623 check

    server      master1 192.168.3.63:22623 check

    server      master2 192.168.3.64:22623 check

    server      master3 192.168.3.65:22623 check

  

# OCP Ingress - layer 4 tcp mode for each. Ingress Controller will handle layer 7.

frontend ocp_http_ingress_frontend

    bind :80

    default_backend ocp_http_ingress_backend

    mode tcp

  

backend ocp_http_ingress_backend

    balance source

    mode tcp

    server      worker1 192.168.3.66:80 check

    server      worker2 192.168.3.67:80 check

  

frontend ocp_https_ingress_frontend

    bind *:443

    default_backend ocp_https_ingress_backend

    mode tcp

  

backend ocp_https_ingress_backend

    mode tcp

    balance source

    server      worker1 192.168.3.66:443 check

    server      worker2 192.168.3.67:443 check

  

 #systemctl start haproxy && systemctl enable haproxy  

安装HTTP服务

#yum install http 

#vi /etc/httpd/conf/httpd.conf

修改监听端口 调整为 9080

#vi  /etc/httpd/conf.d/welcome.conf

 修改Options +Indexes  

启动http并加入开机启动

#systemctl start httpd && systemctl enable httpd

  

**4 准备OCP安装配置文件**

创建OCP安装目录

#mkdir -p /opt/install/ocp/ 

生成ssh-key文件，用于登录bootstrap和master节点

#ssh-keygen -t rsa -b 4096 -N '' -f /opt/install/id_rsa

启动ssh-agent

#eval "$(ssh-agent -s)"

添加key，成功后会有对应提示

#ssh-add  /opt/install/id_rsa

解压缩openshift-install包

#tar zxvf openshift-install-linux.tar.gz

将可执行文件移动至指定目录

#mv openshift-install /usr/local/sbin

编辑install-config.yaml安装文件

#vim /opt/install/ocp/install-config.yaml 

apiVersion: v1  

baseDomain: is.com

compute:

  - hyperthreading: Enabled

    name: worker

    replicas: 0 # Must be set to 0 for User Provisioned Installation as worker nodes will be manually deployed.

controlPlane:

  hyperthreading: Enabled

  name: master

  replicas: 3

metadata:

  name: ocp4 # Cluster name

networking:

  clusterNetwork:

    - cidr: 10.128.0.0/14

      hostPrefix: 23

  networkType: OpenShiftSDN

  serviceNetwork:

    - 172.30.0.0/16

platform:

  none: {}

fips: false

pullSecret: '{"auths":{"registry.ocp4.is.com":{"auth":"YWRtaW46SGFyYm9yMTIzNDU=","email":"xxx@xxx.com"}},"cloud.openshift.com":{"auth":"xxxxxxxxxxxxxxxxxx"}}}'

sshKey: "ssh-rsa ... root@bastion.ocp4.is.com" #bastion的id_rsa.pub

additionalTrustBundle: |   #添加registry的ca.crt

  -----BEGIN CERTIFICATE-----

  ....

  -----END CERTIFICATE-----

imageContentSources: #添加OCP镜像完成后的Imagecontentsources

- mirrors:

  - registry.ocp4.is.com/ocp4/openshift4.11

  source: quay.io/openshift-release-dev/ocp-release

- mirrors:

  - registry.ocp4.is.com/ocp4/openshift4.11

  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

创建openshift manifests文件

#openshift-install create manifests --dir /opt/install/ocp/  

修改/opt/install/ocp/manifests/cluster-scheduler-02-config.yml将masterschedulable由true变更为false，让master节点不会被调度。

  

apiVersion: config.openshift.io/v1

kind: Scheduler

metadata:

  creationTimestamp: null

  name: cluster

spec:

  mastersSchedulable: false

  policy:

    name: ""

status: {}

创建点火文件

# openshift-install create ignition-configs --dir /opt/install/ocp/  

拷贝ign文件至http服务器

#cp /opt/install/ocp/*.ign     /var/www/html/

修改ign文件权限

#chmod 755 /var/www/html/*.ign 

  

**5 安装openshift** 

使用coreos启动bootstrap虚拟机以及master 虚拟机启动后运行如下命令

bootstrap节点

#sudo coreos-installer install /dev/sda -I http://192.168.3.61:9080/bootstrap.ign --insecure --insecure-ignition --append-karg ip=192.168.3.62::192.168.3.1:255.255.255.0:bootstrap.ocp4.is.com:ens192:none --append-karg nameserver=192.168.3.61 

master节点

#sudo coreos-installer install /dev/sda -I http://192.168.3.61:9080/master.ign --insecure --insecure-ignition --append-karg ip=192.168.3.63::192.168.3.1:255.255.255.0:master1.ocp4.is.com:ens192:none --append-karg nameserver=192.168.3.61   

安装完成其余节点

在bastion运行安装命令观察安装状态

#openshift-install --dir /opt/install/ocp/ wait-for bootstrap-complete --log-level=debug  

观察haproxy状态

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

bootstarp安装引导完成后，会提示可以移除bootstrap资源。此时修改haproxy。

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

注释对应bootstrap内容，重启haproxy服务

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

#systemctl restart haproxy

安装worker节点

#sudo coreos-installer install /dev/sda -I http://192.168.3.61:9080/worker.ign --insecure --insecure-ignition --append-karg ip=192.168.3.66::192.168.3.1:255.255.255.0:worker1.ocp4.is.com:ens192:none --append-karg nameserver=192.168.3.61

节点安装完成后，修改kubeconfig文件

cat /opt/install/ocp/auth/kubeconfig > /root/.kube/config

通过oc命令查看csr 是否有pending的内容

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

批准证书

# oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve  

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

由于依赖顺序关系，批准证书的操作需要执行多次

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

在bastion节点运行安装命令观察ocp集群安装状态

#openshift-install --dir /opt/install/ocp/  wait-for install-complete --log-level=debug

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

固化路由节点

标记label

#oc label node worker1.ocp4.is.com node-role.kubernetes.io/infra= --overwrite  

# oc label node worker2.ocp4.is.com node-role.kubernetes.io/infra= --overwrite

修改operator调度

#oc patch ingresscontrollers.operator.openshift.io default -n openshift-ingress-operator --type=merge --patch '{"spec":{"nodePlacement":{"nodeSelector":{"matchLabels":{"node-role.kubernetes.io/infra":""}}}}}'

安装过程中节点会逐个ready

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

oc get co 查看operator启用状态

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

安装完成提示  

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

登录控制台 https://console-openshift-console.apps.ocp4.is.com

#192.168.3.61 ocp-svc api.ocp4.is.com console-openshift-console.apps.ocp4.is.com oauth-openshift.apps.ocp4.is.com downloads-openshift-console.apps.ocp4.is.com alertmanager-main-openshift-monitoring.apps.ocp4.is.com grafana-openshift-monitoring.apps.ocp4.is.com prometheus-k8s-openshift-monitoring.apps.ocp4.is.com thanos-querier-openshift-monitoring.apps.ocp4.is.com#可添加hosts文件方便访问。  

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

登录后会有提示添加身份供应商，配置告警等。

设置集群，选择对应频道，对平台进行更新。

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

openshift bare metal安装之后不提供默认的存储空间，因此管理状态为removed，在集群安装完成后我们可以调整为managed状态

#oc edit configs.imageregistry.operator.openshift.io

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

#vi registry-vol.yaml

apiVersion: v1  

kind: PersistentVolume

metadata:

  name: registry-pv

spec:

  accessModes:

    - ReadWriteMany

  capacity:

    storage: 100Gi

  persistentVolumeReclaimPolicy: Retain

  nfs:

    path: /shares

    server: 192.168.3.61

创建存储

#oc create -f registry-vol.yaml

查看存储挂载状态

#oc get pvc -n openshift-image-registry![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)  

openshift集群需要增加认证方式，增加htpasswd方式

采用htpasswd方式进行加密

#htpasswd -n -B -b <username> <password>

#vi oauth-htpasswd.yaml

  

apiVersion: v1

data:

  htpasswd: YWRtaW46JDJ5JDA1JGl5NVNsQ2guN3gxSGlabUJZYlVody5Rck0zOVpidks5c3lVSlZtQzZ3V3ViZkE4bTRYdTFhCg==

kind: Secret

metadata:

  name: htpasswd-secret

  namespace: openshift-config

type: Opaque

---

apiVersion: config.openshift.io/v1

kind: OAuth

metadata:

  name: cluster

spec:

  identityProviders:

    - name: htpasswd_provider

      mappingMethod: claim

      type: HTPasswd

      htpasswd:

        fileData:

          name: htpasswd-secret

#oc apply -f oauth-htpasswd.yaml

# oc adm policy add-cluster-role-to-user cluster-admin admin

再次登录控制台，选择htpasswd_provider方式登录

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

可以在operatorhub中选择对应的operator进行安装。

![Image](data:image/svg+xml,%3C%3Fxml version='1.0' encoding='UTF-8'%3F%3E%3Csvg width='1px' height='1px' viewBox='0 0 1 1' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Ctitle%3E%3C/title%3E%3Cg stroke='none' stroke-width='1' fill='none' fill-rule='evenodd' fill-opacity='0'%3E%3Cg transform='translate(-249.000000, -126.000000)' fill='%23FFFFFF'%3E%3Crect x='249' y='126' width='1' height='1'%3E%3C/rect%3E%3C/g%3E%3C/g%3E%3C/svg%3E)

至此openshift UPI on bare metal的安装已经完成。

如在安装过程出出现问题，需要进行定位可以使用gather命令收集对应日志

#openshift-install gather bootstrap --dir /opt/install/ocp --bootstrap=192.168.3.62 --master=192.168.3.63 --master=192.168.3.64 --master=192.168.3.65

  

参考链接：https://console.redhat.com/openshift/install/metal/user-provisioned

https://docs.openshift.com/container-platform/4.11/installing/installing_bare_metal/installing-bare-metal.html  

