DO380 "Red Hat OpenShift Administration III : Scaling Kubernetes Deployments in the Enterprise" notes in the margin

tags: `red hat, openshift, kubernetes, containers`

Course description: Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise

:memo: Table of contents

DO380 "Red Hat OpenShift Administration III : Scaling Kubernetes Deployments in the Enterprise" notes in the margin
- :memo: Table of contents

Chapter 1. Moving From Kubernetes to OpenShift

Kubernetes Docs: Organizing Cluster Access Using kubeconfig Files

Kubernetes Docs: Configure Access to Multiple Clusters

Reference Docs for Kustomize
Kubernetes Docs: The Kustomization File An overview of the content of a kustomization file.

DC vs deployment: CAP theorem

Latest Openshift Clients and installer

https://kustomize.io/

https://github.com/kubernetes-sigs/kustomize
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
kustomize targets kubernetes; it understands and can patch kubernetes style API objects. It's like make, in that what it does is declared in a file, and it's like sed, in that it emits edited text.

An Introduction to Kustomize
Southbridge blog post

Kustomize cfg: Configuration Basics

The Kubernetes Resource Model (KRM)

Kubernetes docs: Изучение объектов Kubernetes

Using Admission Controllers

OpenShift 3 Demystified. For Developers

Using Red Hat OpenShift image streams with Kubernetes deployments

How to Simplify Container Image Management in Kubernetes with OpenShift Image Streams

Chapter 2. Introduce automation on OpenShift

https://kubernetes.io/ru/docs/concepts/overview/working-with-objects/kubernetes-objects/

https://docs.okd.io/latest/rest_api/index.html

jq is a lightweight and flexible command-line JSON processor.

curl -s http://api.ocp4.domain.com:6443/apis/project.openshift.io/v1/projects \
|jq '.items[].metadata.name'

FX : Command-line tool and terminal JSON viewer

Automate OpenShift administration tasks using bash scripts and Ansible playbooks.

https://www.ansibleforkubernetes.com/
Youtube Video: Jeff Geerlings Kubernetes 101

Chapter 3. Manage operators with OpenShift

Openshift Docs: What Operators are?

Openshift Docs: Understanding Operator Lifecycle Manager

Resource	Short name	Description	looks like
ClusterServiceVersion (CSV)	csv	Primary metadata resource describes operator. For example: name, version, icon, required resources.	rpm package
CatalogSource	catsrc	Information for accessing repository of CSVs (Operators), CRDs, and packages that define an application. (packagemanifests from catalogsource)	yum repository
Subscription	sub	Keeps CSVs up to date by tracking a channel in a package. (channel,approval mode)	yum install
InstallPlan	ip	Calculated list of resources to be created to automatically install or upgrade a CSV.	file list and scriptlets that runs to install rpm
OperatorGroup	og	Configures all Operators deployed in the same namespace as the OperatorGroup object to watch for their custom resource (CR) in a list of namespaces or cluster-wide.
OperatorConditions	-	Creates a communication channel between OLM and an Operator it manages. Operators can write to the Status.Conditions array to communicate complex states to OLM.

OpenShift Commons Briefing: Operator Lifecycle Management with Evan Cordell (Red Hat)

techbloc.net blog post: Using Operator Lifecycle Manager and create custom Operator Catalog for Kubernetes

Demystifying Operator deployment in OpenShift
Делаем понятным развёртывание оператора в OpenShift

CatalogSource and PackageManifest

Channel and Subscription

Installed operator
O'Reilly free e-book: Kubernetes Operators: Automating the Container Orchestration Platform

Chapter 4. Implementing GitOps with Jenkins

https://www.prolinux.org/post/2017/09/vstuplenie-v-declarative-jenkins-pipelines/

Continuous integration vs. continuous delivery vs. continuous deployment

RedHat Developers Blogs: Building modern CI/CD workflows for serverless applications with Red Hat OpenShift Pipelines and Argo CD, Part 1
RedHat Developers Blogs: Building modern CI/CD workflows for serverless applications with Red Hat OpenShift Pipelines and Argo CD, Part 2

Youtube video: Что же такое GitOps? Его свойства и недостатки

Implement a GitOps workflow using containerized Jenkins to administer an OpenShift cluster.
https://habr.com/ru/company/southbridge/blog/535954/
https://developers.redhat.com/blog/2020/11/03/argo-cd-and-tekton-match-made-in-kubernetes-heaven

E-books The Path to GitOps Christian Hernandez

Chapter 5. Configuring Enterprise Authentication

Kubernetes Documentation | Reference | API Access Control | Authenticating

Openshift Docs: Supported Identity Providers

arctiq.ca blog post: OpenShift 4 Authentication via Azure AD

Chapter 6. Configuring Trusted TLS certificates

Openshift Docs: Adding API server certificates

Create a secret that contains the certificate chain and private key in the openshift-config namespace.

 oc create secret tls <secret> --cert=</path/to/cert.crt> \
                               --key=</path/to/cert.key>  \
                               -n openshift-config

secret is the name of the secret that will contain the certificate chain and private key.
/path/to/cert.crt is the path to the certificate chain on your local file system.
/path/to/cert.key is the path to the private key associated with this certificate.

Update the API server to reference the created secret.

 oc patch apiserver cluster \
     --type=merge -p \
     '{"spec":{"servingCerts": {"namedCertificates":
     [{"names": ["<FQDN>"], 
     "servingCertificate": {"name": "<secret>"}}]}}}'

Replace FQDN with the FQDN that the API server should provide the certificate for
Replace secret with the name used for the secret in the previous step.

Examine the apiserver/cluster object and confirm the secret is now referenced.

oc get apiserver cluster -o yaml
...
spec:
  servingCerts:
    namedCertificates:
    - names:
      - <FQDN>
      servingCertificate:
        name: <secret>
...

Ingress Operator in OpenShift Container Platform

Openshift Docs: Ingress Operator: Setting a custom default certificate

Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

oc --namespace openshift-ingress-operator get ingresscontrollers
NAME      AGE
default   10m

Update the IngressController CR to reference the new certificate secret.

oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key
oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default  --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

Verify the update was effective:

oc get --namespace openshift-ingress-operator ingresscontrollers/default --output jsonpath='{.spec.defaultCertificate}'

Configure OpenShift with trusted TLS certificates for external access to cluster services and applications.

Chapter 7. Configure Dedicated Node Pools

RedHat KB: How to add OpenShift 4 RHCOS Worker Nodes in UPI in new installations (< 24 hours)

Adding worker nodes to the OCP 4 UPI cluster existing 24+ hours

Adding new nodes to UPI cluster fails after upgrading to OpenShift 4.6+

OCP 4.6 Release notes: Ignition Spec updated to v3

Recommended cluster scaling practices

GitHub repo openshift/machine-config-operator: MachineConfigDaemon docs

GitHub repo openshift/machine-config-operator: Custom pool docs

Openshift docs: Red Hat CoreOS Server: Architecture
github openshift-installer docs: Troubleshooting Bootstrap Failures

https://www.youtube.com/watch?v=zQ7QfUY5Ulk

Chapter 8. Configure persistent storage

Understanding ephemeral storage

Pods use ephemeral local storage for scratch space, caching, and logs. Issues related to the lack of local storage accounting and isolation include the following:

Pods do not know how much local storage is available to them.
Pods cannot request guaranteed local storage.
Local storage is a best effort resource.
Pods can be evicted due to other pods filling the local storage, after which new pods are not admitted until sufficient storage has been reclaimed.

Unlike persistent volumes, ephemeral storage is unstructured and the space is shared between all pods running on a node, in addition to other uses by the system, the container runtime, and OpenShift Container Platform. The ephemeral storage framework allows pods to specify their transient local storage needs. It also allows OpenShift Container Platform to schedule pods where appropriate, and to protect the node against excessive use of local storage.

Kubernetes Docs: Volumes

Openshift Docs: Understanding persistent storage

Openshift Docs: Available storage options

Kubernetes Blog: Kubernetes 1.14: Local Persistent Volumes GA

Chapter 9. Manage cluster monitoring and metrics

Configure and manage the OpenShift monitoring stack.

https://prometheus.io/docs/: QUERYING PROMETHEUS
)
PromQL examples:

sum(container_memory_usage_bytes) / 1024 / 1024
sum(container_memory_usage_bytes{image!='',node="master0.domain.tld"}/1024/1024) by (node)
rate(apiserver_request_total{code!~"2.*"}[2m])
100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)
node_load5

Мониторинг и Kubernetes (Дмитрий Столяров, Флант, RootConf 2018)
Полное руководство по Prometheus в 2019 году

RedHat KB: Is it possible to add custom dashboards to OpenShift web console?
RedHat KB: Custom Grafana Dashboard for custom metrics
RedHat KB: Is it possible to add custom dashboards to OpenShift web console?

Ask an OpenShift Admin (Ep 31): Alertmanager configuration and customization
Полное руководство по Prometheus в 2019 году

Chapter 10. Provision and inspect cluster logging

Kubernetes Documentation |
Concepts | Cluster Administration | Logging Architecture

Chapter 10. Recover failed worker nodes

Inspect, troubleshoot, and remediate worker nodes in a variety of failure scenarios.

Openshift Docs: Replacing an unhealthy etcd member

https://cloud.redhat.com/blog/ocp-disaster-recovery-part-1-how-to-create-automated-etcd-backup-in-openshift-4.x

https://cloud.redhat.com/blog/ocp-disaster-recovery-part-2-recovering-an-openshift-4-ipi-cluster-with-the-loss-of-one-master-node

Disclaimer: You will find here notes and links to official docs with additional information on products and technologies that described on RedHat training.
THIS DOCUMENT DOES NOT REPRINT ANY COPYRIGHTED CONTENT FROM REDHAT TRAINING. You will find here only public accessible outline.