# K8s CRD/Controllers (1/2)

---

### k8s Master Components

- kube-apiserver
- etcd
- kube-scheduler
- kube-controller-manager
* Node/Replication/Endpoints(Services/Pods)/Service Account & Token
- cloud-controller-manager

---

### What is cloud-providers?

https://kubernetes.io/docs/concepts/architecture/cloud-controller/
- cloud controller manager -> kops
* Node controller
* Route controller
* Service controller
* Volume controller (> CSI)

- for os-in-os cases

---

### OpenStack Cloud Provider

- OpenStack Cloud Controller Manager
- Octavia Ingress Controller
- Cinder CSI Plugin
- Keystone Webhook Authentication - Authorization
- Client Keystone
- Cinder Standalone Provisioner
- Manila CSI Plugin
- Manila Provisioner
- Barbican KMS Plugin

---

### [AWS operator](https://aws.amazon.com/blogs/opensource/aws-service-operator-kubernetes-available/)

- DynamoDB Tables
- S3 Buckets
- Amazon Elastic Container Registry (Amazon ECR)
- Repositories
- SNS Topics
- SQS Queues
- SNS Subscriptions

---

## K8s is API-centric

| Declarative | State | Level based | Transparent |
| ----------- | ------- | ----------- | ----------- |
| What vs How | Desired | Doesn't rely on individual change | One control plane |
| build from config | Observed | Facilities composability| Less prone to error |
| Record changes | | | |

---

### k8s to build a pod/cake?

* Built-in
* node / pod / deployment / service

 curl localhost:8080/api/v1/namespaces/kube-system/services/ingress/

* CRD
* [cake](https://kccncosschn19eng.sched.com/event/NrnG/to-crd-or-not-to-crd-that-is-the-question-ed-king-sam-gunaratne-pivotal)
* room/light
* node/vm
* mysql/Spark/Cassandra

curl localhost:8080/apis/apiextensions.k8s.io/v1beta1

---

### Stateless is easy, stateful is hard

- k8s can handle stateless
- manage other stateful applications need extra knowledge

- CRD - StatefulSets, Services, ConfigMaps


---

## kubevirt


---

## kubevirt components

- `CRD`
- are added to the Kubernetes API
- `virt-controller`
- cluster wide logic associated with these new types
- `virt-handler`
- node specific logic associated with new types

---

### kubevirt yaml

apiVersion: kubevirt.io/v1alpha3
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  imagePullPolicy: Always

---

### rook - ceph crd


---

### Timeline
[Kubernetes API 与 Operator：不为人知的开发者战争](https://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2019/01/08/kubernetes-api-and-operator-history.html)

- v2017/01 v1.6 ThirdPartyResource
- 2017/04 v1.7 CustomResourceDefinition
- 2017/10 v1.9 CustomResourceValidation
- 2018/04 v1.11
- crd subresources /status, /scale
- multiple versions
- redhat http://operatorhub.io/
- knative / https://github.com/keleustes/oslc-operator

---

### crd sdk - pattern/template

- [kubebuilder](https://itnext.io/under-the-hood-of-kubebuilder-framework-ff6b38c10796)
- controller-runtime/ controller-tools: controller-gen/...
- code-generator > create kube api

- [operator-sdk](https://itnext.io/under-the-hood-of-the-operator-sdk-eebc8fdeebbf) [code-generator](https://github.com/kubernetes/code-generator)
- scale, upgrade, or backup

- [metacontroller](https://metacontroller.app/) Lightweight Kubernetes controllers as a service

---

#### kubebuilder


---

#### operator-sdk


---

### operator-sdk devel flow


---

### operator framework

- [到底 Operator Framework 用起來會長得如何?](https://blog.pichuang.com.tw/20190407-how-look-like-operator-framework-in-production/)


---

### Operator Lifycycle Management


---

### kubebuilder vs Operator SDK

- Operator SDK also has support for Ansible and Helm operators, which make it easy to write operators without having to learn Go and if you already have experience with Ansible or Helm
- Operator SDK includes integrations with the Operator Lifecycle Manager (OLM), which is a key component of the Operator Framework that is important to Day 2 cluster operations, like managing a live upgrade of your operator.
- Operator SDK includes a scorecard subcommand that helps you understand if your operator follows best practices.
- Operator SDK includes an e2e testing framework that simplifies testing your operator against an actual cluster.
---
- Kubebuilder includes an envtest package that allows operator developers to run simple tests with a standalone etcd and apiserver.
- Kubebuilder scaffolds a Makefile to assist users in operator tasks (build, test, run, code generation, etc.); Operator SDK is currently using built-in subcommands. Each has pros and cons. The SDK team will likely be migrating to a Makefile-based approach in the future.
- Kubebuilder uses Kustomize to build deployment manifests; Operator SDK uses static files with placeholders.
- Kubebuilder has recently improved its support for admission and CRD conversion webhooks, which has not yet made it into SDK.

---

### Informer list-watch

[Kubernetes Informer 详解](https://www.kubernetes.org.cn/2693.html) / [理解 K8S 的设计精髓之 list-watch](http://wsfdl.com/kubernetes/2019/01/10/list_watch_in_k8s.html) / [理解 K8S 的设计精髓之 List-Watch机制和Informer模块](https://zhuanlan.zhihu.com/p/59660536)


---


---


---

### kubebuilder

 $ go mod init example.easystack.cn # go v1.11
 $ kubebuilder init --domain my.domain

---

#### kubebuilder init

go mod tidy
Running make...
make
which: no controller-gen in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/shawn/.local/bin:/home/shawn/bin)
go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.2.0
go: finding sigs.k8s.io/controller-tools/cmd/controller-gen v0.2.0
go: finding sigs.k8s.io/controller-tools/cmd v0.2.0
/home/shawn/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: Define a resource with:
$ kubebuilder create api

---

config/
  certmanager/  default/  manager/  rbac/  webhook/
Dockerfile
hack
main.go
Makefile
PROJECT

---

main.go

func main () {
...
       mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
                Scheme:             scheme,
                MetricsBindAddress: metricsAddr,
                LeaderElection:     enableLeaderElection,
        })
...
        // +kubebuilder:scaffold:builder
... // mgr.Start(ctrl.SetupSignalHandler())
}

---

#### kubebuilder create api

$ kubebuilder create api --group webapp --version v1 --kind Guestbook
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1/guestbook_types.go
controllers/guestbook_controller.go
Running make...

* api/v1/guestbook_types.go
* controller/guestbook_controller.go

---

#### kubebuilder apply crd

kubectl apply -f config/samples/

make docker-build docker-push IMG=<some-registry>/controller

make deploy IMG=<some-registry>/controller