# ScratchPad: Identify ghost diff during kapp controller reconcilliation
Kapp controller is a package manager compatible with Gitops philosophy. In each reconcilliation cycle, it monitors current state of the resources on cluster and tries to bring it to the desired state if there is any mismatch. It does so with the help of kapp. kapp tries to do a diff of the current live state of the resources with the desired state. Sometimes, even though there is no change, kapp still thinks some resources have changed and tries to redeploy them as part of the reconcilliation. We call them as `ghost` diffs. As package consumers are aware of the package and package install only, it becomes difficult for them to identify what part of the resource configuration is causing these diff.
In this blog, we will see how to identify the resources causing these ghost diffs and also what part of their configuration is participating in it.
## You will need these to start your journey:
* carvel tool set
* Kubernetes cluster(I'm using minikube)
Since this blog will be using `HorizontalPodAutoscaler`, we have to enable `metrics-server` on minikube.
```console
$ minikube addons enable metrics-server
▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
The 'metrics-server' addon is enabled
```
## Install the
For the purpose of this blog, I have already created a package `simple-app-package`. This package is part of the package repository `my-pkg-repo`. If you are interested in how to create package and package repository, I would recommend you to go for this link(https://carvel.dev/kapp-controller/docs/v0.34.0/packaging-tutorial/).
First, we need to install the package repository. To interact with package and package Repository, I will be using `kctrl`
```console
$ kctrl package repository add -r demo-pkg-repo --url docker.io/rohitagg2020/my-pkg-repo:1.0.0
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Waiting for package repository to be added
1:38:45PM: packagerepository/demo-pkg-repo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
1:38:50PM: packagerepository/demo-pkg-repo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded
Succeeded
```
```cat > repo.yml << EOF
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
name: simple-package-repository
spec:
fetch:
imgpkgBundle:
image: docker.io/rohitagg2020/my-pkg-repo:1.0.0
EOF
```
```console
$ kapp deploy -a repo -f repo.yml -y
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Changes
Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
default simple-package-repository PackageRepository - - create - reconcile - -
Op: 1 create, 0 delete, 0 update, 0 noop, 0 exists
Wait to: 1 reconcile, 0 delete, 0 noop
1:10:24PM: ---- applying 1 changes [0/1 done] ----
1:10:24PM: create packagerepository/simple-package-repository (packaging.carvel.dev/v1alpha1) namespace: default
1:10:24PM: ---- waiting on 1 changes [0/1 done] ----
1:10:24PM: ongoing: reconcile packagerepository/simple-package-repository (packaging.carvel.dev/v1alpha1) namespace: default
1:10:24PM: ^ Waiting for generation 1 to be observed
1:10:25PM: ongoing: reconcile packagerepository/simple-package-repository (packaging.carvel.dev/v1alpha1) namespace: default
1:10:25PM: ^ Reconciling
1:10:30PM: ok: reconcile packagerepository/simple-package-repository (packaging.carvel.dev/v1alpha1) namespace: default
1:10:30PM: ---- applying complete [1/1 done] ----
1:10:30PM: ---- waiting complete [1/1 done] ----
Succeeded
```
Once the package repository is installed, we can check the list of available packages.
```console
$ kctrl package available list --summary=false
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Available packages in namespace 'default'
Name Version Released at
simple-app.corp.com 1.0.0 0001-01-01 00:00:00 +0000 UTC
Succeeded
$ kubectl get packages
NAME PACKAGEMETADATA NAME VERSION AGE
simple-app.corp.com.1.0.0 simple-app.corp.com 1.0.0 5m11s
```
Let's install the package.
```console
$ kctrl package install -i pkg-demo -p simple-app.corp.com --version 1.0.0
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Creating service account 'pkg-demo-default-sa'
Creating cluster admin role 'pkg-demo-default-cluster-role'
Creating cluster role binding 'pkg-demo-default-cluster-rolebinding'
Creating package install resource
Waiting for PackageInstall reconciliation for 'pkg-demo'
1:40:20PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
1:40:30PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded
Succeeded
$ kapp deploy -a pkg-demo -f pkginstall.yml -y
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Changes
Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
default pkg-demo PackageInstall - - create - reconcile - -
^ pkg-demo-values Secret - - create - reconcile - -
Op: 2 create, 0 delete, 0 update, 0 noop, 0 exists
Wait to: 2 reconcile, 0 delete, 0 noop
1:10:56PM: ---- applying 1 changes [0/2 done] ----
1:10:57PM: create secret/pkg-demo-values (v1) namespace: default
1:10:57PM: ---- waiting on 1 changes [0/2 done] ----
1:10:57PM: ok: reconcile secret/pkg-demo-values (v1) namespace: default
1:10:57PM: ---- applying 1 changes [1/2 done] ----
1:10:57PM: create packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default
1:10:57PM: ---- waiting on 1 changes [1/2 done] ----
1:10:57PM: ongoing: reconcile packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default
1:10:57PM: ^ Waiting for generation 1 to be observed
1:10:58PM: ongoing: reconcile packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default
1:10:58PM: ^ Reconciling
1:11:08PM: ok: reconcile packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default
1:11:08PM: ---- applying complete [2/2 done] ----
1:11:08PM: ---- waiting complete [2/2 done] ----
Succeeded
```
After the deploy has finished, kapp-controller will have installed the package in the cluster. We can verify this by checking the pods to see that we have a workload pod running. The output should show two running pods which is part of simple-app:
```console
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
simple-app-8648457765-8jtpq 1/1 Running 0 56s
simple-app-8648457765-p5lzp 1/1 Running 0 56s
```
In our package, we have set `syncPeriod` to 10 min. This means after every 10 min, kapp controller will try to reconcile the package. And if there are any ghost diff's generated, we will get to know by looking at the configmaps. To identify the configmaps related to an installed package, we have to look at the configmap with Installed Package Prefix. Lets check the configmaps.
```console
$ kubectl get configmaps | grep pkg-demo
pkg-demo-ctrl 1 1m40s
pkg-demo-ctrl-change-ndr7w 1 1m40s
```
Lets do the same after 10 min of package installation, so that reconcilliation cycle would have run once.
```console
$ kubectl get configmaps | grep pkg-demo
pkg-demo-ctrl 1 12m
pkg-demo-ctrl-change-ndr7w 1 12m
pkg-demo-ctrl-change-t7zgc 1 56s
```
We can see that one more configmap is generated. If I check the configmap content, I can see that there has been one update, but I dont know which resource got updated and what is causing this update.
```console
$ kubectl get configmap pkg-demo-ctrl-change-t7zgc -oyaml
apiVersion: v1
data:
spec: '{"startedAt":"2022-04-14T08:21:28.264641525Z","finishedAt":"2022-04-14T08:21:32.42972746Z","successful":true,"description":"update:
Op: 0 create, 0 delete, 1 update, 0 noop, 0 exists / Wait to: 1 reconcile, 0 delete,
0 noop","namespaces":["default"]}'
kind: ConfigMap
```
As a package consumer, I can see that there are ghost diff's appearing. To identify what is causing them, we will make a copy of the package. We will modify the deploy section of the package. It will help us to get the configuration applied by `kapp`. Let's start:
```console
$ kubectl get pkg simple-app.corp.com.1.0.0 -oyaml > copy-simple-app-package.yaml
```
Open copy-simple-app-package.yaml. Remove labels starting with `kapp`. Add below snippet to the kapp section. Setting the `diff-changes` to true will enable the kapp to show changes.
```
- kapp:
rawOptions:
- --diff-changes=true
```
I would recommend not to tinker with the original package. Hence, lets change the pacakge version at all places. Lets change it from 1.0.0 to 2.0.0. Specifically, change the `spec.Version` and `metadata.labels.name`. Now, apply this package in the cluster so that it will be available for install.
```console
$ kubectl apply -f copy-simple-app-package.yaml
package.data.packaging.carvel.dev/simple-app.corp.com.2.0.0 created
```
Now, if we will see list of available packages, we can see our copied package as well.
```console
$ kctrl package available list --summary=false
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Available packages in namespace 'default'
Name Version Released at
simple-app.corp.com 1.0.0 0001-01-01 00:00:00 +0000 UTC
simple-app.corp.com 2.0.0 0001-01-01 00:00:00 +0000 UTC
Succeeded
```
Let's uninstall the previous package and deploy locally created package.
```console
$ kctrl package installed delete -i pkg-demo -y
Delete package install 'pkg-demo' from namespace 'default'
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Deleting package install 'pkg-demo' from namespace 'default'
Waiting for deletion of package install 'pkg-demo' from namespace 'default'
2:08:42PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Deleting
2:08:43PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: DeletionSucceeded
Deleting 'ClusterRole': pkg-demo-default-cluster-role
Deleting 'ClusterRoleBinding': pkg-demo-default-cluster-rolebinding
Deleting 'ServiceAccount': pkg-demo-default-sa
Succeeded
$ kctrl package install -i pkg-demo -p simple-app.corp.com --version 2.0.0
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Creating service account 'pkg-demo-default-sa'
Creating cluster admin role 'pkg-demo-default-cluster-role'
Creating cluster role binding 'pkg-demo-default-cluster-rolebinding'
Creating package install resource
Waiting for PackageInstall reconciliation for 'pkg-demo'
2:12:19PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
2:12:28PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded
Succeeded
```
After the package is deployed successfully, lets see how the initial configuration of the resources deployed by package looks like. We can get that by describing App(link to the app section) linked to the package. Similar to configmap, package creates `App` with same name as its own name.As the output is long, I have added only small snippet.
```console
$ kubectl describe app pkg-demo
@@ create deployment/simple-app (apps/v1) namespace: default @@
0 + apiVersion: apps/v1
1 + kind: Deployment
2 + metadata:
3 + annotations:
4 + kbld.k14s.io/images: |
5 + - origins:
6 + - preresolved:
7 + url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
8 + url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
9 + labels:
10 + kapp.k14s.io/app: "1649925743708215040"
11 + kapp.k14s.io/association: v1.22a4cbb25c518f776737777e8407b8d9
12 + name: simple-app
13 + namespace: default
14 + spec:
15 + progressDeadlineSeconds: 600
16 + replicas: 2
17 + revisionHistoryLimit:
```
Let the reconcilliation loop run once. After the reconcilliation loop is run, we will see that another configmap has been generated. Now, if we will run `app describe` again, we will see the exact diff.
```console
```
This is how, as a package consumer, you can discover the reason for phantom diff's and take appropriate action. In this case, one can add the rebase rule(https://carvel.dev/kapp/docs/v0.46.0/hpa-deployment-rebase/#docs) to remove the phantom diffs.
Sign in with Wallet
Connect another wallet