---
# System prepended metadata

title: Identify ghost diff during Kapp Controller reconciliation

---

# Identify ghost diff during Kapp Controller reconciliation

Kapp controller, a Package manager, is compatible with Gitops philosophy. It is continuously ensuring that the cluster is or converging towards the desired state. It does so by running the reconciliation loop after every `SyncPeriod` duration. In each reconciliation cycle, it monitors the current state of the resources on the cluster and tries to bring it to the desired state if there is any mismatch. It does so with the help of Kapp. Kapp tries to do a diff of the current live state of the resources with the desired state. Sometimes, even though there is no change in the desired state by the user explicitly, Kapp still thinks some resources have changed and tries to redeploy them as part of the reconciliation. We call them `ghost` diffs. 

Since Package consumers are aware of the `Package` configuration only, it becomes difficult for them to identify which part of the underlying resource configuration is causing these diffs.

In this blog, we will see how to identify the resources causing these ghost diffs and also what part of their configuration is participating in it.


## Prerequisites

* carvel tool set
* Kubernetes cluster(I'm using minikube)

Since this blog will be using `HorizontalPodAutoscaler`, we have to enable `metrics-server` on minikube.

```console
$ minikube addons enable metrics-server
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
  The 'metrics-server' addon is enabled
```

I will be using [kctrl](https://carvel.dev/kapp-controller/docs/v0.34.0/install/#installing-kapp-controller-cli-kctrl) to interact with `kapp-controller` resources. 

## Install the Package

For the purpose of this blog, I have already created a Carvel package `simple-app-package`. This Package is part of the package repository `my-pkg-repo`.  If you are interested in how to create package and package repository, I would recommend to go over [packaging-tutorial](https://carvel.dev/kapp-controller/docs/v0.34.0/packaging-tutorial/).

First, we need to install the Package Repository. 

```console
$ kctrl package repository add -r demo-pkg-repo --url docker.io/rohitagg2020/my-pkg-repo:1.0.0
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)

Waiting for package repository to be added

1:38:45PM: packagerepository/demo-pkg-repo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
1:38:50PM: packagerepository/demo-pkg-repo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded

Succeeded
```

Once the Package Repository is installed, we can check the list of available Packages.
```console
$ kctrl package available list --summary=false
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)

Available packages in namespace 'default'

Name                 Version  Released at
simple-app.corp.com  1.0.0    0001-01-01 00:00:00 +0000 UTC

Succeeded
```

Let's install the Package.

```console
$ kctrl package install -i pkg-demo -p simple-app.corp.com --version 1.0.0
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)

Creating service account 'pkg-demo-default-sa'
Creating cluster admin role 'pkg-demo-default-cluster-role'
Creating cluster role binding 'pkg-demo-default-cluster-rolebinding'
Creating package install resource
Waiting for PackageInstall reconciliation for 'pkg-demo'

1:40:20PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
1:40:30PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded

Succeeded
```

After the deploy has finished, kapp-controller would have installed the Package in the cluster. We can verify this by checking the pods to see that we have a workload pod running. The output should show two running pods which is part of simple-app:

```console
$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
simple-app-8648457765-8jtpq   1/1     Running   0          56s
simple-app-8648457765-p5lzp   1/1     Running   0          56s
```

## Identify ghost diffs appear or not

In our package, we have set `syncPeriod` to 10 min. This means after every 10 min, Kapp controller will try to reconcile the package. Kapp creates a configmap every time it sees that there are some resources that needs to be redeployed. Thus, if we see new configmaps appearing, it means ghost diffs are being generated. To identify the configmaps related to an installed package, we have to look at the configmap with Installed Package Prefix. Let's check the configmaps.

```console 
$ kubectl get configmaps | grep pkg-demo
pkg-demo-ctrl                     1      1m40s
pkg-demo-ctrl-change-ndr7w        1      1m40s
```

Lets do the same after 10 min of package installation, so that reconcilliation cycle would have run once. 

```console
$ kubectl get configmaps | grep pkg-demo
pkg-demo-ctrl                     1      12m
pkg-demo-ctrl-change-ndr7w        1      12m
pkg-demo-ctrl-change-t7zgc        1      56s
```
We can see that one more configmap is generated. If I check the configmap content, I can see that there has been one update, but I don't know which resource got updated and what is causing this update. 

```console
$ kubectl get configmap pkg-demo-ctrl-change-t7zgc -oyaml
apiVersion: v1
data:
  spec: '{"startedAt":"2022-04-14T08:21:28.264641525Z","finishedAt":"2022-04-14T08:21:32.42972746Z","successful":true,"description":"update:
    Op: 0 create, 0 delete, 1 update, 0 noop, 0 exists / Wait to: 1 reconcile, 0 delete,
    0 noop","namespaces":["default"]}'
kind: ConfigMap
...
```

As a Package consumer, I can see that there are ghost diff's appearing. 

## Identify actual configuration causing the diffs

To identify what is causing them, we will make a copy of the package. We will modify the deploy section of the Package. It will help us to get the configuration applied by `Kapp`. Let's start:

```console
$ kubectl get pkg simple-app.corp.com.1.0.0 -oyaml > copy-simple-app-package.yaml
```

Open copy-simple-app-package.yaml. Remove labels starting with `Kapp`. Add the below snippet to the Kapp section. Setting the `diff-changes` to true will enable the `Kapp` to show changes.
```
...
- kapp:
    rawOptions:
    - --diff-changes=true
...
```

I would recommend not to tinker with the original package. Hence, let's change the Package version. Lets change it from 1.0.0 to 2.0.0. Specifically, change the `spec.Version` and `metadata.labels.name`. Now, apply this Package in the cluster so that it will be available for install.

```console 
$ kubectl apply -f copy-simple-app-package.yaml
package.data.packaging.carvel.dev/simple-app.corp.com.2.0.0 created
```

Now, if we will see list of available Packages, we can see our locally created Package as well. 

```console
$ kctrl package available list --summary=false
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)

Available packages in namespace 'default'

Name                 Version  Released at
simple-app.corp.com  1.0.0    0001-01-01 00:00:00 +0000 UTC
simple-app.corp.com  2.0.0    0001-01-01 00:00:00 +0000 UTC

Succeeded
```

Let's uninstall the previous package and deploy locally created package.

```console
$ kctrl package installed delete -i pkg-demo -y

Delete package install 'pkg-demo' from namespace 'default'
Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Deleting package install 'pkg-demo' from namespace 'default'
Waiting for deletion of package install 'pkg-demo' from namespace 'default'

2:08:42PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Deleting
2:08:43PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: DeletionSucceeded

Deleting 'ClusterRole': pkg-demo-default-cluster-role
Deleting 'ClusterRoleBinding': pkg-demo-default-cluster-rolebinding
Deleting 'ServiceAccount': pkg-demo-default-sa

Succeeded

$ kctrl package install -i pkg-demo -p simple-app.corp.com --version 2.0.0

Target cluster 'https://192.168.64.82:8443' (nodes: minikube)
Creating service account 'pkg-demo-default-sa'
Creating cluster admin role 'pkg-demo-default-cluster-role'
Creating cluster role binding 'pkg-demo-default-cluster-rolebinding'
Creating package install resource
Waiting for PackageInstall reconciliation for 'pkg-demo'

2:12:19PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: Reconciling
2:12:28PM: packageinstall/pkg-demo (packaging.carvel.dev/v1alpha1) namespace: default: ReconcileSucceeded

Succeeded
```
After the Package is deployed successfully, let's see what the initial configuration of the resources looks like. We can get that by describing App(link to the app section) linked to the Package. Similar to configmap, the package creates `App` with the same name as its name. As the output is long, I have added only a small snippet.

```console
$ kubectl describe app pkg-demo
...
@@ create deployment/simple-app (apps/v1) namespace: default @@
      0 + apiVersion: apps/v1
      1 + kind: Deployment
      2 + metadata:
      3 +   annotations:
      4 +     kbld.k14s.io/images: |
      5 +       - origins:
      6 +         - preresolved:
      7 +             url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
      8 +         url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
      9 +   labels:
     10 +     kapp.k14s.io/app: "1649925743708215040"
     11 +     kapp.k14s.io/association: v1.22a4cbb25c518f776737777e8407b8d9
     12 +   name: simple-app
     13 +   namespace: default
     14 + spec:
     15 +   progressDeadlineSeconds: 600
     16 +   replicas: 2
     17 +   revisionHistoryLimit:
...
```

You will see that it is creating a deployment, service and HPA.

Let the reconcilliation loop run once. After the reconcilliation loop is run, we will see that another configmap has been generated. Now, if we will run `app describe` again, we will see the exact diff.

```console
$ kubectl get configmaps | grep pkg-demo
pkg-demo-ctrl                     1      12m
pkg-demo-ctrl-change-4vpdq        1      2m24s
pkg-demo-ctrl-change-rnjxf        1      12m

$ kubectl describe app pkg-demo
...
Status:
  Conditions:
    Status:                         True
    Type:                           ReconcileSucceeded
  Consecutive Reconcile Successes:  2
  Deploy:
    Exit Code:   0
    Finished:    true
    Started At:  2022-04-14T08:52:35Z
    Stdout:      Target cluster 'https://10.96.0.1:443' (nodes: minikube)
@@ update deployment/simple-app (apps/v1) namespace: default @@
  ...
125,125     progressDeadlineSeconds: 600
126     -   replicas: 1
    126 +   replicas: 2
127,127     revisionHistoryLimit: 10
128,128     selector:
Changes
Namespace  Name        Kind        Conds.  Age  Op      Op st.  Wait to    Rs  Ri
default    simple-app  Deployment  2/2 t   10m  update  -       reconcile  ok  -
Op:      0 create, 0 delete, 1 update, 0 noop, 0 exists
Wait to: 1 reconcile, 0 delete, 0 noop
...
```

Here, as we can see, the change in the number of replicas has resulted in the creation of ghost diffs. This is because HPA has reduced the no. of Replicas since there is no load on the server.

This is how a Package consumer can discover the reason for ghost diffs and take appropriate action. In this case, adding a [rebase rule](https://carvel.dev/kapp/docs/v0.46.0/hpa-deployment-rebase/#docs) will remove the ghost diffs.

