# Deploying State Improvements to Kubernetes
## Steps
**Determine a docker image containing the changes**
Check the prysm docker images pipeline on buildkite [here](https://buildkite.com/prysmatic-labs/prysm-docker-images/builds/5254#6f4bcd62-cdff-4f19-9d15-4bd2d54b7584) as an example. The very bottom of this example lists the following logs:
```
2021/08/09 16:43:25 Successfully pushed Docker image to index.docker.io/prysmaticlabs/prysm-beacon-chain:HEAD-387873
2021/08/09 16:43:26 Successfully pushed Docker image to index.docker.io/prysmaticlabs/prysm-beacon-chain:latest
2021/08/09 16:43:26 Successfully pushed Docker image to gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873
2021/08/09 16:43:26 Successfully pushed Docker image to gcr.io/prysmaticlabs/prysm/beacon-chain:latest
```
So we'll take this docker tag `gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873`.
**Generate a kubernetes manifest using helm for prater**
Clone `https://github.com/prysmaticlabs/manifests` and navigate to `cd development/beacon-chain-prater`.
Generate the kubernetes manifest with [helm](https://helm.sh/docs/intro/install/).
Run
```
helm template beacon-chain -f beacon-chain/values.yaml --dry-run --set=version=prod > /tmp/my-manifest.yaml
```
**Edit the generated manifest to include the feature flag and labels**
You will have a generated file in `/tmp/my-manifest.yaml`. We edit the following things in the generated file:
1. Edit replicas to your choosing, perhaps 3
2. Edit all mentions of the word `prod` to a custom one, such as `optimized-state`
3. Remove eth2-stats as it is not needed
4. Edit the container image to the one needed
5. Add any custom feature flags you desire
Here is the final diff between the generated and edited yaml
```
6c6
< name: beacon-chain-prod
---
> name: beacon-chain-optimized-state
13c13
< serviceName: beacon-chain-prod
---
> serviceName: beacon-chain-optimized-state
20,21c20,21
< version: prod
< scope: beacon-chain-prod
---
> version: optimized-state
> scope: beacon-chain-optimized-state
29,30c29,30
< version: prod
< scope: beacon-chain-prod
---
> version: optimized-state
> scope: beacon-chain-optimized-state
63c63
< image: gcr.io/prysmaticlabs/prysm/beacon-chain
---
> image: gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873
82a83
> - --enable-historical-state-representation
124,135d124
< # TODO: Add startupProbe with kubernetes 1.16.
< - name: eth2stats
< image: alethio/eth2stats-client@sha256:c765cb4fad18146d40feaeba7f6f3059397f7e8df107b079da0a337d7bc2544a
< args:
< - run
< - --eth2stats.node-name=prylabs-$(POD_NAME)
< - --eth2stats.addr=grpc.prater.eth2.wtf:8080
< - --eth2stats.tls=false
< - --beacon.type=prysm
< - --beacon.addr=$(POD_IP):4000
< - --beacon.metrics-addr=$(POD_IP):9090/metrics
< - --data.folder=/data
```
**Deploy the new nodes**
Simply apply the manifest with
```
kubectl apply -f /tmp/my-manifest.yaml
```
**Wait on the prater namespace**
Watch for new pods in the beacon-chain-prater namespace
```
watch kubectl get pods -n beacon-chain-prater
```
You will see something like this:
```
Every 2.0s: kubectl get pods -n beacon-chain-prater AkiraKurusu.local: Mon Aug 9 11:26:08 2021
NAME READY STATUS RESTARTS AGE
beacon-chain-baseline-0 2/2 Running 0 20h
beacon-chain-optimized-state-0 0/2 Init:0/2 0 6s
beacon-chain-optimized-state-1 0/2 Init:0/2 0 6s
beacon-chain-optimized-state-2 0/2 Pending 0 6s
beacon-chain-prod-0 2/2 Running 0 114m
beacon-chain-prod-1 2/2 Running 0 17h
beacon-chain-prod-2 2/2 Running 1 20h
beacon-chain-prod-3 2/2 Running 0 114m
beacon-chain-prod-4 2/2 Running 1 17h
beacon-chain-slasher-0 2/2 Running 0 20h
beacon-chain-slasher-1 2/2 Running 0 114m
beacon-chain-slasher-2 2/2 Running 0 17h
validator-0 2/2 Running 0 17h
validator-1 2/2 Running 0 20h
validator-2 2/2 Running 0 114m
validator-3 2/2 Running 0 7d1h
validator-4 2/2 Running 0 21h
validator-5 2/2 Running 0 17h
validator-6 2/2 Running 0 6d1h
validator-7 2/2 Running 0 21h
validator-8 2/2 Running 0 113m
validator-9 2/2 Running 0 17h
```
Your new deployment should show up in there shortly.
**Check for logs**
Inspect the logs of your new pods to see if things are as expected.
```
kubectl logs beacon-chain-optimized-state-0 -c beacon-chain -n beacon-chain-prater -f
```
```
{"addr":"http://127.0.0.1:6060/debug/pprof","message":"Starting pprof server","severity":"INFO"}
{"message":"Starting Jaeger exporter endpoint at address = http://jaeger-collector.observability.svc.cluster.local:14268/api/traces","prefix":"tracing","severity":"INFO"}
{"message":"Running on the Prater Testnet","prefix":"flags","severity":"WARNING"}
{"enable-historical-state-representation":"Enables the beacon chain to save historical states in a space efficient manner. (Warning): Once enabled, this feature migrates your database in to a new schema and there is no going back. At worst, your entire database might get corrupted.","message":"Enabled feature flag","prefix":"flags","severity":"WARNING"}
{"database-path":"/data/beaconchaindata","message":"Checking DB","prefix":"node","severity":"INFO"}
{"message":"Performing a one-time migration to a more efficient database schema for state. It will take few minutes","prefix":"db","severity":"INFO"}
{"message":"total keys = 0","prefix":"db","severity":"INFO"}
{"message":"migration done for bucket state.","prefix":"db","severity":"INFO"
```
**Prepare grafana dashboard for comparisons**
You can add metrics to grafana for your new deployment by filtering in the beacon-chain-prater namespace and your custom labels when you edited the manifest.
**Clean-up: Deleting the deployment later on**
Delete the stateful set you deployed with
```
kubectl delete sts/beacon-chain-optimized-state -n beacon-chain-prater
```
Pods with a stateful set will allocate a persistent volume claim which should be deleted when you are no longer using the resource. To list the persistent volume claims (pvc) in the namespace, do
```
kubectl get pvc -n beacon-chain-prater
```
and delete each that you allocated such as
```
kubectl delete pvc beacondb-beacon-chain-optimized-state-0 -n beacon-chain-prater
...
```