# Deploying State Improvements to Kubernetes ## Steps **Determine a docker image containing the changes** Check the prysm docker images pipeline on buildkite [here](https://buildkite.com/prysmatic-labs/prysm-docker-images/builds/5254#6f4bcd62-cdff-4f19-9d15-4bd2d54b7584) as an example. The very bottom of this example lists the following logs: ``` 2021/08/09 16:43:25 Successfully pushed Docker image to index.docker.io/prysmaticlabs/prysm-beacon-chain:HEAD-387873 2021/08/09 16:43:26 Successfully pushed Docker image to index.docker.io/prysmaticlabs/prysm-beacon-chain:latest 2021/08/09 16:43:26 Successfully pushed Docker image to gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873 2021/08/09 16:43:26 Successfully pushed Docker image to gcr.io/prysmaticlabs/prysm/beacon-chain:latest ``` So we'll take this docker tag `gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873`. **Generate a kubernetes manifest using helm for prater** Clone `https://github.com/prysmaticlabs/manifests` and navigate to `cd development/beacon-chain-prater`. Generate the kubernetes manifest with [helm](https://helm.sh/docs/intro/install/). Run ``` helm template beacon-chain -f beacon-chain/values.yaml --dry-run --set=version=prod > /tmp/my-manifest.yaml ``` **Edit the generated manifest to include the feature flag and labels** You will have a generated file in `/tmp/my-manifest.yaml`. We edit the following things in the generated file: 1. Edit replicas to your choosing, perhaps 3 2. Edit all mentions of the word `prod` to a custom one, such as `optimized-state` 3. Remove eth2-stats as it is not needed 4. Edit the container image to the one needed 5. Add any custom feature flags you desire Here is the final diff between the generated and edited yaml ``` 6c6 < name: beacon-chain-prod --- > name: beacon-chain-optimized-state 13c13 < serviceName: beacon-chain-prod --- > serviceName: beacon-chain-optimized-state 20,21c20,21 < version: prod < scope: beacon-chain-prod --- > version: optimized-state > scope: beacon-chain-optimized-state 29,30c29,30 < version: prod < scope: beacon-chain-prod --- > version: optimized-state > scope: beacon-chain-optimized-state 63c63 < image: gcr.io/prysmaticlabs/prysm/beacon-chain --- > image: gcr.io/prysmaticlabs/prysm/beacon-chain:HEAD-387873 82a83 > - --enable-historical-state-representation 124,135d124 < # TODO: Add startupProbe with kubernetes 1.16. < - name: eth2stats < image: alethio/eth2stats-client@sha256:c765cb4fad18146d40feaeba7f6f3059397f7e8df107b079da0a337d7bc2544a < args: < - run < - --eth2stats.node-name=prylabs-$(POD_NAME) < - --eth2stats.addr=grpc.prater.eth2.wtf:8080 < - --eth2stats.tls=false < - --beacon.type=prysm < - --beacon.addr=$(POD_IP):4000 < - --beacon.metrics-addr=$(POD_IP):9090/metrics < - --data.folder=/data ``` **Deploy the new nodes** Simply apply the manifest with ``` kubectl apply -f /tmp/my-manifest.yaml ``` **Wait on the prater namespace** Watch for new pods in the beacon-chain-prater namespace ``` watch kubectl get pods -n beacon-chain-prater ``` You will see something like this: ``` Every 2.0s: kubectl get pods -n beacon-chain-prater AkiraKurusu.local: Mon Aug 9 11:26:08 2021 NAME READY STATUS RESTARTS AGE beacon-chain-baseline-0 2/2 Running 0 20h beacon-chain-optimized-state-0 0/2 Init:0/2 0 6s beacon-chain-optimized-state-1 0/2 Init:0/2 0 6s beacon-chain-optimized-state-2 0/2 Pending 0 6s beacon-chain-prod-0 2/2 Running 0 114m beacon-chain-prod-1 2/2 Running 0 17h beacon-chain-prod-2 2/2 Running 1 20h beacon-chain-prod-3 2/2 Running 0 114m beacon-chain-prod-4 2/2 Running 1 17h beacon-chain-slasher-0 2/2 Running 0 20h beacon-chain-slasher-1 2/2 Running 0 114m beacon-chain-slasher-2 2/2 Running 0 17h validator-0 2/2 Running 0 17h validator-1 2/2 Running 0 20h validator-2 2/2 Running 0 114m validator-3 2/2 Running 0 7d1h validator-4 2/2 Running 0 21h validator-5 2/2 Running 0 17h validator-6 2/2 Running 0 6d1h validator-7 2/2 Running 0 21h validator-8 2/2 Running 0 113m validator-9 2/2 Running 0 17h ``` Your new deployment should show up in there shortly. **Check for logs** Inspect the logs of your new pods to see if things are as expected. ``` kubectl logs beacon-chain-optimized-state-0 -c beacon-chain -n beacon-chain-prater -f ``` ``` {"addr":"http://127.0.0.1:6060/debug/pprof","message":"Starting pprof server","severity":"INFO"} {"message":"Starting Jaeger exporter endpoint at address = http://jaeger-collector.observability.svc.cluster.local:14268/api/traces","prefix":"tracing","severity":"INFO"} {"message":"Running on the Prater Testnet","prefix":"flags","severity":"WARNING"} {"enable-historical-state-representation":"Enables the beacon chain to save historical states in a space efficient manner. (Warning): Once enabled, this feature migrates your database in to a new schema and there is no going back. At worst, your entire database might get corrupted.","message":"Enabled feature flag","prefix":"flags","severity":"WARNING"} {"database-path":"/data/beaconchaindata","message":"Checking DB","prefix":"node","severity":"INFO"} {"message":"Performing a one-time migration to a more efficient database schema for state. It will take few minutes","prefix":"db","severity":"INFO"} {"message":"total keys = 0","prefix":"db","severity":"INFO"} {"message":"migration done for bucket state.","prefix":"db","severity":"INFO" ``` **Prepare grafana dashboard for comparisons** You can add metrics to grafana for your new deployment by filtering in the beacon-chain-prater namespace and your custom labels when you edited the manifest. **Clean-up: Deleting the deployment later on** Delete the stateful set you deployed with ``` kubectl delete sts/beacon-chain-optimized-state -n beacon-chain-prater ``` Pods with a stateful set will allocate a persistent volume claim which should be deleted when you are no longer using the resource. To list the persistent volume claims (pvc) in the namespace, do ``` kubectl get pvc -n beacon-chain-prater ``` and delete each that you allocated such as ``` kubectl delete pvc beacondb-beacon-chain-optimized-state-0 -n beacon-chain-prater ... ```