# ANT Chaos Document Guide ## Table of Contents * [**Introduction**](#Introduction-to-Chaos-Engineering) * [**Architecture**](#Architecture) * [**Installation**](#Installation) * [**Usage**](#Usage) * [**FAQ**](#FAQ) * [**Trouble Shooting Guide**](#Trouble-Shooting-Guide) * [**Developer Guide**](#Developer-Guide) ## Introduction to Chaos Engineering Chaos engineering is a disciplined approach to identifying failures before they become outages. It lets you compare what you ‘think will happen’ during an outage to what ‘actually happens’ in your systems. This is how [principlesofchaos.org](https://principlesofchaos.org/) defines chaos engineering: *“Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.”* ### Why do we need Chaos Engineering? The chaos engineering helps make the system more resilient and reliable. It lets you identify the weakness of the system before it becomes a major outage. This helps in taking a step back and redesign our system. ### How do we do Chaos Engineering? * Start by defining the steady state of the system which can later be used for comparing it with the status of the system under chaos. * Introduce failures in the system * Test the functionality of the system while the system is under chaos and stress * Compare the results of steady state and the system under chaos ## Pre requisites: * Running Kubernetes cluster(version > 1.15) with application running * Access token with privileges enough to install few helm charts containing, deployments, CRDs, CRs, SA, role, role bindings, Namespace * The Environment under test should have network connectivity to artifactory * ANT portal with latest version of code. * Kubernetes details added in the appropriate CNF ## Architecture ANT Chaos is constructed out of four parts * Core - Core consists of Argo Workflow and LitmusChaos * Chaos Experiments - Chaos experiments contain the actual chaos details like pod-delete, packet-loss. * Chaos Workflow Templates - Argo workflow templates that deploys the Chaos expermiments * ANT Test Scripts - ANT Test Scripts contains the logic of executing different chaos based on the user inputs   ## Installation ### Deploy Cluster Components You can find the installation steps at: 1. **Manual procedure:** 1. Create the aqua namespace ``` kubectl create ns aqua ``` 2. Label K8s nodes Label the nodes in which you want to install the argo,litmus and other helm charts. ``` kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane,master 48m v1.20.2 kubectl label node kind-control-plane ucp-control-plane=enabled ``` 3. Deploy argo Helm Chart ``` wget https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/argo-2bef8e8b3a62.6dd39da6ad7f.tgz tar -xvf argo-2bef8e8b3a62.6dd39da6ad7f.tgz helm2 repo add nc-charts https://artifacts-nc.auk3.cci.att.com/artifactory/helm-charts/ helm2 dependency update ./argo helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/argo-2bef8e8b3a62.6dd39da6ad7f.tgz -n argo --namespace=aqua --wait ``` 4. Deploy litmus Helm Chart ``` helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus --namespace=aqua --wait ``` 5. Deploy Litmus Chaos expermiments Helm Chart ``` helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/chaosexperiments-2bef8e8b3a62.6dd39da6ad7f.tgz -n chaosexperiments \ --namespace=aqua \ --set experiments.pod_delete.enabled=true,experiments.network_loss.enabled=true,experiments.network_latency.enabled=true \ --wait ``` 6. Deploy Argo workflow templates Helm Chart * if the CRI is Containerd ``` helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-argo-workflow-templates-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus-argo-workflow-templates \ --namespace=aqua \ --set workflowTemplates.pod_delete.enabled=true,workflowTemplates.network_loss.enabled=true,workflowTemplates.network_latency.enabled=true \ --wait ``` * if the CRI is docker ``` helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-argo-workflow-templates-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus-argo-workflow-templates \ --namespace=aqua \ --set workflowTemplates.pod_delete.enabled=true,workflowTemplates.network_loss.enabled=true,workflowTemplates.network_latency.enabled=true \ --set env.container_runtime=docker,env.socket_path=/var/run/docker.sock \ --wait ``` 7. Create ClusterRole Binding for ANT to fetch Pods/Labels dynamically ``` kubectl create clusterrolebinding aqua-view-binding --clusterrole=view --serviceaccount=aqua:chaos-manager ``` 2. **Automated procedure** **Prerequisites:** Make sure the target cluster control-plane node has helm2 and a tiller pod running **Installation Steps** Below helm charts will be deployed as part of this process * argo * litmus * chaosexperiments * litmus-argo-workflow-templates </br> 1. Run below command to generate the sample container_config.env in your current working directory: ``` $ docker run --net=host --rm \ docker-nc.zc1.cti.att.com/nc/nc-ant-test- cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090 \ cat /tmp/scripts/container_config.env >./container_config.env ``` 2. container_config.env has below-highlighted content, modify the container_config.env as per your requirement and save it ``` #Specify the target namespace to deploy the helm charts NAMESPACE=aqua #Specify the target control-plane node name, where it will add a node label "ucp-control-plane=enabled" CONTROL_PLANE_NODE_NAME=kind-control-plane #Specify the litmus experiments true/false based on the experiments you want to install POD_DELETE=true POD_NETWORK_LATENCY=true POD_NETWORK_LOSS=true POD_MEMORY_HOG=true POD_CPU_HOG=true #specify the container runtime ex: docker/containerd CONTAINER_RUNTIME=containerd #specify the respective socket_path for the above-specified container runtime SOCKET_PATH=/run/containerd/containerd.soc ``` 3. Run the below docker container to perform the installation: ``` IMAGE_NAME== docker-nc.zc1.cti.att.com/nc/nc-ant-test-cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090 $ docker run --net=host --rm \ -v <path_to_admin.conf or config>:/tmp/scripts/admin.conf \ # In the target cluster look for /etc/kubernetes/admin.conf or $HOME/.kube/config and mount to /tmp/scripts/admin.conf in the container --env-file=./container_config.env \ # The container_config.env from your present working directory $IMAGE_NAME Example: docker run --net=host --rm -v $HOME/.kube/config:/tmp/scripts/admin.conf --env-file=./container_config.env docker-nc.zc1.cti.att.com/nc/nc-ant-test-cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090 ``` #### Verify the installation: Verify if the ChaosOperator is running ``` $ kubectl get pods -n aqua | grep litmus litmus-5cbc65959f-zfqrx 1/1 Running 0 10d ``` Verify if chaos experiments are installed ``` $ kubectl get chaosexperiments -n aqua NAME AGE pod-delete 10d pod-network-latency 10d pod-network-loss. 10d ``` Verify if litmus-argo-workflowtemplate are installed ``` $ kubectl get workflowtemplates -n aqua NAME AGE network-latency-chaos-workflow 11s network-packet-loss-chaos-workflow 11s pod-chaos-workflow 11s ``` ### Deploy Chaos Testscripts to ANT Portal * Login to ANT Portal * Navigate to Test Script Inventory page by clicking `Inventory` --> `Test Script Inventory` from the top menu Bar. * Click on `Upload ZIP File` button to upload the testscript * Once upload is success, then Click on `Sync` button to sync the uploaded script to ATE(s)  ## Usage * Click on APPS and then select Test Management from the top menu bar * Select the appropriate DUT. * Select the type of chaos you want to run(Available options: `pakcet loss`, `network latency`, and `pod chaos`). * Provide the information regarding the target application. * Pakcet loss: This helps inject packet loss into application pod. This type of chaos would need following information to run: * Pod Label: Label of the pods you want to target. * Target Namespace: The namespace of the application you wish to target. * CNF Name: The name of the CNF you want to target. * Chaos Duration: The time in seconds for how long you want to run the chaos. * Percentage: The percentage packet loss you want to introduce. * Target Interface: The interface you want to target(Defaults to eth0). * Network Latency: This helps inject latency into the application pod. This type of chaos would need following information to run: * Pod Label: Label of the pods you want to target. * Target Namespace: The namespace of the application you wish to target. * CNF Name: The name of the CNF you want to target. * Chaos Duration: The time in seconds for how long you want to run the chaos. * Target Pod Percentage: The percentage of pods you want to target. * Latency: The amount of latency you would want to introduce. This is expected in milliseconds. * Target Interface: The interface you want to target(Defaults to eth0). * Pod Chaos: This helps deleting the pods based on the passed labels. This type of chaos would need following information to run: * Pod Label: Label of the pods you want to target * Target Namespace: The namespace of the application you wish to target. * CNF Name: The name of the CNF you want to target. * Chaos Duration: The time in seconds for how long you want to run the chaos. * Chaos Interval: The time in seconds of how frequently you would like to delete the pods. Basically, this is time between two kill operations.  ## FAQ TODO ## Trouble Shooting Guide TODO ## Developer Guide ### Structure of the Code ``` |-- Makefile |-- README.md |-- antchaos | |-- ant | | `-- test_scripts | | |-- __init__.py | | |-- startchaos | | | |-- script_meta_data | | | | |-- Default_Filter.yaml | | | | |-- UI_Columns.yaml | | | | |-- namespace.yaml | | | | |-- startchaos.yaml | | | | `-- vars | | | | |-- chaos_test_data.yaml | | | | |-- k8s_cluster_details.yaml | | | | |-- k8s_details.yaml | | | | |-- logId.yaml | | | | `-- testsuite_id.yaml | | | `-- test_cases | | | |-- startchaos.robot | | | `-- lib | | | `-- start_chaos.py | | `-- stopchaos | | |-- script_meta_data | | | |-- Default_Filter.yaml | | | |-- UI_Columns.yaml | | | |-- namespace.yaml | | | |-- stopchaos.yaml | | | `-- vars | | | |-- k8s_cluster_details.yaml | | | |-- k8s_details.yaml | | | |-- logId.yaml | | | `-- testsuite_id.yaml | | `-- test_cases | | |-- stopchaos.robot | | `-- lib | | `-- stop_chaos.py | |-- common | | |-- __about__.py | | |-- __init__.py | | |-- chaoslib | | | |-- chaos.py | | | |-- chaos_params.py | | | |-- k8s_client.py | | | `-- workflow.py | | `-- data_formatter | | `-- ant_data_formatter.py | `-- tests | |-- __init__.py | |-- functional | | `-- __init__.py | `-- units | |-- __init__.py | `-- test_k8s_client.py |-- docs | |-- Makefile | |-- conf.py | |-- index.rst | `-- readme.rst |-- image_builder | |-- Dockerfile | |-- README.md | `-- scripts | |-- container_config.env | `-- installation_script.sh |-- pylintrc |-- requirements.txt |-- test-requirements.txt |-- tools | |-- __init__.py | |-- builder.py | |-- metadata.py | `-- yapf-with-message.sh `-- tox.ini ``` ### How to add new Experiment Adding a new experiment would need following changes: 1. Add a chaos experiment in the chaosexperiments chart in [nc-helm repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-helm/browse/chaosexperiments?at=refs%2Fheads%2Fmain) 2. Add a chaos experiment argo workflow template in litmus-argo-workflow-templates chart in [nc-helm repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-helm/browse/litmus-argo-workflow-templates?at=refs%2Fheads%2Fmain) 3. Update antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml with required parameters in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml) 4. Update antchaos/common/chaoslib/chaos_params.py with an appropriate class and parameters. The name of the parameters should be similar to what is added in UI_Columns.yaml in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/common/chaoslib/chaos_params.py) 5. Update the antchaos/common/data_formatter/ant_data_formatter.py in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/common/data_formatter/ant_data_formatter.py) We will take the example of adding memory hog experiment and walk you through all the steps in the process: #### Step 1. Add new chaos experiment in the chaosexperiments directory of nc-helm repo: Clone the repo ``` ATTUID=<enter your att uid here> git clone "ssh://${ATTUID}@gerrit.mtn5.cci.att.com:29418/nc-helm" && scp -p -P 29418 ${ATTUID}@gerrit.mtn5.cci.att.com:hooks/commit-msg "nc-helm/.git/hooks/" ``` Go into the chaosexperiments directory under nc-helm directory ``` cd nc-helm/chaosexperiments ``` Structure of the directory: ``` # |-- Chart.yaml # |-- templates <-- Directory which holds the go template of resources to be created # | |-- argo-workflow-rbac.yaml # | |-- experiment-pod-cpu-hog.yaml # | |-- experiment-pod-delete.yaml # | |-- experiment-pod-network-latency.yaml # | |-- experiment-pod-network-loss.yaml # | |-- litmus-chaos-rbac.yaml # | |-- test-chaosexperiments-serviceaccount.yaml # | `-- tests <-- Directory to hold helm tests # | |-- test-chaosexperiments-pod-cpu-hog.yaml # | |-- test-chaosexperiments-pod-delete.yaml # | |-- test-chaosexperiments-pod-network-latency.yaml # | `-- test-chaosexperiments-pod-network-loss.yaml # `-- values.yaml <-- File to pass values to the go templates. Can be overridden via helm command --set option. ``` Create a file under templates directory ``` cat <<EOF > templates/experiment-pod-memory-hog.yaml # Install this experiment only if pod_memory_hog.enabled is set to true in values {{ if .Values.experiments.pod_memory_hog.enabled }} apiVersion: litmuschaos.io/v1alpha1 description: message: | Injects memory consumption on pods belonging to an app deployment kind: ChaosExperiment metadata: name: pod-memory-hog namespace: {{ .Values.namespace }} labels: name: pod-memory-hog app.kubernetes.io/part-of: litmus app.kubernetes.io/component: chaosexperiment app.kubernetes.io/version: 1.13.3 spec: definition: scope: Namespaced image: "{{ .Values.images.tags.go_runner }}" imagePullPolicy: {{ .Values.images.pull_policy }} args: - -c - ./experiments -name pod-memory-hog command: - /bin/bash env: - name: TOTAL_CHAOS_DURATION value: '60' ## enter the amount of memory in megabytes to be consumed by the application pod - name: MEMORY_CONSUMPTION value: '500' ## percentage of total pods to target - name: PODS_AFFECTED_PERC value: '' ## Period to wait before and after injection of chaos in sec - name: RAMP_TIME value: '' ## env var that describes the library used to execute the chaos ## default: litmus. Supported values: litmus, pumba - name: LIB value: 'litmus' # provide the socket file path # it is used in pumba lib - name: SOCKET_PATH value: '/run/containerd/containerd.sock' ## it defines the sequence of chaos execution for multiple target pods ## supported values: serial, parallel - name: SEQUENCE value: 'parallel' - name: TARGET_PODS value: '' labels: name: pod-memory-hog app.kubernetes.io/part-of: litmus app.kubernetes.io/component: experiment-job app.kubernetes.io/version: 1.13.3 {{ end }} EOF ``` Add following section under experiments: in values.yaml. If there is something else you wish to pass to the experiment would go in this file as well. ``` # Injects memory consumption on pods belonging to an app deployment pod_memory_hog: enabled: false ``` *NOTE: There are some default privileges which are required for pod and network chaos already present in **templates/litmus-chaos-rbac.yaml**. Please update this file if you wish to assign some extra privileges.* Lets add a helm test which will help us validate if the experiment is installed correctly or not. ``` cat << EOF > templates/tests/test-chaosexperiments-pod-memory-hog.yaml # Install this helm test only if pod_memory_hog.enabled is set to true in values {{ if .Values.experiments.pod_memory_hog.enabled }} apiVersion: v1 kind: Pod metadata: name: test-pod-memory-hog-chaosexperiment annotations: "helm.sh/hook": test-success "helm.sh/hook-delete-policy": hook-succeeded spec: serviceAccountName: {{ .Values.helm_test.user }} containers: - name: test-pod-memory-hog-chaosexperiment image: {{ .Values.images.tags.chaosexperiments_helm_test }} command: ["/bin/bash"] args: ["-c", "kubectl get chaosexperiments pod-memory-hog -n aqua -o yaml"] imagePullPolicy: {{ .Values.images.pull_policy }} restartPolicy: Never {{ end }} EOF ``` ##### Step 2. Add a chaos experiment argo workflow template in litmus-argo-workflow-templates chart in nc-helm repo Go into the litmus-argo-workflow-templates under nc-helm repo ``` cd litmus-argo-workflow-templates ``` Structure of the directory would be similar to what we saw in the case of chaosexperiments ``` # |-- Chart.yaml # |-- templates <-- Directory which holds the go template of resources to be created # | |-- chaos_manager_service_account.yaml # | |-- pod-cpu-hog-template.yaml # | |-- pod-delete-template.yaml # | |-- pod-memory-hog-template.yaml # | |-- pod-network-latency-template.yaml # | |-- pod-network-loss-template.yaml # | |-- test-workflowtemplates-serviceaccount.yaml # | `-- tests <-- Directory to hold helm tests # | |-- test-workflowtemplate-pod-cpu-hog.yaml # | |-- test-workflowtemplate-pod-delete.yaml # | |-- test-workflowtemplate-pod-memory-hog.yaml # | |-- test-workflowtemplate-pod-network-latency.yaml # | `-- test-workflowtemplate-pod-network-loss.yaml # `-- values.yaml <-- File to pass values to the go templates. Can be overridden via helm command --set option. ``` Create a file under templates directory ``` cat <<EOF > templates/pod-memory-hog-template.yaml # Install this experiment only if pod_memory_hog.enabled is set to true in values {{ if .Values.workflowTemplates.pod_memory_hog.enabled }} apiVersion: argoproj.io/v1alpha1 kind: WorkflowTemplate metadata: name: pod-memory-hog spec: arguments: parameters: - name: target_namespace - name: pod_label - name: memory_consumption - name: cleanup_policy - name: duration - name: target_pod_percentage templates: - name: pod-memory-hog inputs: parameters: - name: target_namespace - name: pod_label - name: memory_consumption - name: cleanup_policy - name: duration - name: target_pod_percentage resource: # indicates that this is a resource template action: create # can be any kubectl action (e.g. create, delete, apply, patch) successCondition: status.engineStatus == completed failureCondition: status.engineStatus == failed manifest: | #put your kubernetes spec here apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: generateName: pod-memory-hog- namespace: {{ .Values.chaos_namespace }} ownerReferences: - apiVersion: argoproj.io/v1alpha1 blockOwnerDeletion: true controller: true kind: Workflow name: '{{`{{workflow.name}}`}}' uid: '{{`{{workflow.uid}}`}}' spec: # It can be delete/retain jobCleanUpPolicy: '{{`{{inputs.parameters.cleanup_policy}}`}}' # It can be true/false annotationCheck: 'false' # It can be active/stop engineState: 'active' monitoring: false appinfo: appns: '{{`{{inputs.parameters.target_namespace}}`}}' # FYI, To see app label, apply kubectl get pods --show-labels applabel: '{{`{{inputs.parameters.pod_label}}`}}' chaosServiceAccount: {{ .Values.chaos_service_account }} components: runner: {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 18 }} {{- end }} experiments: - name: pod-memory-hog spec: components: {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 22 }} {{- end }} env: - name: MEMORY_CONSUMPTION value: '{{`{{inputs.parameters.memory_consumption}}`}}' - name: LIB_IMAGE value: {{ .Values.images.tags.go_runner }} - name: TOTAL_CHAOS_DURATION value: '{{`{{inputs.parameters.duration}}`}}' # in seconds - name: PODS_AFFECTED_PERC value: '{{`{{inputs.parameters.target_pod_percentage}}`}}' - name: LIB value: {{ .Values.env.library }} - name: CONTAINER_RUNTIME value: {{ .Values.env.container_runtime }} - name: SOCKET_PATH value: {{ .Values.env.socket_path }} - name: CHAOS_KILL_COMMAND value: "kill -9 $(ps afx | grep \"[dd] if /dev/zero\" | awk '{print $1}' | tr '\n' ' ')" {{ end }} EOF ``` Add following section under workflowTemplates: in values.yaml. If there is something else you wish to pass to the experiment would go in this file as well. ``` pod_memory_hog: enabled: false ``` NOTE: There are some default privileges which are required for pod and network chaos already present in **templates/chaos_manager_service_account.yaml**. Please update this file if you wish to assign some extra privileges. Lets add a helm test which will help us validate if the argo workflow template is installed correctly or not. ``` cat <<EOF >templates/tests/test-workflowtemplate-pod-memory-hog.yaml # Install this helm test only if pod_memory_hog.enabled is set to true in values {{ if .Values.workflowTemplates.pod_memory_hog.enabled }} apiVersion: v1 kind: Pod metadata: name: test-pod-memory-hog-workflowtemplate annotations: "helm.sh/hook": test-success "helm.sh/hook-delete-policy": hook-succeeded spec: serviceAccountName: {{ .Values.helm_test.user }} containers: - name: test-pod-memory-hog-workflowtemplate image: {{ .Values.images.tags.workflowtemplates_helm_test }} command: ["/bin/bash"] args: ["-c", "kubectl get workflowtemplate pod-memory-hog -n aqua -o yaml"] imagePullPolicy: {{ .Values.images.pull_policy }} restartPolicy: Never {{ end }} EOF ``` Once you are done with above steps you will have to push your change to gerrit for review. ``` git add chaosexperiments/templates/experiment-pod-memory-hog.yaml \ chaosexperiments/values.yaml \ chaosexperiments/templates/tests/test-chaosexperiments-pod-memory-hog.yaml \ litmus-argo-workflow-templates/templates/pod-memory-hog-template.yaml \ litmus-argo-workflow-templates/values.yaml \ litmus-argo-workflow-templates/templates/tests/test-workflowtemplate-pod-memory-hog.yaml git commit -m "Add memory hog experiment" git push origin HEAD:/refs/for/main ``` Note: Pass all the files to git add command which you add or modify. #### Step 3. Update antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml with required parameters in nc-ant-test-cases repo Clone the repo ``` ATTUID=<enter your att uid here> git clone "ssh://${ATTUID}@gerrit.mtn5.cci.att.com:29418/nc-ant-test-cases" && scp -p -P 29418 ${ATTUID}@gerrit.mtn5.cci.att.com:hooks/commit-msg "nc-ant-test-cases/.git/hooks/" ``` The structure of the code is already pasted above in [Structure of the Code](###Structure-of-the-Code) section Go into the script_meta_data directory under startchaos ``` cd antchaos/ant/test_scripts/startchaos/script_meta_data/ ``` You will see following files: ``` # |-- Default_Filter.yaml # |-- UI_Columns.yaml # |-- namespace.yaml # |-- startchaos.yaml # `-- vars # |-- chaos_test_data.yaml # |-- k8s_cluster_details.yaml # |-- k8s_details.yaml # |-- logId.yaml # `-- testsuite_id.yaml ``` We will just update the *UI_Columns.yaml* for this experiment. Lets add a new type of chaos in chaos test type. Currently it should look something like this: ``` elements: - chaos_test_type: displayName: Chaos Test Type value: value: '' possibleValues: value: - Pod Chaos - Network Latency chaos - Packet Loss Chaos ``` We will change it to: ``` elements: - chaos_test_type: displayName: Chaos Test Type value: value: '' possibleValues: value: - Pod Chaos - Network Latency chaos - Packet Loss Chaos - Pod Memory Hog ``` We already have some parameters which we need for all the experiments like, chaos_cnf_name, target_namespace, duration and, pod_label. We will add the type of chaos to each of them. Let's take chaos_cnf_name as example and rest of them are to be updated in similar fashion. Here is how the chaos_cnf_name looks originally ``` chaos_cnf_name: displayName: CNF NAME value: value: '' possibleValues: value: - AMF - NRF - UPF - SMF required: value: 'true' defaultValue: value: '' fieldType: value: dropdown dependantColumns: dependency: columns: - name: chaos_test_type value: - Pod Chaos - Network Latency chaos - Packet Loss Chaos ``` This is how it looks after adding memory hog: ``` chaos_cnf_name: displayName: CNF NAME value: value: '' possibleValues: value: - AMF - NRF - UPF - SMF required: value: 'true' defaultValue: value: '' fieldType: value: dropdown dependantColumns: dependency: columns: - name: chaos_test_type value: - Pod Chaos - Network Latency chaos - Packet Loss Chaos - Pod Memory Hog ``` We need to add the type of chaos, in this case "Pod Memory Hog", in the dependantColumns section of all those fields which we need for that particular chaos. Now, the chaos would have some parameters which are not part of any experiment. In this case that parameter is memory consumption. We can add that parameter in the end. Add following lines in the end: ``` memory_consumption: displayName: Memory Consumption (in MB) value: value: '' possibleValues: value: '' required: value: 'true' defaultValue: value: '' fieldType: value: text dependantColumns: dependency: columns: - name: chaos_test_type value: Pod Memory Hog selectedValues: {} showColumn: 'true' derivationQuery: - source: '' output: '' collection: '' filter: {} projection: '' ``` *Note: Please make sure of matching the indentation level as the last parameter(field)* #### Step 4. Update antchaos/common/chaoslib/chaos_params.py with an appropriate class and parameters Assuming you are at the top directory of the repo. ``` cd antchaos/common/chaoslib/ ``` Following will be the structure of code in this directory: ``` # |-- chaos.py # |-- chaos_params.py # |-- k8s_client.py # `-- workflow.py ``` We would be mostly focussing on chaos_params.py as this file has all the parameters we need to run the chaos. We have a base class "BaseChaosParameter" which has all the parameters which are common amongst all type of chaoses. So, in this we won't be touching the base class but we would be adding a new class, "MemoryHogChaosParameter", for memory hog chaos. We would add this class in chaos_params.py in the end of the params classes( we can add this anywhere in file). Here is how the new class looks like: ``` class MemoryHogChaosParameter(BaseChaosParameter): """Holds the paramteres for pod delete chaos""" def __init__(self, data): super().__init__(data) # If you change any of the 2 values please make sure to update the # workflow template as well. # This is the name of the WorkflowTemplate CRD. self.workflow_template_name = 'pod-memory-hog' # This is the name of the template under WorkflowTemplate CRD self.template_name = 'pod-memory-hog' self._params.update({ 'memory_consumption': '', 'target_pod_percentage': '', }) self.set_value(data) ``` The 2 parameters, memory consumption and target pod percentage, can be seen in the above code. Now, we will have to update the new type of chaos in the factory method. The " _chaos_parameters()" method in the "ChaosParams" class used to look like: ``` @property def _chaos_parameters(self): """Returns the appropriate object depending on the type of chaos""" if self.data['type'] == 'pod-delete': return PodChaosParameter(self.data['params']) elif self.data['type'] == 'network-latency': return NetworkLatencyChaosParameter(self.data['params']) elif self.data['type'] == 'network-packet-loss': return NetworkPacketLossChaosParameter(self.data['params']) else: raise ValueError(self.data['type']) ``` We will change it to: ``` @property def _chaos_parameters(self): """Returns the appropriate object depending on the type of chaos""" if self.data['type'] == 'pod-delete': return PodChaosParameter(self.data['params']) elif self.data['type'] == 'network-latency': return NetworkLatencyChaosParameter(self.data['params']) elif self.data['type'] == 'network-packet-loss': return NetworkPacketLossChaosParameter(self.data['params']) elif self.data['type'] == 'pod-memory-hog': return MemoryHogChaosParameter(self.data['params']) else: raise ValueError(self.data['type']) ``` #### Step 5. Update the antchaos/common/data_formatter/ant_data_formatter.py** We will have to update the ant_workflow_template_mapper dict in the antchaos/common/data_formatter/ant_data_formatter.py file. This mapping helps us in mapping the type of chaos coming from ANT UI to an appropriate class in the chaos_params.py file. We will have to update the "format_ant_chaos_data()" method in the "ANTDataFormatter". The method originally has only 3 types of chaoses: ``` ant_workflow_template_mapper = { 'Network Latency chaos': 'network-latency', 'Pod Chaos': 'pod-delete', 'Packet Loss Chaos': 'packet-loss', } ``` We will update it to: ``` ant_workflow_template_mapper = { 'Network Latency chaos': 'network-latency', 'Pod Chaos': 'pod-delete', 'Packet Loss Chaos': 'packet-loss', 'Pod Memory Hog': 'pod-memory-hog' } ``` Once you are done with above steps you will have to push your change to gerrit for review. ``` git add antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml \ antchaos/common/chaoslib/chaos_params.py \ antchaos/common/data_formatter/ant_data_formatter.py git commit -m "Add memory hog experiment" git push origin HEAD:/refs/for/main ```