# ANT Chaos Document Guide

## Table of Contents
* [**Introduction**](#Introduction-to-Chaos-Engineering)
* [**Architecture**](#Architecture)
* [**Installation**](#Installation)
* [**Usage**](#Usage)
* [**FAQ**](#FAQ)
* [**Trouble Shooting Guide**](#Trouble-Shooting-Guide)
* [**Developer Guide**](#Developer-Guide)

## Introduction to Chaos Engineering
Chaos engineering is a disciplined approach to identifying failures before they become outages. It lets you compare what you ‘think will happen’ during an outage to what ‘actually happens’ in your systems.

This is how [principlesofchaos.org](https://principlesofchaos.org/) defines chaos engineering:
*“Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.”*

### Why do we need Chaos Engineering?
The chaos engineering helps make the system more resilient and reliable. It lets you identify the weakness of the system before it becomes a major outage. This helps in taking a step back and redesign our system.

### How do we do Chaos Engineering?
* Start by defining the steady state of the system which can later be used for comparing it with the status of the system under chaos.
* Introduce failures in the system
* Test the functionality of the system while the system is under chaos and stress
* Compare the results of steady state and the system under chaos

## Pre requisites:

* Running Kubernetes cluster(version > 1.15) with application running
* Access token with privileges enough to install few helm charts containing, deployments, CRDs, CRs, SA, role, role bindings, Namespace
* The Environment under test should have network connectivity to artifactory
* ANT portal with latest version of code.
* Kubernetes details added in the appropriate CNF

## Architecture

ANT Chaos is constructed out of four parts

* Core - Core consists of Argo Workflow and LitmusChaos
* Chaos Experiments - Chaos experiments contain the actual chaos details like pod-delete, packet-loss.
* Chaos Workflow Templates - Argo workflow templates that deploys the Chaos expermiments
* ANT Test Scripts - ANT Test Scripts contains the logic of executing different chaos based on the user inputs



## Installation
### Deploy Cluster Components
You can find the installation steps at:
1. **Manual procedure:**
1. Create the aqua namespace

    kubectl create ns aqua

2. Label K8s nodes

Label the nodes in which you want to install the argo,litmus and other helm charts.

    kubectl get nodes

    NAME STATUS ROLES AGE VERSION
    kind-control-plane Ready control-plane,master 48m v1.20.2

    kubectl label node kind-control-plane ucp-control-plane=enabled

3. Deploy argo Helm Chart

    wget https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/argo-2bef8e8b3a62.6dd39da6ad7f.tgz

    tar -xvf argo-2bef8e8b3a62.6dd39da6ad7f.tgz

    helm2 repo add nc-charts https://artifacts-nc.auk3.cci.att.com/artifactory/helm-charts/

    helm2 dependency update ./argo

    helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/argo-2bef8e8b3a62.6dd39da6ad7f.tgz -n argo --namespace=aqua --wait

4. Deploy litmus Helm Chart

    helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus --namespace=aqua --wait

5. Deploy Litmus Chaos expermiments Helm Chart

    helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/chaosexperiments-2bef8e8b3a62.6dd39da6ad7f.tgz -n chaosexperiments \
    --namespace=aqua \
    --set experiments.pod_delete.enabled=true,experiments.network_loss.enabled=true,experiments.network_latency.enabled=true \
    --wait

6. Deploy Argo workflow templates Helm Chart

* if the CRI is Containerd

    helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-argo-workflow-templates-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus-argo-workflow-templates \
    --namespace=aqua \
    --set workflowTemplates.pod_delete.enabled=true,workflowTemplates.network_loss.enabled=true,workflowTemplates.network_latency.enabled=true \
    --wait

* if the CRI is docker

    helm2 install https://artifacts-nc.mtn57z.cti.att.com/artifactory/nc-helm-local/nc-helm/litmus-argo-workflow-templates-2bef8e8b3a62.6dd39da6ad7f.tgz -n litmus-argo-workflow-templates \
    --namespace=aqua \
    --set workflowTemplates.pod_delete.enabled=true,workflowTemplates.network_loss.enabled=true,workflowTemplates.network_latency.enabled=true \
    --set  env.container_runtime=docker,env.socket_path=/var/run/docker.sock \
    --wait

7. Create ClusterRole Binding for ANT to fetch Pods/Labels dynamically

    kubectl create clusterrolebinding aqua-view-binding --clusterrole=view --serviceaccount=aqua:chaos-manager

2. **Automated procedure**

**Prerequisites:**
Make sure the target cluster control-plane node has helm2 and a tiller pod running

**Installation Steps**
Below helm charts will be deployed as part of this process

* argo
* litmus
* chaosexperiments
* litmus-argo-workflow-templates
</br>
1. Run below command to generate the sample container_config.env in your current working directory:

    $ docker run --net=host --rm \
    docker-nc.zc1.cti.att.com/nc/nc-ant-test-    cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090 \
    cat /tmp/scripts/container_config.env >./container_config.env

2. container_config.env has below-highlighted content, modify the container_config.env as per your requirement and save it

    #Specify the target namespace to deploy the helm charts
    NAMESPACE=aqua
    #Specify the target control-plane node name, where it will add a node label "ucp-control-plane=enabled"
    CONTROL_PLANE_NODE_NAME=kind-control-plane
    #Specify the litmus experiments true/false based on the experiments you want to install
    POD_DELETE=true
    POD_NETWORK_LATENCY=true
    POD_NETWORK_LOSS=true
    POD_MEMORY_HOG=true
    POD_CPU_HOG=true
    #specify the container runtime ex: docker/containerd
    CONTAINER_RUNTIME=containerd
    #specify the respective socket_path for the above-specified container runtime
    SOCKET_PATH=/run/containerd/containerd.soc

3. Run the below docker container to perform the installation:

    IMAGE_NAME== docker-nc.zc1.cti.att.com/nc/nc-ant-test-cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090
    $ docker run --net=host --rm \
        -v <path_to_admin.conf or config>:/tmp/scripts/admin.conf \      # In the target cluster look for /etc/kubernetes/admin.conf or $HOME/.kube/config and mount to /tmp/scripts/admin.conf in the container 
        --env-file=./container_config.env \                              # The container_config.env from your present working directory
        $IMAGE_NAME

    Example:
    docker run --net=host --rm -v $HOME/.kube/config:/tmp/scripts/admin.conf --env-file=./container_config.env docker-nc.zc1.cti.att.com/nc/nc-ant-test-cases/aqua-ant-chaos-charts@sha256:dd01eb63683972d565ae6348032b332d1c4ac25fca3d25b2f83777d3d7a3e090

#### Verify the installation:

Verify if the ChaosOperator is running

$ kubectl get pods -n aqua | grep litmus
litmus-5cbc65959f-zfqrx                                  1/1     Running     0          10d

Verify if chaos experiments are installed

$ kubectl get chaosexperiments -n aqua              

NAME                             AGE
pod-delete                       10d
pod-network-latency              10d
pod-network-loss.                10d

Verify if litmus-argo-workflowtemplate are installed

$ kubectl get workflowtemplates -n aqua
NAME                                                     AGE
network-latency-chaos-workflow                           11s
network-packet-loss-chaos-workflow                       11s
pod-chaos-workflow                                       11s

### Deploy Chaos Testscripts to ANT Portal
* Login to ANT Portal
* Navigate to Test Script Inventory page by clicking `Inventory` --> `Test Script Inventory` from the top menu Bar.
* Click on `Upload ZIP File` button to upload the testscript
* Once upload is success, then Click on `Sync` button to sync the uploaded script to ATE(s)


## Usage

* Click on APPS and then select Test Management from the top menu bar
* Select the appropriate DUT.
* Select the type of chaos you want to run(Available options: `pakcet loss`, `network latency`, and `pod chaos`).

* Provide the information regarding the target application.
* Pakcet loss: This helps inject packet loss into application pod.
This type of chaos would need following information to run:
* Pod Label: Label of the pods you want to target.
* Target Namespace: The namespace of the application you wish to target.
* CNF Name: The name of the CNF you want to target.
* Chaos Duration: The time in seconds for how long you want to run the chaos.
* Percentage: The percentage packet loss you want to introduce.
* Target Interface: The interface you want to target(Defaults to eth0).

* Network Latency: This helps inject latency into the application pod.
This type of chaos would need following information to run:
* Pod Label: Label of the pods you want to target.
* Target Namespace: The namespace of the application you wish to target.
* CNF Name: The name of the CNF you want to target.
* Chaos Duration: The time in seconds for how long you want to run the chaos.
* Target Pod Percentage: The percentage of pods you want to target.
* Latency: The amount of latency you would want to introduce. This is expected in milliseconds.
* Target Interface: The interface you want to target(Defaults to eth0).

* Pod Chaos: This helps deleting the pods based on the passed labels.
This type of chaos would need following information to run:
* Pod Label: Label of the pods you want to target
* Target Namespace: The namespace of the application you wish to target.
* CNF Name: The name of the CNF you want to target.
* Chaos Duration: The time in seconds for how long you want to run the chaos.
* Chaos Interval: The time in seconds of how frequently you would like to delete the pods. Basically, this is time between two kill operations.


## FAQ

TODO

## Trouble Shooting Guide

TODO

## Developer Guide

### Structure of the Code

|-- Makefile
|-- README.md
|-- antchaos
|   |-- ant
|   |   `-- test_scripts
|   |       |-- __init__.py
|   |       |-- startchaos
|   |       |   |-- script_meta_data
|   |       |   |   |-- Default_Filter.yaml
|   |       |   |   |-- UI_Columns.yaml
|   |       |   |   |-- namespace.yaml
|   |       |   |   |-- startchaos.yaml
|   |       |   |   `-- vars
|   |       |   |       |-- chaos_test_data.yaml
|   |       |   |       |-- k8s_cluster_details.yaml
|   |       |   |       |-- k8s_details.yaml
|   |       |   |       |-- logId.yaml
|   |       |   |       `-- testsuite_id.yaml
|   |       |   `-- test_cases
|   |       |       |-- startchaos.robot
|   |       |       `-- lib
|   |       |           `-- start_chaos.py
|   |       `-- stopchaos
|   |           |-- script_meta_data
|   |           |   |-- Default_Filter.yaml
|   |           |   |-- UI_Columns.yaml
|   |           |   |-- namespace.yaml
|   |           |   |-- stopchaos.yaml
|   |           |   `-- vars
|   |           |       |-- k8s_cluster_details.yaml
|   |           |       |-- k8s_details.yaml
|   |           |       |-- logId.yaml
|   |           |       `-- testsuite_id.yaml
|   |           `-- test_cases
|   |               |-- stopchaos.robot
|   |               `-- lib
|   |                   `-- stop_chaos.py
|   |-- common
|   |   |-- __about__.py
|   |   |-- __init__.py
|   |   |-- chaoslib
|   |   |   |-- chaos.py
|   |   |   |-- chaos_params.py
|   |   |   |-- k8s_client.py
|   |   |   `-- workflow.py
|   |   `-- data_formatter
|   |       `-- ant_data_formatter.py
|   `-- tests
|       |-- __init__.py
|       |-- functional
|       |   `-- __init__.py
|       `-- units
|           |-- __init__.py
|           `-- test_k8s_client.py
|-- docs
|   |-- Makefile
|   |-- conf.py
|   |-- index.rst
|   `-- readme.rst
|-- image_builder
|   |-- Dockerfile
|   |-- README.md
|   `-- scripts
|       |-- container_config.env
|       `-- installation_script.sh
|-- pylintrc
|-- requirements.txt
|-- test-requirements.txt
|-- tools
|   |-- __init__.py
|   |-- builder.py
|   |-- metadata.py
|   `-- yapf-with-message.sh
`-- tox.ini

### How to add new Experiment
Adding a new experiment would need following changes:
1. Add a chaos experiment in the chaosexperiments chart in [nc-helm repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-helm/browse/chaosexperiments?at=refs%2Fheads%2Fmain)
2. Add a chaos experiment argo workflow template in litmus-argo-workflow-templates chart in [nc-helm repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-helm/browse/litmus-argo-workflow-templates?at=refs%2Fheads%2Fmain)
3. Update antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml with required parameters in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml)
4. Update antchaos/common/chaoslib/chaos_params.py with an appropriate class and parameters. The name of the parameters should be similar to what is added in UI_Columns.yaml in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/common/chaoslib/chaos_params.py)
5. Update the antchaos/common/data_formatter/ant_data_formatter.py in [nc-ant-test-cases repo](https://codecloud.web.att.com/projects/ST_CCP/repos/nc-ant-test-cases/browse/antchaos/common/data_formatter/ant_data_formatter.py)

We will take the example of adding memory hog experiment and walk you through all the steps in the process:

#### Step 1. Add new chaos experiment in the chaosexperiments directory of nc-helm repo:

Clone the repo

ATTUID=<enter your att uid here>
git clone "ssh://${ATTUID}@gerrit.mtn5.cci.att.com:29418/nc-helm" && scp -p -P 29418 ${ATTUID}@gerrit.mtn5.cci.att.com:hooks/commit-msg "nc-helm/.git/hooks/"

Go into the chaosexperiments directory under nc-helm directory

cd nc-helm/chaosexperiments

Structure of the directory:

# |-- Chart.yaml
# |-- templates    <-- Directory which holds the go template of resources to be created
# |   |-- argo-workflow-rbac.yaml
# |   |-- experiment-pod-cpu-hog.yaml
# |   |-- experiment-pod-delete.yaml
# |   |-- experiment-pod-network-latency.yaml
# |   |-- experiment-pod-network-loss.yaml
# |   |-- litmus-chaos-rbac.yaml
# |   |-- test-chaosexperiments-serviceaccount.yaml
# |   `-- tests    <-- Directory to hold helm tests
# |       |-- test-chaosexperiments-pod-cpu-hog.yaml
# |       |-- test-chaosexperiments-pod-delete.yaml
# |       |-- test-chaosexperiments-pod-network-latency.yaml
# |       `-- test-chaosexperiments-pod-network-loss.yaml
# `-- values.yaml    <-- File to pass values to the go templates. Can be overridden via helm command --set option.

Create a file under templates directory

cat <<EOF > templates/experiment-pod-memory-hog.yaml
# Install this experiment only if pod_memory_hog.enabled is set to true in values
{{ if .Values.experiments.pod_memory_hog.enabled }}
apiVersion: litmuschaos.io/v1alpha1
description:
  message: |
    Injects memory consumption on pods belonging to an app deployment
kind: ChaosExperiment
metadata:
  name: pod-memory-hog
  namespace: {{ .Values.namespace }}
  labels:
    name: pod-memory-hog
    app.kubernetes.io/part-of: litmus
    app.kubernetes.io/component: chaosexperiment
    app.kubernetes.io/version: 1.13.3
spec:
  definition:
    scope: Namespaced
    image: "{{ .Values.images.tags.go_runner }}"
    imagePullPolicy: {{ .Values.images.pull_policy }}
    args:
    - -c
    - ./experiments -name pod-memory-hog
    command:
    - /bin/bash
    env:
      - name: TOTAL_CHAOS_DURATION
        value: '60'

      ## enter the amount of memory in megabytes to be consumed by the application pod
      - name: MEMORY_CONSUMPTION
        value: '500'

      ## percentage of total pods to target
      - name: PODS_AFFECTED_PERC
        value: ''

      ## Period to wait before and after injection of chaos in sec
      - name: RAMP_TIME
        value: ''

      ## env var that describes the library used to execute the chaos
      ## default: litmus. Supported values: litmus, pumba
      - name: LIB
        value: 'litmus'

      # provide the socket file path
      # it is used in pumba lib
      - name: SOCKET_PATH
        value: '/run/containerd/containerd.sock'

      ## it defines the sequence of chaos execution for multiple target pods
      ## supported values: serial, parallel
      - name: SEQUENCE
        value: 'parallel'

      - name: TARGET_PODS
        value: ''

    labels:
      name: pod-memory-hog
      app.kubernetes.io/part-of: litmus
      app.kubernetes.io/component: experiment-job
      app.kubernetes.io/version: 1.13.3
{{ end }}
EOF

Add following section under experiments: in values.yaml. If there is something else you wish to pass to the experiment would go in this file as well.

  # Injects memory consumption on pods belonging to an app deployment
  pod_memory_hog:
    enabled: false

*NOTE: There are some default privileges which are required for pod and network chaos already present in **templates/litmus-chaos-rbac.yaml**. Please update this file if you wish to assign some extra privileges.*

Lets add a helm test which will help us validate if the experiment is installed correctly or not.

cat << EOF > templates/tests/test-chaosexperiments-pod-memory-hog.yaml
# Install this helm test only if pod_memory_hog.enabled is set to true in values
{{ if .Values.experiments.pod_memory_hog.enabled }}
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-memory-hog-chaosexperiment
  annotations:
    "helm.sh/hook": test-success
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  serviceAccountName: {{ .Values.helm_test.user }}
  containers:
  - name: test-pod-memory-hog-chaosexperiment
    image: {{ .Values.images.tags.chaosexperiments_helm_test }}
    command: ["/bin/bash"]
    args: ["-c", "kubectl get chaosexperiments pod-memory-hog -n aqua -o yaml"]
    imagePullPolicy: {{ .Values.images.pull_policy }}
  restartPolicy: Never
{{ end }}
EOF

##### Step 2. Add a chaos experiment argo workflow template in litmus-argo-workflow-templates chart in nc-helm repo

Go into the litmus-argo-workflow-templates under nc-helm repo

cd litmus-argo-workflow-templates

Structure of the directory would be similar to what we saw in the case of chaosexperiments

# |-- Chart.yaml
# |-- templates    <-- Directory which holds the go template of resources to be created
# |   |-- chaos_manager_service_account.yaml
# |   |-- pod-cpu-hog-template.yaml
# |   |-- pod-delete-template.yaml
# |   |-- pod-memory-hog-template.yaml
# |   |-- pod-network-latency-template.yaml
# |   |-- pod-network-loss-template.yaml
# |   |-- test-workflowtemplates-serviceaccount.yaml
# |   `-- tests    <-- Directory to hold helm tests
# |       |-- test-workflowtemplate-pod-cpu-hog.yaml
# |       |-- test-workflowtemplate-pod-delete.yaml
# |       |-- test-workflowtemplate-pod-memory-hog.yaml
# |       |-- test-workflowtemplate-pod-network-latency.yaml
# |       `-- test-workflowtemplate-pod-network-loss.yaml
# `-- values.yaml    <-- File to pass values to the go templates. Can be overridden via helm command --set option.

Create a file under templates directory

cat <<EOF > templates/pod-memory-hog-template.yaml
# Install this experiment only if pod_memory_hog.enabled is set to true in values
{{ if .Values.workflowTemplates.pod_memory_hog.enabled }}
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: pod-memory-hog
spec:
  arguments:
    parameters:
      - name: target_namespace
      - name: pod_label
      - name: memory_consumption
      - name: cleanup_policy
      - name: duration
      - name: target_pod_percentage
  templates:
    - name: pod-memory-hog
      inputs:
        parameters:
          - name: target_namespace
          - name: pod_label
          - name: memory_consumption
          - name: cleanup_policy
          - name: duration
          - name: target_pod_percentage
      resource:                   # indicates that this is a resource template
        action: create            # can be any kubectl action (e.g. create, delete, apply, patch)
        successCondition: status.engineStatus == completed
        failureCondition: status.engineStatus == failed
        manifest: |               #put your kubernetes spec here
          apiVersion: litmuschaos.io/v1alpha1
          kind: ChaosEngine
          metadata:
            generateName: pod-memory-hog-
            namespace: {{ .Values.chaos_namespace }}
            ownerReferences:
            - apiVersion: argoproj.io/v1alpha1
              blockOwnerDeletion: true
              controller: true
              kind: Workflow
              name: '{{`{{workflow.name}}`}}'
              uid: '{{`{{workflow.uid}}`}}'
          spec:
            # It can be delete/retain
            jobCleanUpPolicy: '{{`{{inputs.parameters.cleanup_policy}}`}}'
            # It can be true/false
            annotationCheck: 'false'
            # It can be active/stop
            engineState: 'active'
            monitoring: false
            appinfo:
              appns: '{{`{{inputs.parameters.target_namespace}}`}}'
              # FYI, To see app label, apply kubectl get pods --show-labels
              applabel: '{{`{{inputs.parameters.pod_label}}`}}'
            chaosServiceAccount: {{ .Values.chaos_service_account }}
            components:
              runner:
                {{- with .Values.nodeSelector }}
                nodeSelector:
                  {{- toYaml . | nindent 18 }}
                {{- end }}
            experiments:
              - name: pod-memory-hog
                spec:
                  components:
                    {{- with .Values.nodeSelector }}
                    nodeSelector:
                      {{- toYaml . | nindent 22 }}
                    {{- end }}
                    env:
                      - name: MEMORY_CONSUMPTION
                        value: '{{`{{inputs.parameters.memory_consumption}}`}}'
                      - name: LIB_IMAGE
                        value: {{ .Values.images.tags.go_runner }}
                      - name: TOTAL_CHAOS_DURATION
                        value: '{{`{{inputs.parameters.duration}}`}}' # in seconds
                      - name: PODS_AFFECTED_PERC
                        value: '{{`{{inputs.parameters.target_pod_percentage}}`}}'
                      - name: LIB
                        value: {{ .Values.env.library }}
                      - name: CONTAINER_RUNTIME
                        value: {{ .Values.env.container_runtime }}
                      - name: SOCKET_PATH
                        value: {{ .Values.env.socket_path }}
                      - name: CHAOS_KILL_COMMAND
                        value: "kill -9 $(ps afx | grep \"[dd] if /dev/zero\" | awk '{print $1}' | tr '\n' ' ')"
{{ end }}
EOF

Add following section under workflowTemplates: in values.yaml. If there is something else you wish to pass to the experiment would go in this file as well.

  pod_memory_hog:
    enabled: false

NOTE: There are some default privileges which are required for pod and network chaos already present in **templates/chaos_manager_service_account.yaml**. Please update this file if you wish to assign some extra privileges.

Lets add a helm test which will help us validate if the argo workflow template is installed correctly or not.

cat <<EOF >templates/tests/test-workflowtemplate-pod-memory-hog.yaml
# Install this helm test only if pod_memory_hog.enabled is set to true in values
{{ if .Values.workflowTemplates.pod_memory_hog.enabled }}
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-memory-hog-workflowtemplate
  annotations:
    "helm.sh/hook": test-success
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  serviceAccountName: {{ .Values.helm_test.user }}
  containers:
  - name: test-pod-memory-hog-workflowtemplate
    image: {{ .Values.images.tags.workflowtemplates_helm_test }}
    command: ["/bin/bash"]
    args: ["-c", "kubectl get workflowtemplate pod-memory-hog -n aqua -o yaml"]
    imagePullPolicy: {{ .Values.images.pull_policy }}
  restartPolicy: Never
{{ end }}
EOF

Once you are done with above steps you will have to push your change to gerrit for review.

git add chaosexperiments/templates/experiment-pod-memory-hog.yaml \
chaosexperiments/values.yaml \
chaosexperiments/templates/tests/test-chaosexperiments-pod-memory-hog.yaml \
litmus-argo-workflow-templates/templates/pod-memory-hog-template.yaml \
litmus-argo-workflow-templates/values.yaml \
litmus-argo-workflow-templates/templates/tests/test-workflowtemplate-pod-memory-hog.yaml

git commit -m "Add memory hog experiment"
git push origin HEAD:/refs/for/main

Note: Pass all the files to git add command which you add or modify.

#### Step 3. Update antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml with required parameters in nc-ant-test-cases repo

Clone the repo

ATTUID=<enter your att uid here>

git clone "ssh://${ATTUID}@gerrit.mtn5.cci.att.com:29418/nc-ant-test-cases" && scp -p -P 29418 ${ATTUID}@gerrit.mtn5.cci.att.com:hooks/commit-msg "nc-ant-test-cases/.git/hooks/"

The structure of the code is already pasted above in [Structure of the Code](###Structure-of-the-Code) section

Go into the script_meta_data directory under startchaos

cd antchaos/ant/test_scripts/startchaos/script_meta_data/

You will see following files:

# |-- Default_Filter.yaml
# |-- UI_Columns.yaml
# |-- namespace.yaml
# |-- startchaos.yaml
# `-- vars
#     |-- chaos_test_data.yaml
#     |-- k8s_cluster_details.yaml
#     |-- k8s_details.yaml
#     |-- logId.yaml
#     `-- testsuite_id.yaml

We will just update the *UI_Columns.yaml* for this experiment.

Lets add a new type of chaos in chaos test type. Currently it should look something like this:

    elements:
    - chaos_test_type:
        displayName: Chaos Test Type
        value:
          value: ''
        possibleValues:
          value:
          - Pod Chaos
          - Network Latency chaos
          - Packet Loss Chaos

We will change it to:

    elements:
    - chaos_test_type:
        displayName: Chaos Test Type
        value:
          value: ''
        possibleValues:
          value:
          - Pod Chaos
          - Network Latency chaos
          - Packet Loss Chaos
          - Pod Memory Hog

We already have some parameters which we need for all the experiments like, chaos_cnf_name, target_namespace, duration and, pod_label. We will add the type of chaos to each of them. Let's take chaos_cnf_name as example and rest of them are to be updated in similar fashion.

Here is how the chaos_cnf_name looks originally

chaos_cnf_name:
        displayName: CNF NAME
        value:
          value: ''
        possibleValues:
          value:
          - AMF
          - NRF
          - UPF
          - SMF
        required:
          value: 'true'
        defaultValue:
          value: ''
        fieldType:
          value: dropdown
        dependantColumns:
          dependency:
            columns:
            - name: chaos_test_type
              value:
              - Pod Chaos
              - Network Latency chaos
              - Packet Loss Chaos

This is how it looks after adding memory hog:

chaos_cnf_name:
        displayName: CNF NAME
        value:
          value: ''
        possibleValues:
          value:
          - AMF
          - NRF
          - UPF
          - SMF
        required:
          value: 'true'
        defaultValue:
          value: ''
        fieldType:
          value: dropdown
        dependantColumns:
          dependency:
            columns:
            - name: chaos_test_type
              value:
              - Pod Chaos
              - Network Latency chaos
              - Packet Loss Chaos
              - Pod Memory Hog

We need to add the type of chaos, in this case "Pod Memory Hog", in the dependantColumns section of all those fields which we need for that particular chaos.

Now, the chaos would have some parameters which are not part of any experiment. In this case that parameter is memory consumption. We can add that parameter in the end. Add following lines in the end:

memory_consumption:
        displayName: Memory Consumption (in MB)
        value:
          value: ''
        possibleValues:
          value: ''
        required:
          value: 'true'
        defaultValue:
          value: ''
        fieldType:
          value: text
        dependantColumns:
          dependency:
            columns:
            - name: chaos_test_type
              value: Pod Memory Hog
          selectedValues: {}
          showColumn: 'true'
        derivationQuery:
        - source: ''
          output: ''
          collection: ''
          filter: {}
          projection: ''

*Note: Please make sure of matching the indentation level as the last parameter(field)*

#### Step 4. Update antchaos/common/chaoslib/chaos_params.py with an appropriate class and parameters

Assuming you are at the top directory of the repo.

cd antchaos/common/chaoslib/

Following will be the structure of code in this directory:

# |-- chaos.py
# |-- chaos_params.py
# |-- k8s_client.py
# `-- workflow.py

We would be mostly focussing on chaos_params.py as this file has all the parameters we need to run the chaos. We have a base class "BaseChaosParameter" which has all the parameters which are common amongst all type of chaoses. So, in this we won't be touching the base class but we would be adding a new class, "MemoryHogChaosParameter", for memory hog chaos.

We would add this class in chaos_params.py in the end of the params classes( we can add this anywhere in file). Here is how the new class looks like:

class MemoryHogChaosParameter(BaseChaosParameter):
    """Holds the paramteres for pod delete chaos"""
    def __init__(self, data):
        super().__init__(data)
        # If you change any of the 2 values please make sure to update the
        # workflow template as well.
        # This is the name of the WorkflowTemplate CRD.
        self.workflow_template_name = 'pod-memory-hog'
        # This is the name of the template under WorkflowTemplate CRD
        self.template_name = 'pod-memory-hog'
        self._params.update({
            'memory_consumption': '',
            'target_pod_percentage': '',
        })
        self.set_value(data)

The 2 parameters, memory consumption and target pod percentage, can be seen in the above code.

Now, we will have to update the new type of chaos in the factory method. The " _chaos_parameters()" method in the "ChaosParams" class used to look like:

    @property
    def _chaos_parameters(self):
        """Returns the appropriate object depending on the type of chaos"""
        if self.data['type'] == 'pod-delete':
            return PodChaosParameter(self.data['params'])
        elif self.data['type'] == 'network-latency':
            return NetworkLatencyChaosParameter(self.data['params'])
        elif self.data['type'] == 'network-packet-loss':
            return NetworkPacketLossChaosParameter(self.data['params'])
        else:
            raise ValueError(self.data['type'])

We will change it to:

    @property
    def _chaos_parameters(self):
        """Returns the appropriate object depending on the type of chaos"""
        if self.data['type'] == 'pod-delete':
            return PodChaosParameter(self.data['params'])
        elif self.data['type'] == 'network-latency':
            return NetworkLatencyChaosParameter(self.data['params'])
        elif self.data['type'] == 'network-packet-loss':
            return NetworkPacketLossChaosParameter(self.data['params'])
        elif self.data['type'] == 'pod-memory-hog':
            return MemoryHogChaosParameter(self.data['params'])
        else:
            raise ValueError(self.data['type'])

#### Step 5. Update the antchaos/common/data_formatter/ant_data_formatter.py**

We will have to update the ant_workflow_template_mapper dict in the antchaos/common/data_formatter/ant_data_formatter.py file.

This mapping helps us in mapping the type of chaos coming from ANT UI to an appropriate class in the chaos_params.py file.

We will have to update the "format_ant_chaos_data()" method in the "ANTDataFormatter". The method originally has only 3 types of chaoses:

        ant_workflow_template_mapper = {
            'Network Latency chaos': 'network-latency',
            'Pod Chaos': 'pod-delete',
            'Packet Loss Chaos': 'packet-loss',
        }

We will update it to:

        ant_workflow_template_mapper = {
            'Network Latency chaos': 'network-latency',
            'Pod Chaos': 'pod-delete',
            'Packet Loss Chaos': 'packet-loss',
            'Pod Memory Hog': 'pod-memory-hog'
        }

Once you are done with above steps you will have to push your change to gerrit for review.

git add antchaos/ant/test_scripts/startchaos/script_meta_data/UI_Columns.yaml \
antchaos/common/chaoslib/chaos_params.py \
antchaos/common/data_formatter/ant_data_formatter.py
git commit -m "Add memory hog experiment"
git push origin HEAD:/refs/for/main