.. Chaos Engineering master file, created by
sphinx-quickstart on Mon Dec 3 16:11:36 2018.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

Our vision <#>__
Understand the basics <#understand-the-basics>__
Prerequisites to use chaos engineering <#prerequisites-to-use-chaos-engineering>__
Prerequisites to use chaos Template <#prerequisites-to-use-chaos-template>__
Tutorial <#tutorials>__
Scenarios <#scenarios-to-test>__

=========================================================
Our vision

To bring Chaos engineering into ING and upscale the applications to reduce the Application outage time.

.. toctree::
:maxdepth: 2
:caption: Contents:

Understand the basics

The tool which we used to conduct the chaos engineering in ICHP is Kubemonkey.

kube-monkey is an implementation of Netflix's Chaos Monkey for Kubernetes clusters. It randomly deletes Kubernetes (k8s) pods in the cluster encouraging and validating the development of failure-resilient services.

kube-monkey works on an opt-in model and will only schedule terminations for Kubernetes (k8s) apps that have explicitly agreed to have their pods terminated by kube-monkey.

Prerequisites to use chaos engineering

1. The application should be containerized and runnable in ICHP
2. Possiblity to add Kubemonkey labels in the Application deployment yaml.
   Note: This is to identify the victim pod by kubemonkey
3. While starting the Kubemonkey, the user can mention the Namespace where the Application is running

Prerequisites to use chaos Template

1. Be onboarded on Azure Devops <https://theforge.ing.net/product/42426/documentation/latest/gettingstarted>_
2. Have a basic understanding of how Azure DevOps works by watching the CDaaS Bootcamp <https://theforge.ing.net/product/42426/documentation/latest/journey>_ and/or Pluralsight <https://www.pluralsight.com/courses/azure-functions-fundamentals>_
3. Have an ICHP namespace <https://confluence.ing.net/pages/viewpage.action?pageId=733195192>_ for your all your environments
4. Have a basic understanding of kubernetes/openshift pluralsight course <https://www.pluralsight.com/courses/kubernetes-getting-started>_
5. Have a basic understanding of how ICHP <https://confluence.ing.net/display/ICHP/FAQ>_ works

Tutorials

Getting started with Templates

This template is used to start the kubemonkey application and run it in the kubernetes cluster (ICHP).

Template dependencies

±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
| Parameter | Type | Required | Description | Default |
+++=+=====================++
|config.dryRun |boolean |Yes |will not kill pods, only logs behaviour |true |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.runHour |int |Yes | schedule start time in 24hr format |8 |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.startHour |int |Yes | pod killing start time in 24hr format |10 |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.endHour |int |Yes | pod killing stop time in 24hr format |16 |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.whitelistedNamespaces |string |Yes | pods in this namespace that opt-in will be killed | |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.blacklistedNamespaces |string |Yes | pods in this namespace that opt-in will be killed |kube-system |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.timeZone |string |Yes | time zone in DZ format |America/New_York|
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.debug.enabled |boolean |Yes | debug mode,need to be enabled to see debuging behaviour |true |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+
|config.debug.schedule_immediate_kill |boolean |Yes | immediate pod kill matching other rules apart from time |true |
±––––––––––––––––––-±–––-±–––––±––––––––––––––––––––––––––––±–––––––-+

Usage

.. code-block:: yaml
:linenos:

resources:
repositories:

• repository: chaos
  type: git
  name: 'IngOne/[P00173]-templates' #
  ref: master ### refers to the branch in the repository

stages:

Call on the 4 build templates:

#- push-to-checkmarx
#- build-app/maven
#- get-resources (optional)
#- build-docker-image

• template: api/build-app-container/chaostemplate.yml@chaos
  parameters:

  These are the minimum REQUIRED parameters for the three templates

  incomingFeed: 'P0xxxx-incoming-private' ### Fill in your artifact feed
  targetBinaryDirectory: 'web/target/' ### Fill in your target binary folder (commonly web/target/)
  targetConfigurationDirectory: 'build-ichp-configs/target/' ### Fill in your target config folder (commonly conf/target/)

  These are the minimum OPTIONAL parameters for the 4th template -> get-resources

  autoResources: true
  appName: 'TestMyApp'
  appId: '0548bc72-5fe8-4305-b08b-12a8ee8e95b9'
  purposeID: 'P0xxxx'

Scenarios to test

±–––––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+
|kube-monkey/kill-mode | The kube-monkey's default behavior is to kill only one pod in the cluster, but you can change it to add more: |
±–––––––––––±–––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+
| |kill-all | Kill every pod, no matter what is happening with a pod |
±–––––––––––±–––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+
| |fixed | Pick a number of pods you want to kill |
±–––––––––––±–––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+
| |Random-max-percent | to specify a maximum % with kill-value that can be killed. At the scheduled time, a uniform random specified % of the running pods will be terminated |
±–––––––––––±–––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+
| |fixed-percent | Kill a fixed percent of pods (e.g., 50%) |
±–––––––––––±–––––––––±–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-+

Who are we?

We are the Optimus Prime <https://cdportal.ing.net/teams/BEEnvironmentServicesMonitoring/tab/empl>_ team, and this is our ChaosTestingICHP templates code. You can reach us for any questions regarding The ChaosTestingICHP pipeline templates on our Mattermost channel.