# Creating a new Service

This tutorial aims to guide engineers from product teams through the creation and rollout of a new service to staging in our Kubernetes and ArgoCD setup.
We'll cover a few relevant topics and explore different parts of our Kubernetes setup.
But before we start, we must first explore some of the technologies, conventions, and resources here utilized in this document.

## 0 - Introduction

### 0.1 - Repository:

Our Kubernetes setup is located in the [k8s-manifests repo](https://github.com/onrunning/k8s-manifests).
This repository contains our Kubernetes setup.

### 0.2 - Technologies:

Besides Kubernetes, our setup is composed of two other major components:

#### 0.2.1 - Helm

Helm is the package manager for Kubernetes.
In our setup, Helm is used to package our services into Charts.
> **From the Helm Docs:** A chart is a collection of files that describe a related set of Kubernetes resources.

**More on Helm here:** https://helm.sh/docs/

----

#### 0.2.2 - Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
In our setup, Argo CD monitors changes in the `k8s-manifests` repository, more specifically in the `manifests` folder.
> **Important:** We encourage readers to get familiar with the basics of `Argo CD` and `Helm` before proceeding.

**More on Argo CD here:** https://argo-cd.readthedocs.io/en/stable/

### 0.3 - Folder structure:

> tree -L 1 .
.
├── CODEOWNERS
├── Makefile
├── README.md
├── bin
├── charts
├── docs
├── manifests
└── tools

Above, we have the root folder structure for the `k8s-manifests` repository.
The `charts` and `manifests` are the two most important folders.
These are also the folders we'll focus on for the length of this tutorial.
Here's a brief description of each of these folders:

**Charts:** It's responsible for holding the modules (Charts) for each of our services.
The modules within this folder are Helm Charts, which Argo CD uses to build and roll out our services to Kubernetes.

**Manifests:** This folder stores our Application Manifests and the current version (image tag) for each service. Using its GitOps capabilities, Argo CD monitors this folder and triggers the appropriate rollouts and updates when services are changed.
***For instance, this is how our deployments work:*** *A Github Action changes the version of a service in this folder and commits the changes. Argo CD does the rest.*

> **Important:** Every path in this document is relative to the root of our `k8s-manifests` repository. Before you get started, make sure you have a local copy.

### 0.4 - Naming convention:

At On, we have adopted the `kebab-case` naming convention for all of our Kubernetes components.
It's essential to follow this naming convention consistently since our documentation and tooling rely on everything being named using the `kebab-case` naming convention.
Failing to do so will certainly create problems when creating or rolling out a service.

### 0.5 - Goals:

By the end of this tutorial, you'll have learned how to:

- Create an Application Manifest for your services
- Create a Chart for your services
- Work and manage values and secrets
- Safely roll out your services to staging

### 0.6 - Disclaimer:

We'll cover the creation of a new service in staging environment, as this is the environment that should be **first rolled out**.

Nevertheless, the steps described in this tutorial can be easily transferred to other environments, **except for Preview.**

Setting up the Preview environment requires a few extra steps, which will be covered in a separate tutorial.

### 0.7 - Table of Contents

- In [section 1](1---Creating-the-Application-Manifest), we create an Application Manifest template for our new service and then define the values used by our newly created template.
- In [section 2](#2---Creating-the-Service-Chart), we look into the `lib` and `hello-world` charts and then create the Chart for our new service.
In [section 3](#3---Values-and-Secrets), we explore the use of `values` and `secrets` and discuss important configuration values used to set up our new service.
- In [section 4](#4---Rollout), we look into roll out and the steps we need to take in order to have our service up and running.

## 1 - Creating the Application Manifest

Argo CD uses the Application Manifest to locate and resolve each service's resources, such as its Chart, values, and secrets.
The Application Manifest also contains the metadata for naming and describing the service.

> **From the [ArgoCD docs](https://argo-cd.readthedocs.io/en/stable/core_concepts):** The Application Manifest defines a group of Kubernetes resources responsible for a service.

In this tutorial, we'll divide the creation of the Application Manifest in two separate steps:
- [In section 1.1](#11---Creating-an-Application-Manifest-Template), we'll create the Application Manifest template.
- [In section 1.2](#12---Application-Manifest-values), we'll define the values that will be handed to our template before it can be rendered into a valid Application Manifest.

### 1.1 - Creating an Application Manifest Template

When creating a new service, we first create its Application Manifest template.

#### Here's how we do it

To create an Application Manifest template for the `my-new-svc` service in Staging, we have to create the `./manifests/staging/bootstrap-business/templates/my-new-svc.yaml` file in `k8s-manifests` with the following contents:

> **Info:** We're only creating the Application Manifest in Staging to keep things simple. However, you must create an Application Manifest template for every environment where you intend to roll out your service.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-new-svc
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: business
  source:
    repoURL: '[e-mail]:onrunning/k8s-manifests.git'
    targetRevision: HEAD
    path: charts/on/my-new-app
    helm:
      parameters:
      - name: 'image.tag'
        value: {{ index .Values "my-new-svc" "tag" }}
      valueFiles:
        - secrets-staging.yaml
        - values-staging.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated: {}
    syncOptions:
      - ApplyOutOfSyncOnly=true

> **Info:** In line 13, we reference a chart (`chart/on/my-new-svc`) which is yet to be created. We'll cover the chart creation in section [2 - Creating the Service Chart](#2---Creating-the-Service-Chart).

In the above code snippet, we have created an Application Manifest template.
Once rendered with the correct values, this template will produce a valid Application Manifest.

As you might have noticed, in line 17, our template expects the following value: `{{ index .Values "my-new-app" "tag" }}`.
This value is required so a valid Application Manifest can be generated.
The following section will cover how to set this value, [1.2 Application Manifest values](#12-Application-Manifest-values).

### 1.2 - Application Manifest values

In the previous section, we created the Application Manifest template.
We have also determined that we need certain values before creating a valid Application Manifest, which Argo CD can use to build and roll out our new service.

The following snippet of `pseudo-code` shows a high-level overview of how the Application Manifest is rendered and used:

applicationManifestTemplate = readFile(
    "./manifests/staging/bootstrap-business/templates/my-new-svc.yaml"
)

applicationManifest = renderTemplate(
    applicationManifestTemplate,
    { "my-new-app": { "tag": "f8019215b49daf87af46531215101ca0407ed689" }}
)

ArgoCD.sync(applicationManifest)

As seen in the above example, before the Application Manifest Template can be used to generate our Application Manifest, **it needs to be rendered with specific values**.

#### Here's how we do it

To add the aforementioned values, open the following file in `k8s-manifests`: `./manifests/staging/bootstrap-business/values.yaml`.

This file contains the values of each of our services' Application Manifest templates.
Next, we must add the Application Manifest values for our new service to the end of the file.

> **Info:** Like creating the Application Manifest template, you must perform this step for every environment to which you intend to roll out your service.

**See the example below:**
*Appended to:* `./manifests/staging/bootstrap-business/values.yaml`

my-new-app:
  tag: f8019215b49daf87af46531215101ca0407ed689

The YAML object we have appended to the file contains the values that will be used by Argo CD when creating an Application Manifest for our service in Staging.
> **Info:** This same snippet can be used for other services. Just remember to change the name and use a valid container image tag.

In this case, the only value set is `my-new-app.tag`, which defines the container (Docker) image tag that will be rolled out for our service.
> **Info:** You only need to manage the image tag when first creating your service. Our deployment pipeline should automatically handle subsequent updates.

### 1.3 - Working with different environments (optional)

In the previous sections, we have configured an Application Manifest and its values for running a service in Staging.

However, our Argo CD setup supports four different environments.
That means that you need to create both the Application Manifest and its values for every environment in which you wish to roll out your service.

Below, you'll find the list of environments and their root paths:
- **Preview**: `./manifests/preview/bootstrap-business/`
- **Staging**: `./manifests/staging/bootstrap-business/`
- **Pre-release**: `./manifests/pre-release/bootstrap-business/`
- **Production**: `./manifests/production/bootstrap-business/`

In the real world, our services are usually deployed in all four environments.
So, after rolling out to Staging and testing the new service, you will also need to create additional Application Manifests and values for the other environments in which you wish to roll it out.

**With the exception of the `preview` environment**, you can create an Application Manifest for other environments using the same steps described in sections [1.1](#11-Creating-an-Application-Manifest-Template) and [1.2](#12-Application-Manifest-values). Just remember to change the paths, values, and secret files according to each environment.

> **Important:** preview environments require a few extra steps and will be addressed in a separate tutorial.

## 2 - Creating the Service Chart

Now that we have created the files that inform ArgoCD about our new service, it's time to
create the service Chart.

In practical terms, the service Chart contains the building blocks that allow our setup (Helm, Argo CD, Kubernetes) to build, roll out, and run our service in a resilient and scalable fashion. There we can define important details such as `autoscaling`, `resource consumption`, `self-healing`, `load-balancing`, and many others.

### 2.1 - The lib-chart

Most of our service Charts are usually composed of the same Kubernetes components that use slightly different configurations to set up a service.
So, to simplify chart creation and reduce code duplication, our SREs have created a Chart of reusable Kubernetes components.
This utility Chart is called `lib` and is located in the `./charts/on/lib` folder.

The `lib` Chart is a utility Chart imported into all service Charts as a dependency.

**Here's an example extracted from the `Solidus` chart:**
`./charts/on/on-solidus/Chart.yaml`

apiVersion: v2
name: on-solidus
description: On's ecommerce application using solidus
type: application
version: 2.0.0
dependencies:
  - name: lib
    version: 2.2.7
    repository: file://../lib

> **Info:** In the above example, the Solidus Chart lists the `lib` Chart as a dependency. Charts can have multiple dependencies, always listed in the `Chart.yaml` file.

Helm requires dependencies to be built before a Chart can be used.
This is achieved with the `helm dependency build` command.
Running this command will create or update a `Chart.lock` file according to your dependencies and their versions.

**More on Helm dependency management**:
- [Helm dependency management](https://helm.sh/docs/helm/helm_dependency/)
- [helm dependency build command](https://helm.sh/docs/helm/helm_dependency_build/)

### 2.2 - The hello-world Chart

To facilitate the creation of new service Charts, the Dev Platform has created a new "model" Chart that provides sensible defaults for future services. It's called the `hello-world` Chart.
This model chart can be found in the following folder in the `k8s-manifests`: `./charts/on/hello-world`

It has a production-ready setup for small HTTP services, but it can also be extended to support more sophisticated services.

We can use the `hello-world` model chart to start our new service chart quickly.

### 2.3 Here's how we do it

Using the following commands, we'll create a new valid based on the `hello-world` model chart:

> **Important:** Make sure the path and name of our Chart match the one in the Application Manifest Template defined in section [1.1](#11-Creating-an-Application-Manifest-Template).

# copies the hello-world Chart to a new folder with the new service name
cp -r ./charts/on/hello-world ./charts/on/my-new-svc

# replaces occurrences of `hello-world` with our new service name
# since we follow the same dash-case naming convention everywhere -
#  every occurrence of `hello-world` will be replaced with `my-new-svc`.
sed -i '' 's/hello-world/my-new-svc/g' ./charts/on/my-new-svc/**/*.yaml

# as described in section 2.1, we need to build our Chart's dependencies
# this will update the Chart.lock file or create a new one if it doesn't exist.
helm dependency build ./charts/on/my-new-svc

Our new service chart folder `./charts/on/my-new-svc`, should have the following structure:

./charts/on/my-new-svc/
├── charts
│   └── lib-2.2.7.tgz
├── templates
|   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── Ingress.yaml
│   ├── secret.yaml
│   └── service.yaml
├── Chart.lock
├── Chart.yaml
├── secrets-production.yaml
├── secrets-staging.yaml
├── values-production.yaml
├── values-staging.yaml
└── values.yaml

Though the above commands produced a valid service chart, we must first check our Chart's values and secrets to ensure they match our new service's expectations.

We'll explore that in the next section: [3 - Values and Secrets](#3---Values-and-Secrets).

## 3 - Values and Secrets

Now that we have created our Application Manifest and the service Chart, it's time to check its configuration values and secrets.

### 3.1 - Values

The diagram in [section 2.3](#23-Heres-how-we-do-it) shows three different value files, `values.yaml`, `values-staging.yaml` and, `values-production.yaml`.

With the exception of the base `values.yaml` file, all value files are named after the environment they target, using the environment name as a suffix (`values-{env}.yaml`), ***e.g.*** `values-production.yaml`.

Environment-specific files like `values-production.yaml` use the attributes and values set in `values.yaml` as default and only override the defaults as needed.
The result is a large `values.yaml` file containing most of the attributes and values, followed by a set of very small environment-specific files that override only a few environment-specific values as needed.

Next, we'll look at a few critical attributes we need to know before rolling out our new service.

#### 3.1.1 - The Image attribute

The `image` is usually not environment-specific, and therefore, it is defined only once in the `values.yaml` base file.

**Example from our new service chart:**
`./charts/on/my-new-svc/values.yaml:`

image:
  repository: "924960756958.dkr.ecr.us-east-1.amazonaws.com/my-new-svc"
  pullPolicy: IfNotPresent

**Important image attributes**
`repository`: Container image repository in which our service's images are stored. You can find the repository for your service in the AWS ECR registry https://us-east-1.console.aws.amazon.com/ecr/private-registry/repositories

`pullPolicy:` Determines when Kubernetes should attempt to pull (download) the specified image. By default we use "IfNotPresent" for most use cases, which means the image is pulled only if it is not already present locally - **For more info:** https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

### 3.1.2 - The Service attribute

In our setup, we define a Kubernetes service the following way:

`./charts/on/my-new-svc/values.yaml:`

service:
  type: ClusterIP
  port: 3000

*In the above example we see the default service code generated for our new service in the defaults `values.yaml` file.*

`type:` This defines the type of Kubernetes service we're using. Different types of services are used for different use cases, but `ClusterIP` is most suitable for most applications, that's why we use it by default, **for more info:** https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

`port:` This tells the Kubernetes service which port to use when forwarding requests to your service. Always make sure that the port defined here matches the one exposed by your service and container.

### 3.1.3 - The Autoscaling attribute

The `autoscaling` attribute is environment-specific and tells Kubernetes how to scale your service.
As each environment handles different loads, this is an **environment-specific attribute** that should be set for each specific environment.

**Example from our new service chart:**
`./charts/on/my-new-svc/values-staging.yaml:`

autoscaling:
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 70

`minReplicas`: Minimum number of pods that will run your service.
`maxReplicas`: Maximum number of pods your service can scale to.
`targetCPUUtilizationPercentage:` CPU utilization threshold used by Kubernetes to autoscale services.

**Important:** When first deploying your service to Staging, use the default values set by the `hello-world` Chart. The values here should be discussed with the SREs before they can be changed.

#### 3.1.4 - The Ingress attribute

In our setup, the Ingress attribute configures external access to services running inside a Kubernetes cluster.

> **Info:** Ingress is a much bigger topic that has been simplified to a simple attribute in our setup. For more info: https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress

**Internal services:**

If you're creating an internal service that should not be exposed to the web (**e.g**, `cyclon`), remove all `ingress` definitions from the value files and feel free to skip the rest section.

**Public services:**

If your service needs to be exposed to the web or accessed externally in any way, you need to define an Ingress object.

The `ingress` attribute is also an environment-specific attribute as **different environments are exposed under different domains**, so this attribute must also be set up for every environment.

**Example for our new service chart:**
`./charts/on/my-new-svc/values-staging.yaml:`

...
ingress:
  hosts:
    - host: my-new-svc-staging.on.com
      paths:
        - path: /

As seen in the above example, the Ingress attribute allows us to define how our service can be accessed externally.

Many things are going on under the hood here. However, in simple terms, this Ingress definition enables a load balancer to forward external requests with the `Host` header equal to `my-new-svc-staging.on.com` to our `my-new-svc` service.

This exact definition can be used to set up other services; all you need to remember is to change the domain according to the target service and environment.

### 3.2 - Secrets

Since our service is intentionally very simple, all its values are stored in plain text; therefore, it has no secrets.
Nevertheless, if you want to learn how to manage and use secrets, we got your back!
Here's a special tutorial on how to work with secrets: [Managing Secrets](https://github.com/onrunning/k8s-manifests/blob/master/docs/Managing_Secrets.md)

## 4 - Rollout

### 4.1 - Validate your chart

The last step before rolling out our service is to validate the chart we created.

You can validate your chart using the following command:

helm secrets template . --values values-staging.yaml  --values secrets-staging.yaml

The result of the following command is a large multi-document Yaml containing the manifests for every Kubernetes component used for your service.

**To ensure your chart is valid, do the following:**
- 1 - make sure the generated YAML is valid; I recommend using the [yq tool](https://github.com/mikefarah/yq)
- 2 - check every generated Kubernetes manifest and ensure the values make sense

**Important:**

In the above example, we have validated our chart for Staging since this is the only environment we're rolling out this service to.

However, you should **always validate your services for every environment** you wish to roll out to. You can use that same command for other services but change the value and secret files according to the environment you wish to validate.

### 4.2 - Open a PR

Having validated our service, we are ready to commit the modified files and open a PR in the `k8s-manifests` repository.

**Here's an example of the PR creating the `hello-world` service:**
https://github.com/onrunning/k8s-manifests/pull/1089

While we aim to make service creation as straightforward as possible, a little back-and-forth is always expected to fine-tune the smaller configuration details. Nevertheless, the bulk of it should be now done.

### 4.3 - Checking your service in ArgoCD

As we've seen in the past sections, our Kubernetes services and their components are managed by ArgoCD. In addition to that, ArgoCD also provides a web application that allows users to search and inspect services as well as their inner Kubernetes components.

To visualize how our services run in ArgoCD and Kubernetes, we can access the ArgoCD web application present in each of our clusters:

- **Staging:** https://argocd-staging.on.com
- **Production:** https://argocd-production.on.com

With that in mind, we can return to our new service.
Once your PR has been merged, the new service should start running in staging in a matter of minutes. Then, we must ensure that our service runs properly in the staging cluster.

#### Here's how we do it

You can check your service by accessing the ArgoCD web application in the [staging cluster](https://argocd-staging.on.com). Below we're searching and inspecting the `compliance service`, but the same works for any other service:


Once you find your service, you can access the service page by clicking on your service's box:


The above image shows the service page for the `compliance service`. This is an example of a working service. As you can see, every internal component is **Healthy** and **Synced**.

A component or service with a status other than **Healthy** or **Progressing** indicates failure and should be immediately inspected. You can do that by clicking on the component and accessing its page. There, you'll find the component's logs, events, configuration, and status.

It's normal for Components to be briefly **OutOfSync** during rollouts and updates; however, if that persists for a long time, it most likely indicates a problem and that the component should be inspected.

When inspecting your newly created service, make sure that each of its internal components is also **Healthy** and **Synced** before proceeding to the next steps.

> **Important:** If once you inspect the service, you find any errors, reach out to the **Dev Platform** team.

### 4.4 - Rollout Instructions

In our setup, each environment is handled separately so that each environment can be individually rolled out.

That flexibility allows us to test services in lower environments without compromising production. The `Dev platform` team recommends rolling out services and testing them in Staging before moving on to other environments.

Once the service is load-tested and verified to work correctly in the staging Kubernetes environment, moving on to other environments should be okay.

> **Important:** Running load tests for services in Staging is strongly recommended as it makes it easier to detect bottlenecks and estimate resources upfront.

## Summary

It's been a pleasure working with you in this tutorial.
We have covered quite a lot of ground, so give yourself a pat on the back ❤️.

You should now have a clear idea of how to create a service in Staging, but perhaps more importantly, you should now feel a bit more familiar with our Kubernetes setup.

If you have any questions, feel free to reach out.