# 5G e2e NMS framework setup/program development guide

## Install x-k8s v1.3.2
1. Clone x-k8s and checkout to tag v1.3.2
2. Remember to change the option in extraVars

## Monitor install Prometheus and Grafana
monitor_enabled: true
grafana_password: "admin"

3. Install
`./x-k8s install`

Or, ask Jace.

## Environemt validation
When x-k8s is installed. Check if our prometheus deployment is running.
You should see following deployments.
1. alertmanger
2. prometheus-adapter
3. grafana
4. prometheus-operator

root@node1:~# kubectl  get deployment
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
pa-prometheus-adapter             1/1     1            1           30m
po-grafana                        1/1     1            1           30m
po-kube-state-metrics             1/1     1            1           30m
po-prometheus-operator-operator   1/1     1            1           30m

And following pods

root@node1:~# kubectl  get pod
NAME                                                 READY   STATUS    RESTARTS   AGE
alertmanager-po-prometheus-operator-alertmanager-0   2/2     Running   0          28m
pa-prometheus-adapter-b47df69d9-qcxqq                1/1     Running   0          28m
po-grafana-7d64cfbf64-mm6jp                          3/3     Running   0          29m
po-kube-state-metrics-6c6d45c94c-xtrnk               1/1     Running   0          29m
po-prometheus-node-exporter-p4jhj                    1/1     Running   0          29m
po-prometheus-operator-operator-569b4df996-hf2br     2/2     Running   0          29m
prometheus-po-prometheus-operator-prometheus-0       3/3     Running   1          28m

And you should be able get metrics and value from

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

This will return **a lot of message**

## Development Guide for Network Services

Here's an example deployment wich expose HTTP api for prometheus adapter.

1. Create deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: mjace/autoscale-demo-amd64:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080

And here's the source code for the sample-app,
You can take as example. Remember the follow the responce structure.
Remember to add the `# HELP` and `# TYPE`

var http = require('http');
var os = require('os');

var totalrequests = 0;

http.createServer(function(request, response) {
  totalrequests += 1

  response.writeHead(200);

  if (request.url == "/metrics") {
    response.end("# HELP http_requests_total_custom The amount of requests served by the server in total\n# TYPE http_requests_total_custom counter\nhttp_requests_total_custom " + totalrequests + "\n");
    return;
  }
  response.end("Hello! My name is " + os.hostname() + ". I have served "+ totalrequests + " requests so far.\n");
}).listen(8080)

2. Create service for our deployment

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
    release: po
  name: sample-svc
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

3. Check if the HTTP API from our deployment is accessable.

curl http://$(kubectl get service sample-svc -o jsonpath='{ .spec.clusterIP }')/metrics

# HELP http_requests_total_custom The amount of requests served by the server in total
# TYPE http_requests_total_custom counter
http_requests_total_custom 1

4. Create ServcieMonitor
Service monitor will allow prometheus operator to get our deployment's api

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: sample-app
    release: po
  name: sample-app
  namespace: default
spec:
  endpoints:
  - port: http
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: sample-app
      release: po

5. Check if prometheus is able to get our exposed data.
a. On terminal use kubectl to forward promutheus website

    kubectl port-forward svc/po-prometheus-operator-prometheus 9090

b. Use mobaXterm or other terminal tool with X11 forwarding.

    firefox localhost:9090/targets

You should see the metrics of sample deployment in the 'target' tab

## Prometheus API example for III NMS

1. On terminal use kubectl to forward promutheus website

kubectl port-forward svc/po-prometheus-operator-prometheus 9090

2. Call Prometheus HTTP API

curl 'http://localhost:9090/api/v1/query?query=http_requests_total_custom'

{
   "status":"success",
   "data":{
      "resultType":"vector",
      "result":[
         {
            "metric":{
               "__name__":"http_requests_total_custom",
               "endpoint":"http",
               "instance":"10.233.64.18:8080",
               "job":"sample-svc",
               "namespace":"default",
               "pod":"sample-app-6684b85666-7dm5b",
               "service":"sample-svc"
            },
            "value":[
               1584953911.557,
               "19"
            ]
         }
      ]
   }
}

---

## Pushgateway Installation and Usage

[Pushgateway](https://github.com/helm/charts/tree/master/stable/prometheus-pushgateway)

helm install --name pg --set serviceMonitor.enabled=true \
  --set serviceAccount.name=pushgateway \
  --set serviceMonitor.namespace=default \
  --set serviceMonitor.additionalLabels."release"=po \
  stable/prometheus-pushgateway

Change the label from **pg** to **po** of serviceMonitor

` kubectl edit servicemonitor pg-prometheus-pushgateway`

`release:pg` -> `release: po`

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2020-04-22T03:06:11Z"
  generation: 1
  labels:
    app: prometheus-pushgateway
    chart: prometheus-pushgateway-1.3.0
    heritage: Tiller
    release: po

## Push data to pushgateway

1. Check ClusterIP of pushgateway

root@node1:~# kubectl get svc pg-prometheus-pushgateway
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
pg-prometheus-pushgateway   ClusterIP   10.233.2.227   <none>        9091/TCP   5d3h

so the cluster IP is 10.233.2.227

we can push a sample metrics to this ip on our host machine

echo "cpu_utilization 20.25" | curl --data-binary @- http://10.233.2.227:9091/metrics/job/my_custom_metrics/instance/10.20.0.1:9000/provider/hetzner

or

echo "some_metric 3.14" | curl --data-binary @- http://10.233.2.227:9091/metrics/job/some_job

If you like to push metric from your own pod, replace the cluster IP by service name.

If you like to push metrics by your program, refer to [Pushing Metrics](https://prometheus.io/docs/instrumenting/pushing/)

[For the more information about URL of pushgateway](https://github.com/prometheus/pushgateway#url)

---

## Reference
1. [Horizontal Pod Autoscale with Custom Prometheus Metrics🚀](/IOFAdEv0S7mTeioOMxjw5Q)
2. [Prometheus Http API](https://prometheus.io/docs/prometheus/latest/querying/api/)