# Introduction - Azure AKS Cilium EE Install Guide
This guide will help you to do a basic Cilium installation with some additional, advanced features:
- Cilium Enterprise
- KubeProxyReplacement (mode: `strict` or `true` if 1.14.x)
- IPv4-only
- Overlay routing with VXLAN-based encapsulation
- Hubble & Hubble Relay
- Cilium DNS Proxy HA
- Cilium Enterprise Dashboards
- Hubble UI Enterprise

In additional to the Cilium related tools, we'll install the following dependencies:
- Cert-Manager
- Grafana
- Kube-Prometheus-Stack

# Table of Content
[TOC]

# Prequisites
## General
- Ensure to have `kubectl` & `helm` CLI installed
- Deploy the AKS cluster with the no cni flag option --network-plugin none assuming you are using the az cli to build the AKS cluster
- Optional:
- Nginx ingress controller installed
- CSI plugin installed & `StorageClass` applied (used for Isovalent demo jobs-app)

## Pull Images / Charts

### Helm Chart Sources

helm repo add isovalent https://helm.isovalent.com
helm repo add jetstack https://charts.jetstack.io
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

### Optional: Pull Helm Charts (Optional)

Only applicable if you're using a local repository. Skip this section if your cluster(s) can pull charts from the internet.

helm pull isovalent/cilium --version 1.13.7
helm pull isovalent/cilium-dnsproxy --version 1.13.3
helm pull isovalent/hubble-ui --version 0.24.0
helm pull isovalent/hubble-enterprise --version 1.11.2
helm pull isovalent/jobs-app --version 0.8.0
helm pull jetstack/cert-manager --version v1.12.2
helm pull grafana/grafana --version 6.57.4
helm pull prometheus-community/kube-prometheus-stack --version 47.3.0

### Optional: Pull and Mirror Images (Optional)

Only applicable if you're using a local repository. Skip this section if your cluster(s) can pull images from the internet.

Helper script to extract image names (after running the provided `helm pull` commands from above):

# Get `yq_darwin_arm64` from https://github.com/mikefarah/yq
for chart in $(ls *.tgz); do
    echo $chart
    helm template $chart | ./yq_darwin_arm64 -Nr '[.. | select(has("image")).image]' | sed -ne '/^- / s/^- "\(.*\)"$/\1/p' | sort -u
done

# Define image array:
IMAGES=(
# cilium-dnsproxy
quay.io/isovalent/cilium-dnsproxy:v1.13.2
# Cilium
quay.io/isovalent/cilium:v1.13.4-cee.1
quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd
quay.io/isovalent/hubble-relay:v1.13.4-cee.1
quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
quay.io/isovalent/operator-generic:v1.13.4-cee.1
quay.io/cilium/startup-script:62093c5c233ea914bfa26a10ba41f8780d9b737f
quay.io/isovalent/clustermesh-apiserver:v1.13.4-cee.1
quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3
# Hubble-UI
cgr.dev/chainguard/oauth2-proxy:7.4.0@sha256:54857c70321004d3e0b8b8d7208ce5422c1f71c18db5be02d2543f187ea8e8e2
quay.io/isovalent/hubble-ui-enterprise-backend:v0.22.0
quay.io/isovalent/hubble-ui-enterprise:v0.22.0
# Hubble-Enterprise
quay.io/isovalent/hubble-enterprise:v1.10.5
quay.io/isovalent/hubble-enterprise-metadata:current
quay.io/isovalent/hubble-export-stdout:v1.0.3
quay.io/isovalent/hubble-export-fluentd:v1.5.1
quay.io/isovalent/hubble-export-s3:v1.0.0
# Isovalent Demos
# Jobs-App
quay.io/isovalent/jobs-app-coreapi:v0.8.0
quay.io/isovalent/jobs-app-crawler:v0.8.0
quay.io/isovalent/jobs-app-jobposting:v0.8.0
quay.io/isovalent/jobs-app-loader:v0.8.0
quay.io/isovalent/jobs-app-recruiter:v0.8.0
quay.io/isovalent/jobs-app-resumes:v0.8.0
# & 2 unknown Kafka Strimzi Operator images
# Star Wars
docker.io/cilium/starwars:latest
docker.io/tgraf/netperf:latest
# Cert-Manager
quay.io/jetstack/cert-manager-controller:v1.12.2
quay.io/jetstack/cert-manager-webhook:v1.12.2
quay.io/jetstack/cert-manager-cainjector:v1.12.2
quay.io/jetstack/cert-manager-acmesolver:v1.12.2
quay.io/jetstack/cert-manager-ctl:v1.12.2
# Grafana
docker.io/grafana/grafana:9.5.5
docker.io/bats/bats:v1.4.1
docker.io/curlimages/curl:7.85.0
docker.io/library/busybox:1.31.1
quay.io/kiwigrid/k8s-sidecar:1.24.3
docker.io/grafana/grafana-image-renderer:latest
quay.io/isovalent-dev/jwks-tools:latest
# Kube-Prometheus-Stack
registry.k8s.io/kube-state-metrics/kube-state-metrics:2.9.2
quay.io/brancz/kube-rbac-proxy:v0.14.0
quay.io/prometheus/node-exporter:v1.6.0
quay.io/prometheus/alertmanager:v0.25.0
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
quay.io/prometheus-operator/prometheus-operator:v0.66.0
quay.io/prometheus-operator/prometheus-config-reloader:v0.66.0
quay.io/thanos/thanos:v0.31.0
quay.io/prometheus/prometheus:v2.45.0
)
# Pull all images:
for IMAGE in ${IMAGES[@]}; do
  docker pull --platform x86_64 $IMAGE
  # Something like this should to the trick to directly mirror the images to the custom registry:
  #skopeo copy --multi-arch all docker://$IMAGE docker://harbor.example.com/project/$IMAGE
done

# Environment Validation
SSH to a worker node to validate some underlying infrastructure things:

# Check SELinux status
sestatus
# Containerd & Kubelet Systemd service status
systemctl status kubelet
systemctl status containerd
# Software versions
cat /etc/os-release
uname -r
crictl info
rpm -qa | grep kube
# Show running containers/pods
crictl ps -a
# Check if `cgroup2` is present
grep cgroup /proc/filesystems
# Validate containerd cgroup2 configuration (source: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd)
grep "SystemdCgroup" /etc/containerd/config.toml

# Installation
## Dependency CRDs

# Install Cert-Manager CRDs
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.crds.yaml
# Install Prometheus CRDs
kubectl apply -f https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.66.0/stripped-down-crds.yaml

## Replace Kube-Proxy wth Cilium Kube-Proxy-Replacement (Optional)
Should you wish to consume Cilium Ingress or GatewayAPI and benefit from our eBPF dataplane then you will need to replace Kube-Proxy (based on IPTables) with our Kube-Proxy-Replacement (KPR) feature following the guide below.

https://docs.isovalent.com/operations-guide/features/disabling-kube-proxy-azure/index.html

After Kube-Proxy has been removed, restart the nodes.

## Cilium
Create `ca-certs.sh` helper script to generate `cilium-ca` cert that's used for Hubble/ClusterMesh internal certificates (**important**: Run this script on Linux!):

#!/bin/bash

set -euo pipefail

# Script to generate Cluster Mesh Certificate Authority certificate and key shared in each cluster
#
# The base64 encoded certificates and keys are used in Helm values file for each Cluster.
#
# Requirements:
#   * openssl
#
# Usage:
#   ./ca-certs.sh up
#
# Cleanup:
#   ./ca-certs.sh down

CILIUM_CA_NAME="cilium-ca"
CILIUM_CA_CRT_FILENAME="${CILIUM_CA_NAME}-crt.pem"
CILIUM_CA_KEY_FILENAME="${CILIUM_CA_NAME}-key.pem"

function down() {
  # Delete the certificates and private keys.
  rm -f "${CILIUM_CA_CRT_FILENAME}" "${CILIUM_CA_KEY_FILENAME}"

}

function info() {
    echo "==> ${1}"
}

function up() {
  # Generate a private key and a certificate for the certificate authority.
  info "Creating a certificate authority..."
  if [[ ! -f "${CILIUM_CA_KEY_FILENAME}" ]];
  then
      openssl genrsa -out "${CILIUM_CA_KEY_FILENAME}" 4096
  fi
  if [[ ! -f "${CILIUM_CA_CRT_FILENAME}" ]];
  then
      openssl req -x509 \
          -days 3650 \
          -key "${CILIUM_CA_KEY_FILENAME}" \
          -new \
          -nodes \
          -out "${CILIUM_CA_CRT_FILENAME}" \
          -sha256 \
          -subj "/CN=${CILIUM_CA_NAME}"
  fi

  # Grab the certificates and private keys into environment variables.
  BASE64_ENCODED_CA_CRT="$(openssl base64 -A < ${CILIUM_CA_CRT_FILENAME})"
  BASE64_ENCODED_CA_KEY="$(openssl base64 -A < ${CILIUM_CA_KEY_FILENAME})"
}

function warn() {
  echo "(!) ${1}"
}

case "${1:-""}" in
  "down")
    down
    ;;
  "up")
    up
    ;;
  *)
    warn "Please specify one of 'up' or 'down'."
    exit 1
    ;;
esac

Generate `cilium-ca` cert:

chmod +x ca-certs.sh
./ca-certs.sh up

Configure `cilium` CA issuer:

kubectl -n kube-system create secret tls cilium-ca \
  --dry-run=client \
  --cert=cilium-ca-crt.pem \
  --key=cilium-ca-key.pem \
  -o yaml | \
kubectl apply --server-side=true --force-conflicts -f -
echo "
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: cilium
  namespace: kube-system
spec:
  ca:
    secretName: cilium-ca
" | kubectl apply -f-

Create Cilium's Helm values file (`cilium-values.yaml`)

Remove the image.override: sections if you are pulling the images from the internet. Otherwise, include the images you have pulled down to your local repository.

# -- Generic Cilium Agent Configurations

# Enable/disable debug logging
debug:
  enabled: false
  # -- Configure verbosity levels for debug logging
  # This option is used to enable debug messages for operations related to such
  # sub-system such as (e.g. kvstore, envoy, datapath or policy), and flow is
  # for enabling debug messages emitted per request, message and connection.
  #
  # Applicable values:
  # - flow
  # - kvstore
  # - envoy
  # - datapath
  # - policy
  verbose: ~

# inform Cilium this is an AKS BYOCNI cluster
aksbyocni:
  enabled: true

# Enable Cilium Ingress for exposing services via the cloud NLBs)
ingress:
  enabled: true

# Configure unique cluster name & ID (used for ClusterMesh in the future)
cluster:
  name: ${CLUSTER_NAME}
  id: ${CLUSTER_ID}

# Configure IPAM/PodCIDR
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList:
    - ${POD_CIDR}
    clusterPoolIPv4MaskSize: 24

# Disable IPv6 support if IPv6 isn't used by the customer
ipv6:
  enabled: false

# Routing/encapsulation mode
tunnel: vxlan

# KubeProxyReplacement
kubeProxyReplacement: "strict or true, depending on 1.13 vs 1.14"
k8sServiceHost: ${EXTERNAL_KUBE_APISERVER_HOST}
k8sServicePort: ${EXTERNAL_KUBE_APISERVER_PORT}

# Optionally override the images:
image:
  override: CUSTOM_REGISTRY/quay.io/isovalent/cilium:v1.13.4-cee.1
#certgen:
#  image:
#    override: CUSTOM_REGISTRY/quay.io/cilium/certgen:v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd
#etcd:
#  image:
#    override: CUSTOM_REGISTRY/quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
#nodeinit:
#  image:
#    override: CUSTOM_REGISTRY/quay.io/cilium/startup-script:62093c5c233ea914bfa26a10ba41f8780d9b737f

# -- Monitoring and Flow Visibility

# Enable Cilium Hubble to gain visibility
hubble:
  enabled: true
  metrics:
    enableOpenMetrics: true
    enabled:
    # https://docs.cilium.io/en/stable/observability/metrics/#hubble-exported-metrics
    # Remove `;query` from the `dns` line for production -> bad metrics cardinality
    - dns:labelsContext=source_namespace,destination_namespace;query
    - drop:labelsContext=source_namespace,destination_namespace
    - tcp:labelsContext=source_namespace,destination_namespace
    - port-distribution:labelsContext=source_namespace,destination_namespace
    - icmp:labelsContext=source_namespace,destination_namespace;sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity
    - flow:sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity;labelsContext=source_namespace,destination_namespace
    - "httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction;sourceContext=workload-name|reserved-identity;destinationContext=workload-name|reserved-identity"
    - "policy:sourceContext=app|workload-name|pod|reserved-identity;destinationContext=app|workload-name|pod|dns|reserved-identity;labelsContext=source_namespace,destination_namespace"
    serviceMonitor:
      enabled: true
    dashboards:
      enabled: true
      namespace: hubble-ui
      annotations:
        grafana_folder: "Hubble"
  tls:
    enabled: true
    auto:
      enabled: true
      method: certmanager
      certManagerIssuerRef:
        group: cert-manager.io
        kind: Issuer
        name: cilium
  relay:
    image:
      override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-relay:v1.13.4-cee.1
    enabled: true
    tls:
      server:
        enabled: true
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true

# Enable Cilium Operator metrics
operator:
  image:
    override: CUSTOM_REGISTRY/quay.io/isovalent/operator-generic:v1.13.4-cee.1
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
# Operator Dashboards will be added with Cilium EE 1.14
#  dashboards:
#    enabled: true
#    annotations:
#      grafana_folder: "Cilium Operator"

# Enable Cilium Agent metrics
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true

# Cilium Agent Dashboards will be added with Cilium EE 1.14
#dashboards:
#  enabled: true
#  annotations:
#    grafana_folder: "Cilium Agent"

# -- Advanced Cilium Feature Configurations

# Enable Cilium agent's support for Cilium DNS Proxy HA
extraConfig:
  external-dns-proxy: "true"

# Optional: Cilium ClusterMesh
# Probably activated at a later point in time
# clustermesh:
#   useAPIServer: true
#   config:
#     enabled: true
#     clusters:
#     - name: peer-cluster-1
#       ips:
#       - <peer-cluster-1-clusterMesh-APIserver-lb-service-ip>
#       port: 2379
#     - name: peer-cluster-1
#       ips:
#       - <peer-cluster-2-clusterMesh-APIserver-lb-service-ip>
#       port: 2379
#   apiserver:
#     image:
#       override: CUSTOM_REGISTRY/quay.io/isovalent/clustermesh-apiserver:v1.13.4-cee.1
#     etcd:
#       image:
#         override: CUSTOM_REGISTRY/quay.io/coreos/etcd:v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3
#     metrics:
#       etcd:
#         enabled: true
#       serviceMonitor:
#         enabled: true
#     service:
#       type: LoadBalancer
#       annotations: {} # Add required AWS annotations
#     tls:
#       auto:
#         enabled: true
#         method: certmanager
#         certManagerIssuerRef:
#           group: cert-manager.io
#           kind: Issuer # or ClusterIssuer
#           name: <issuer-name-placeholder>

Install Cilium:

helm upgrade -i cilium isovalent/cilium \
  -n kube-system \
  --version 1.13.7 (to be confirmed) \
  -n kube-system \
  -f cilium-values.yaml

Verification:

kubectl get ds -n kube-system cilium
kubectl get pods -n kube-system -l k8s-app=cilium

Sources:
- https://docs.isovalent.com/operations-guide/installation/certificates.html#auto-generated-certificates-via-cert-manager
- https://docs.isovalent.com/operations-guide/installation/clean-install.html

## Cilium DNS Proxy HA (Optional)

Remove the cilium-dns-proxy-ha-values.yaml section if you are pulling the images from the internet. Otherwise, include the image configuration below.

Create Helm values file for Cilium DNS Proxy HA (`cilium-dns-proxy-ha-values.yaml`):

image:
  override: CUSTOM_REGISTRY/quay.io/isovalent/cilium-dnsproxy:v1.13.2

Install Cilium DNS Proxy HA:

helm upgrade -i cilium-dnsproxy isovalent/cilium-dnsproxy \
  --version 1.13.2 \
  -n kube-system \
  -f cilium-dns-proxy-ha-values.yaml

Verification:

kubectl rollout status -n kube-system ds/cilium-dnsproxy --watch

Sources:
- https://docs.isovalent.com/operations-guide/features/networking/dnsproxy-ha/index.html

## Cert-Manager

Remove the cert-manager-values.yaml section if you are pulling the images from the internet. Otherwise, include the image configuration below.

Create Helm values file for Cert-Manager (`cert-manager-values.yaml`):

# Either work with the registry value or prepend the custom registry URL to the repository. Don't to both!
image:
  #registry: CUSTOM_REGISTRY
  repository: CUSTOM_REGISTRY/quay.io/jetstack/cert-manager-controller
webhook:
  image:
    #registry: CUSTOM_REGISTRY
    repository: CUSTOM_REGISTRY/quay.io/jetstack/cert-manager-webhook
cainjector:
  image:
    #registry: CUSTOM_REGISTRY
    repository: CUSTOM_REGISTRY/quay.io/jetstack/cert-manager-cainjector
acmesolver:
  image:
    #registry: CUSTOM_REGISTRY
    repository: CUSTOM_REGISTRY/quay.io/jetstack/cert-manager-acmesolver
startupapicheck:
  image:
    #registry: CUSTOM_REGISTRY
    repository: CUSTOM_REGISTRY/quay.io/jetstack/cert-manager-ctl

Install Cert-Manager:

helm upgrade -i cert-manager jetstack/cert-manager \
  --create-namespace \
  --skip-crds \
  -n cert-manager \
  --version v1.12.2 \
  -f cert-manager-values.yaml

Source:
- https://cert-manager.io/docs/installation/helm/

## Hubble-UI
Create `hubble-ui` Namespace and copy over `cilium-ca` secret:

# Create the hubble-ui NS:
kubectl create namespace hubble-ui --dry-run=client -o yaml | kubectl apply -f -

# Import the used cilium-ca from kube-system to hubble-ui NS:
kubectl -n kube-system get secrets cilium-ca --template '{{ index .data "tls.crt" }}'  | base64 --decode > cilium-ca.crt
kubectl -n hubble-ui create secret generic cilium-ca \
--save-config \
--dry-run=client \
--from-file=tls.crt=cilium-ca.crt \
-o yaml | \
kubectl apply -f -

# Create the JWKS for Hubble-UI's authentication on Grafana:
JWKS_DICRECTORY=jwks
if [[ ! -d "$JWKS_DICRECTORY" ]]
then
  mkdir -p $JWKS_DICRECTORY
  docker run -v $(pwd)/$JWKS_DICRECTORY:/usr/src/app/jwks quay.io/isovalent-dev/jwks-tools create hubble-ui.json
  docker run -v $(pwd)/$JWKS_DICRECTORY:/usr/src/app/jwks quay.io/isovalent-dev/jwks-tools public hubble-ui.json > $JWKS_DICRECTORY/hubble-ui-public.json
fi

# Create CM/secrets to access Grafana:
kubectl create configmap -n monitoring hubble-ui-grafana-public-jwks \
  --dry-run=client \
  --from-file=jwks=./jwks/hubble-ui-public.json \
  -o yaml | \
kubectl apply --server-side=true --force-conflicts -f -
export GRAFANA_AUTH_USER=hubble-viewer
kubectl create secret -n hubble-ui generic hubble-ui-grafana-secret \
  --dry-run=client \
  --from-literal=endpoint=http://grafana.monitoring.svc.cluster.local. \
  --from-literal=auth_token="$(docker run -v $(pwd)/$JWKS_DICRECTORY:/usr/src/app/jwks quay.io/isovalent-dev/jwks-tools jwt hubble-ui.json $GRAFANA_AUTH_USER)" \
  -o yaml | \
kubectl apply --server-side=true --force-conflicts -f -

Create Hubble-UI values file (`hubble-ui-values.yaml`):

Remove the image.override: sections if you are pulling the images from the internet. Otherwise, include the images you have pulled down to your local repository.

relay:
  address: "hubble-relay.kube-system.svc.cluster.local."

  tls:
    enabled: true
    ca:
      source: secret
      secret:
        name: cilium-ca
        key: tls.crt

ingress:
  annotations: {}
  enabled: true
  host: hubble-ui.<DOMAIN-POINTING-TO-CLUSTER-INGRESS>
  ingressClassName: cilium
  tls:
    enabled: true
    secretName: "hubble-ui-tls"
  v1: true

#oauth2:
#  proxy:
#    image:
#      override: CUSTOM_REGISTRY/cgr.dev/chainguard/oauth2-proxy:7.4.0
backend:
  image:
    override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-ui-enterprise-backend:v0.22.0
frontend:
  # -- Hubble-ui frontend image.
  image:
    override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-ui-enterprise:v0.22.0

grafana:
  enabled: true
  namespace: "hubble-ui"
  prometheusDatasourseUid: "Prometheus"

Installation:

helm upgrade -i hubble-ui isovalent/hubble-ui \
  --version 0.24.0 \
  --namespace hubble-ui \
  --create-namespace \
  --values hubble-ui-values.yaml

Verification:

kubectl get pods -n hubble-ui
kubectl port-forward -n hubble-ui svc/hubble-ui 12000:80
# ... and access the UI via browser: http://localhost:12000/

Sources:
- https://docs.isovalent.com/operations-guide/features/hubble-ui/install.html
- https://docs.isovalent.com/operations-guide/monitoring/hubble-ui-grafana.html

## Hubble-Enterprise

Remove the image.override: sections if you are pulling the images from the internet. Otherwise, include the images you have pulled down to your local repository.

Used for Hubble flow event export to ElasticSearch.

Create values file for Hubble Enterprise (`hubble-enterprise-values.yaml`):

hostNetwork: false
enterprise:
  enabled: false
#  image:
#    override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-enterprise:v1.10.5
#  metadataImage:
#    override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-enterprise-metadata:current
hubbleEnterpriseOperator:
  enabled: false
export:
  mode: "fluentd"
#  stdout:
#    image:
#      override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-export-stdout:v1.0.3
  filenames:
  - hubble.log
  extraEnv:
    - name: ELASTICSEARCH_HOST
      valueFrom:
        secretKeyRef:
          name: hubble-enterprise-elastic-credentials
          key: host
    - name: ELASTICSEARCH_PORT
      valueFrom:
        secretKeyRef:
          name: hubble-enterprise-elastic-credentials
          key: port
    - name: ELASTICSEARCH_USER
      valueFrom:
        secretKeyRef:
          name: hubble-enterprise-elastic-credentials
          key: user
    - name: ELASTICSEARCH_PASSWORD
      valueFrom:
        secretKeyRef:
          name: hubble-enterprise-elastic-credentials
          key: password
  fluentd:
    image:
      override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-export-fluentd:v1.5.1
    output: |-
      @type elasticsearch
      host "#{ENV['ELASTICSEARCH_HOST']}"
      port "#{ENV['ELASTICSEARCH_PORT']}"
      user "#{ENV['ELASTICSEARCH_USER']}"
      password "#{ENV['ELASTICSEARCH_PASSWORD']}"
      # Names indexes fluentd.hubble.log and fluentd.fgs.log
      index_name "fluentd.${tag}"
      scheme https
      ssl_verify false
      verify_es_version_at_startup false
      validate_client_version false
      default_elasticsearch_version 7
#  s3:
#    image:
#      override: CUSTOM_REGISTRY/quay.io/isovalent/hubble-export-s3:v1.0.0
  grafana:
    dashboards:
      enabled: true

Create secret containing the ElasticSearch credentials:

kubectl -n kube-system create secret generic hubble-enterprise-elastic-credentials \
  --from-literal host="<ES_HOST>" \
  --from-literal port="<ES_PORT>" \
  --from-literal user="<ES_USER>" \
  --from-literal password="<ES_PASSWORD>"

Install Hubble-Enterprise:

helm upgrade -i hubble-enterprise isovalent/hubble-enterprise \
  -n kube-system \
  --version 1.10.5 \
  -f hubble-enterprise-values.yaml

Sources:
- https://docs.isovalent.com/operations-guide/features/tetragon-security/event-export/elastic.html

## Grafana

Remove the image.repository: sections if you are pulling the images from the internet. Otherwise, include the images you have pulled down to your local repository.

Create values file for Grafana (`grafana-values.yaml`):

adminPassword: <generate-top-secret-pw>
sidecar:
  image:
    repository: CUSTOM_REGISTRY/quay.io/kiwigrid/k8s-sidecar
#imageRenderer:
#  image:
#    repository: CUSTOM_REGISTRY/docker.io/grafana/grafana-image-renderer
  dashboards:
    enabled: true
    label: grafana_dashboard
    labelValue: "1"
    searchNamespace: ALL
    folderAnnotation: grafana_folder
    provider:
      foldersFromFilesStructure: true
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local.:9090
      access: proxy
      isDefault: true
image:
  repository: CUSTOM_REGISTRY/docker.io/grafana/grafana
testFramework:
  image: CUSTOM_REGISTRY/docker.io/bats/bats
downloadDashboardsImage:
  repository: CUSTOM_REGISTRY/docker.io/curlimages/curl
initChownData:
  image:
    repository: CUSTOM_REGISTRY/docker.io/library/busybox
persistence:
  enabled: true
  size: 2Gi
  storageClassName: aws-efs
grafana.ini:
  security:
    allow_embedding: true
  auth.jwt:
    enabled: true
    enable_login_token: true
    url_login: true
    username_claim: sub
    header_name: X-Forwarded-Access-Token
    auto_sign_up: false
    jwk_set_file: /var/lib/grafana/jwks/hubble-ui-public.json
extraConfigmapMounts:
  - name: hubble-ui-grafana-public-jwks
    mountPath: /var/lib/grafana/jwks/hubble-ui-public.json
    subPath: jwks
    configMap: hubble-ui-grafana-public-jwks
    readOnly: true
ingress:
  enabled: true
  ingressClassName: cilium
  hosts:
  - grafana.<DOMAIN-POINTING-TO-CLUSTER-INGRESS>

Install Grafana:

helm upgrade -i grafana grafana/grafana \
  --create-namespace \
  -n monitoring \
  --version 6.57.4 \
  -f grafana-values.yaml

Sources:
- https://github.com/grafana/helm-charts

## Kube-Prometheus-Stack

Remove the global.imageRegistry: sections if you are pulling the images from the internet. Otherwise, include the images you have pulled down to your local repository.

Create kube-prometheus-stack values file (`prometheus-values.yaml`):

global:
  imageRegistry: "harbor.example.com/project"
# Disable Grafana since we install it via official Grafana Labs Helm chart
grafana:
  enabled: false
# Disable kube-proxy because we use Cilium KubeProxyReplacement
kubeProxy:
  enabled: false
# The following components are disable because we didn't configure kubeadm to expose those metric endpoints to far:
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
# Override images:
nodeExporter:
  enabled: true
  #kubeRBACProxy:
  #  image:
  #    registry: CUSTOM_REGISTRY/quay.io
#prometheus-node-exporter:
#  image:
#    registry: CUSTOM_REGISTRY/quay.io
#    repository: prometheus/node-exporter
kubeStateMetrics:
  enabled: false
alertmanager:
  ingress:
    enabled: true
    ingressClassName: cilium
    hosts:
    - alertmanager.<DOMAIN-POINTING-TO-CLUSTER-INGRESS>
    paths:
    - /
#  alertmanagerSpec:
#    image:
#      registry: CUSTOM_REGISTRY/quay.io
#      repository: prometheus/alertmanager
prometheus:
  ingress:
    enabled: true
    ingressClassName: cilium
    hosts:
    - prometheus.<DOMAIN-POINTING-TO-CLUSTER-INGRESS>
    paths:
    - /
  prometheusSpec:
#    image:
#      registry: CUSTOM_REGISTRY/quay.io
#      repository: prometheus/prometheus
    # We only deploy a single Prometheus. Therefore this one should watch for all CRs regardless of the CR annotations.
    ruleSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false
    scrapeConfigSelectorNilUsesHelmValues: false
#prometheusOperator:
#  image:
#    registry: CUSTOM_REGISTRY/quay.io
#    repository: prometheus-operator/prometheus-operator
#  prometheusConfigReloader:
#    image:
#      registry: CUSTOM_REGISTRY/quay.io
#      repository: prometheus-operator/prometheus-config-reloader
  #thanosImage:
  #  registry: CUSTOM_REGISTRY/quay.io
#  admissionWebhooks:
#    patch:
#      image:
#        registry: CUSTOM_REGISTRY/registry.k8s.io
#        repository: ingress-nginx/kube-webhook-certgen
#thanosRuler:
#  thanosRulerSpec:
#    image:
#      registry: CUSTOM_REGISTRY/quay.io

Install the Prometheus stack:

helm upgrade -i kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --create-namespace \
  --skip-crds \
  -n monitoring \
  --version 47.3.0 \
  -f prometheus-values.yaml

Sources:
- https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

## Cilium Enterprise Dashboards (Optional)
(Not required in case Hubble-UI's Grafana integration is activated.)
Install Cilium Enterprise dashboards for Grafana:

helm upgrade -i cilium-ee-dashboards isovalent/cilium-ee-dashboards \
  --version 0.1.2 \
  -n monitoring \
  --create-namespace

Sources:
- https://docs.isovalent.com/operations-guide/monitoring/cilium-ee-dashboards.html

## Demo

### Lightweight Star Wars Demo

Remove CUSTOM_REGISTRY/ if you are pulling your images from the internet. Otherwise update the manifest with your local repository path.

Create the Star Wars demo manifest (`star-wars-demo.yaml`):

# https://docs.cilium.io/en/stable/gettingstarted/demo/#getting-started-with-the-star-wars-demo
---
apiVersion: v1
kind: Namespace
metadata:
  name: star-wars-demo
---
apiVersion: v1
kind: Service
metadata:
  name: deathstar
  namespace: star-wars-demo
  labels:
    app.kubernetes.io/name: deathstar
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    org: empire
    class: deathstar
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deathstar
  namespace: star-wars-demo
  labels:
    app.kubernetes.io/name: deathstar
spec:
  replicas: 2
  selector:
    matchLabels:
      org: empire
      class: deathstar
  template:
    metadata:
      labels:
        org: empire
        class: deathstar
        app.kubernetes.io/name: deathstar
    spec:
      containers:
      - name: deathstar
        image: CUSTOM_REGISTRY/docker.io/cilium/starwars
---
apiVersion: v1
kind: Pod
metadata:
  name: tiefighter
  namespace: star-wars-demo
  labels:
    org: empire
    class: tiefighter
    app.kubernetes.io/name: tiefighter
spec:
  containers:
  - name: spaceship
    image: CUSTOM_REGISTRY/docker.io/cilium/json-mock
    command: ["/bin/sh"]
    args: ["-c", "while true; do curl -s -XPOST http://deathstar/v1/request-landing; curl -s https://disney.com; curl -s https://swapi.dev/api/starships; sleep 5; done"]
---
apiVersion: v1
kind: Pod
metadata:
  name: xwing
  namespace: star-wars-demo
  labels:
    app.kubernetes.io/name: xwing
    org: alliance
    class: xwing
spec:
  containers:
  - name: spaceship
    image: CUSTOM_REGISTRY/docker.io/cilium/json-mock
    command: ["/bin/sh"]
    args: ["-c", "while true; do curl -s -XPOST http://deathstar/v1/request-landing; curl -s https://disney.com; curl -s https://swapi.dev/api/starships; sleep 5; done"]

Apply it:

kubectl apply -f star-wars-demo.yaml

### Isovalent DNS Visibility CiliumClusterwideNetworkPolicy

Create the DNS visibility CCNP manifest (`ccnp-cluster-dns-visibility.yaml`):

---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: cluster-dns-visibility
spec:
  endpointSelector: {}
#    matchExpressions:
#    - key: k8s:io.kubernetes.pod.namespace
#      operator: NotIn
#      values:
#      - kube-system
  egress:
  # Allow all egress traffic to not break stuff. Remove for zero trust approach on the cluster!
  - toEntities:
    - all
  # Allow DNS to coredns and matchPattern to activate DNS visibility
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

Apply it:

kubectl apply -f ccnp-cluster-dns-visibility.yaml

Sources:
- https://docs.isovalent.com/quick-start/connectivity_visibility.html#enabling-dns-visibility

### Isovalent HTTP Visibility CiliumClusterwideNetworkPolicy

Create the DNS visibility CCNP manifest (`ccnp-cluster-http-visibility.yaml`):

---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: cluster-http-visibility
spec:
  endpointSelector: {}
#    matchExpressions:
#    - key: k8s:io.kubernetes.pod.namespace
#      operator: NotIn
#      values:
#      - kube-system
  ingress:
  # Allow all ingress traffic to not break stuff. Remove for zero trust approach on the cluster!
  - fromEntities:
    - all
  - toPorts:
    - ports:
      - port: "80"
        protocol: "TCP"
      - port: "8080"
        protocol: "TCP"
      - port: "9080"
        protocol: "TCP"
      - port: "50051"
        protocol: "TCP"
      - port: "9200"
        protocol: "TCP"
      rules:
        http:
        - {}

Apply it:

kubectl apply -f ccnp-cluster-http-visibility.yaml

Sources:
- https://docs.isovalent.com/quick-start/connectivity_visibility.html#enabling-http-visibility

### Optional: Isoavalent Jobs-App
**Important:** The Isovalent Jobs-App Helm chart currently doesn't support to override the images.

Ensure to have a default `StorageClass` available:

kubectl get sc

Install the Isovalent job-app:

helm upgrade -i jobs-app isovalent/jobs-app \
  --create-namespace \
  -n jobs-app \
  --set networkPolicy.enabled=false \
  --version 0.8.0