Using AWS NLB manually targeting an EKS Service exposing UDP traffic

Repost on medium : https://medium.com/@allamand/using-aws-nlb-manually-targeting-an-eks-service-exposing-udp-traffic-17053ecd8f52

Problem encountered with EKS 1.16

If we try to create a service of type Network Load Balancer (NLB) for UDP traffic we can get this error :

Error creating load balancer (will retry): failed to ensure load balancer for service default/test: Only TCP LoadBalancer is supported for AWS ELB

This is because the UDP support for NLB is more recent than the functionality developed inside kubernetes for creating NLB load balancers.

The bug is being report in this issue : #79523 and is currently investigated by AWS.

We are going to work arround this actual limitation

Using NodePort Kubernetes service

Meanwhile we can manually configure an NLB to point to our EKS instances, and configure a Kubernetes NodePort service instead of LoadBalancer.

In NodePort mode, every Instance will listen on a pre-defined port (range between 30000-32767) on each EC2 instances and Kubernetes will forward the traffic to the associate Kubernetes pods:

Example: exposing kube-dns with NLB

As an example, we are going to expose the Kubernetes core-dns pods through a manually created NLB. We choose core-dns, that is expose an UDP service on port 53.

Creating the kubernetes NodePort service

Those pods have label k8s-app: kube-dns so we can create a new service of type NodePort:

cat << EOF > service-kube-dns-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: service-kube-dns-nodeport
  namespace: kube-system
  labels:
    app.kubernetes.io/name: service-kube-dns-nodeport
spec:
  type: NodePort
  ports:
    - name: dns-udp
      port: 53
      targetPort: 53
      nodePort: 30053
      protocol: UDP
    - name: dns-tcp
      port: 53
      targetPort: 53
      nodePort: 30054
      protocol: TCP
  selector:
    k8s-app: kube-dns
EOF

Let's apply the service definition to the EKS cluster.

k apply -f service-kube-dns-nodeport.yaml

We have defined an UDP port 30053 that will be listening on each Instances, and forward to the kube-dns pods on port 53.

We have also defined a TCP port 30054 that will be used by the NLB when configuring the healthcheck port.

This service must be deployed in kube-system namespace as the coredns-pods are deployed there.

(Optional) Test internally the service works

At this stage, we have created the Kubernetes service, which is already listening on each instances. Before creating the NLB, we can check that this is working inside of the Kubernetes cluster:

We can uses a specific kubernetes pod eksutils-pod that may be useful to debug things in kubernetes.

Create the eksutils pod in the default namespace:

kubectl apply -f https://raw.githubusercontent.com/allamand/eksutils/master/eksutils-pod.yaml

Then we can connect inside the pod, and see if we are able to make dns resolution based on the internal services.

In the below example, For each of the Kubernetes nodes I request the grafana service in the metrics namespace. Change this to point to existing services you have in your cluster. It also check the conn ection on the TCP port 30054

$ kubectl exec -n default -ti eksutils-pod zsh
eksutils@eksutils-pod $ for x in $(k get nodes -o wide | awk '{print $6}' | grep -v INTERNAL); do echo $x ; dig @$x -p 30053 grafana.metrics.svc.cluster.local ; telnet $x 30054 ; done

This will make a dns lookup for a grafana service in metrics namespace, on each nodes on port 30053 UDP, and for each port try to connect on port 30054 TCP.

Create the Network Load Balancer

Get the subnets of your instances in the same vpc you have for your EKS cluster

VPC_ID=vpc-027f50fc9d05149f0
aws ec2 describe-instances --filters Name=network-interface.vpc-id,Values=$VPC_ID \
 --query 'Reservations[*].Instances[*].SubnetId' --output text | sort | uniq -c
   5 subnet-0378859fcc9e53fa6
   5 subnet-055b4b800624d3b99

Note: this is ok only if you have a dedicated vpc for your EKS cluster. If this is not the case, you need to filter to match only the instances you want.

First create the NLB using the subnets of your worker nodes

NLB_NAME=kube-dns-nlb
aws elbv2 create-load-balancer --name $NLB_NAME\
  --type network \
  --subnets subnet-0378859fcc9e53fa6 subnet-055b4b800624d3b99

Create the target Group for your NLB

TG_NAME=kube-dns-tg
aws elbv2 create-target-group --name $TG_NAME --protocol UDP --port 30053 --vpc-id $VPC_ID \
  --health-check-protocol TCP \
  --health-check-port 30054 \
  --target-type instance

Register Instances in the target group

We want to add every nodes which are parte of our EKS cluster as target for our NLB target-group. Get the list of instances

INSTANCES=$(kubectl get nodes -o json | jq -r ".items[].spec.providerID"  | cut -d'/' -f5)
IDS=$(for x in `echo $INSTANCES`; do echo Id=$x ; done | tr '\n' ' ')
echo $IDS

Register the instances :

TG_ARN=$(aws elbv2 describe-target-groups --query 'TargetGroups[?TargetGroupName==`kube-dns-tg`].TargetGroupArn' --output text)

aws elbv2 register-targets --target-group-arn $TG_ARN --targets $(echo $IDS)

Create a Listener for the target-group

LB_ARN=$(aws elbv2 describe-load-balancers --names $NLB_NAME --query 'LoadBalancers[0].LoadBalancerArn' --output text)
echo $LB_ARN

aws elbv2 create-listener --load-balancer-arn $LB_ARN \
  --protocol UDP --port 53 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

Check listener

aws elbv2 describe-listeners --load-balancer-arn $LB_ARN

Check Health of the targets

aws elbv2 describe-target-health --target-group-arn $TG_ARN

Should be uneahlthy until we configure the SG of the instances.

Configure Instances Security Groups

In order to allow the health check, we need to allow the port 30054 in the Security Groups of our instances to be reach by the IP of the NLB

Get security group from instances IDs for all instances

SGs=$(for x in $(echo $INSTANCES); do aws ec2 describe-instances --filters Name=instance-id,Values=$x \
 --query 'Reservations[*].Instances[*].SecurityGroups[0].GroupId' --output text ; done | sort | uniq)

Add Rule to the Security Group

for x in $(echo $SGs); do 
  echo SG=$x; 
  aws ec2 authorize-security-group-ingress --group-id $x --protocol tcp --port 30054 --cidr 192.168.0.0/16; 

  aws ec2 authorize-security-group-ingress --group-id $x --protocol udp --port 30053 --cidr 0.0.0.0/0 ; 
done 

Instead of oppening from 192.168.0.0/16 you can open for the real NLB Ip adresses using

NLB_NAME_ID=$(aws elbv2 describe-load-balancers --names $NLB_NAME --query 'LoadBalancers[0].LoadBalancerArn' --output text | awk -F":loadbalancer/" '{print $2}')
aws ec2 describe-network-interfaces \
 --filters Name=description,Values="ELB $NLB_NAME_ID" \
 --query 'NetworkInterfaces[*].PrivateIpAddresses[*].PrivateIpAddress' --output text

Test it

Once the targets are healthy, you can test the access. find the URL of your NLB and test it's access (be sure you have internet access for port 53)

LB_DNS=$(aws elbv2 describe-load-balancers --name $NLB_NAME --query 'LoadBalancers[0].DNSName' --output text

echo $LB_DNS

Test our newly expose domain name server with dig:

$ dig @$LB_DNS grafana.metrics.svc.cluster.local
; <<>> DiG 9.10.6 <<>> @kube-dns-nlb-a688e8ddf1200136.elb.us-east-1.amazonaws.com grafana.metrics.svc.cluster.local
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53980
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grafana.metrics.svc.cluster.local. IN	A

;; ANSWER SECTION:
grafana.metrics.svc.cluster.local. 5 IN	A	10.100.165.230

;; Query time: 98 msec
;; SERVER: 54.164.194.190#53(54.164.194.190)
;; WHEN: Wed May 13 19:27:47 CEST 2020
;; MSG SIZE  rcvd: 111

We can see here that we are able to request our exposed DNS server.

Automate target creation in our load balancer

Scaling Instances

If you add some EKS instances, you'll need to add thoses instances in your NLB target group in order to be able to spread the load on thoses instances too.

When Scaling down, the old instances will become unhealthy, and they'll automagically be deregistered and disapears from the target groups, so there will be no specific action to take in this case.

We want to find a way to automate the addon on new targets in the NLB when we are doing a scale out.

We can configure Amazon EC2 Auto Scaling to send events to CloudWatch Events whenever our Auto Scaling group scales.

The Idea here is that each time an instance is created by our auto scaling groups of our EKS cluster, then the instances are automatically added to the NLB target group.

Create a Lambda function to automate adding instance in the NLB

Create Lambda function which is going to add the instance added via the AutoScaling Group to the NLB Target Group

import json
import boto3
from pprint import pprint

LbName="kube-dns-nlb" #<- change accoring to your setup

print('Loading function')

elb = boto3.client('elbv2')

def find_lb_arn(name):
    # describe load balancer name
    lbs_list_response = elb.describe_load_balancers(Names=[name])
    if lbs_list_response['ResponseMetadata']['HTTPStatusCode'] == 200:
        print ("LBs list: " + ' '.join(p for p in [lb['LoadBalancerName']
                                       for lb in lbs_list_response['LoadBalancers']]))
        #We have only 1 lb    
        lbArn = lbs_list_response['LoadBalancers'][0]['LoadBalancerArn']
    else:
        print ("Describe lbs failed")
    return lbArn


def lambda_handler(event, context):
    print("AutoScalingEvent()")
    print("Debug Event data = " + json.dumps(event, indent=2))
    
    target_id = event['detail']['EC2InstanceId']
    print("We are going to add InstanceID = " + target_id)
    
    #Find load balancer arn
    lbArn = find_lb_arn(LbName)
    print ("lbArn="+lbArn)
    
    # Register targets
    targets_list = [dict(Id=target_id)]
  
    describe_tg_response = elb.describe_target_groups(LoadBalancerArn=lbArn)
    #pprint(describe_tg_response)
    tgId = describe_tg_response['TargetGroups'][0]['TargetGroupArn']
    print ("tgID = " + tgId)

    #Register target in targetGroup
    reg_targets_response = elb.register_targets(TargetGroupArn=tgId, Targets=targets_list)
    if reg_targets_response['ResponseMetadata']['HTTPStatusCode'] == 200:
        print ("Successfully registered targets")
    else:
        print ("Register targets failed")

You need to parameterize LbName to the name of the NLB you want to add the instance into

Create the Role for our Lambda function

NAME=add-instance-to-nlb
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')
ASSUMEPOLICY=$(echo -n '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}')

echo ACCOUNT_ID=$ACCOUNT_ID
echo ASSUMEPOLICY=$ASSUMEPOLICY

LAMBDA_ROLE_ARN=$(aws iam create-role \
  --role-name $NAME \
  --description "Role to allow Lambda function to manage NLB targets" \
  --assume-role-policy-document "$ASSUMEPOLICY" \
  --output text \
  --query 'Role.Arn')
echo $LAMBDA_ROLE_ARN

attach policy to the role

aws iam attach-role-policy \
  --role-name $NAME \
  --policy-arn arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess

LAMBDA_POLICY=$(echo -n '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:'; echo -n "$ACCOUNT_ID"; echo -n ':*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:'; echo -n "$ACCOUNT_ID"; echo -n ':log-group:/aws/lambda/'; echo -n "$NAME"; echo -n ':*"
            ]
        }
    ]
}')
echo $LAMBDA_POLICY

LAMBDA_POLICY_ARN=$(aws iam create-policy \
  --policy-name AWSLambdaBasicExecutionRole-$NAME \
  --policy-document "$LAMBDA_POLICY" \
  --output text \
  --query 'Policy.Arn')
echo $LAMBDA_POLICY_ARN

aws iam attach-role-policy \
  --role-name $NAME \
  --policy-arn $LAMBDA_POLICY_ARN

Create the Lambda function (I've created a zip with the previous lambda code)

aws lambda create-function \
    --function-name $NAME \
    --runtime python3.7 \
    --zip-file fileb://~/environment/add-instance-to-nlb.zip \
    --handler add-instance-to-nlb.lambda_handler \
    --role $LAMBDA_ROLE_ARN

test the function with the json test

{
  "version": "0",
  "id": "12345678-1234-1234-1234-123456789012",
  "detail-type": "EC2 Instance Launch Successful",
  "source": "aws.autoscaling",
  "account": "123456789012",
  "time": "yyyy-mm-ddThh:mm:ssZ",
  "region": "us-west-2",
  "resources": [
      "auto-scaling-group-arn",
      "instance-arn"
  ],
  "detail": {
      "StatusCode": "InProgress",
      "Description": "Launching a new EC2 instance: i-12345678",
      "AutoScalingGroupName": "my-auto-scaling-group",
      "ActivityId": "87654321-4321-4321-4321-210987654321",
      "Details": {
          "Availability Zone": "us-west-2b",
          "Subnet ID": "subnet-12345678"
      },
      "RequestId": "12345678-1234-1234-1234-123456789012",
      "StatusMessage": "",
      "EndTime": "yyyy-mm-ddThh:mm:ssZ",
      "EC2InstanceId": "i-1234567890abcdef0",
      "StartTime": "yyyy-mm-ddThh:mm:ssZ",
      "Cause": "description-text"
  }
}

You should have a normal error saying that:

 The following targets are not valid instances: 'i-1234567890abcdef0'"

Create CloudWatch Rule

Adapt the following Event pattern rule with the name of the AutoScaling Groups you need. In my case, I have 3 auto scaling group associated to my eks cluster.

EVENT_PATTERN=$(echo -n '{
  "source": [
    "aws.autoscaling"
  ],
  "detail-type": [
    "EC2 Instance Launch Successful"
  ],
  "detail": {
    "AutoScalingGroupName": [
      "eksctl-eksworkshop-eksctl-nodegroup-ng-spot-NodeGroup-1MCQMJAIUZCSS",
      "eks-f8b8de05-e964-8e64-5043-60449f530a2b",
      "eks-48b909fd-3aa4-c200-dcc0-cb8c5b637736"
    ]
  }
}')
echo $EVENT_PATTERN

Create the CloudWatch rule

aws events put-rule \
  --name $NAME \
  --event-pattern "$EVENT_PATTERN"

Get Lambda arn

LAMBDA_ARN=$(aws lambda get-function --function-name $NAME --query 'Configuration.FunctionArn' --output text)
echo $LAMBDA_ARN

Create Cloudwatch Event rule target

RULE_TARGET=$(echo -n '[
  {
    "Id": "1", 
    "Arn": "'; echo -n "$LAMBDA_ARN"; echo -n '"
  }
]')
echo $RULE_TARGET

Add the target to the event rule

aws events put-targets \
  --rule $NAME \
  --targets "$RULE_TARGET"

Add permission on the Lambda to be triggered by the Event

aws lambda add-permission \
  --function-name $NAME \
  --statement-id autoscaling-event-rule \
  --action 'lambda:InvokeFunction' \
  --principal events.amazonaws.com \
  --source-arn $(aws events describe-rule --name $NAME --query 'Arn' --output text)

Or Using the Console:

We Create a CloudWatch Rule in order to associate our AutoScaling Group with our Lambda Function.

Select Auto Scaling as event Patter, and select all your autoscaling groups you want to monitor

For the target, simply choose the Lambda function we just create.

Testing

You can now test Scaling out or scaling in your ASG, or simply terminate some instances, the ASG will send an event to the lambda each time a new instance is created in the acording ASGs, and the lambda register the new instance in the NLB target groups.

Check this if you need to manually add instances with the cli

Manually add instances

Here I will just show you how you can add instances to the NLB manually using the CLI. Normally you won't need to do this instead perhaps the first time, if you created the instances before our previous automatic setup.

Retrieve the list of instances in your VPC you want to add to the Load balancer..

VPC_ID=vpc-027f50fc9d05149f0
INSTANCES=$(aws ec2 describe-instances --filters Name=network-interface.vpc-id,Values=$VPC_ID \
 --query 'Reservations[*].Instances[*].InstanceId' \
 --output text)

IDS=$(for x in `echo $INSTANCES`; do echo Id=$x ; done | tr '\n' ' ')

..Or we can get instances from the Kubernetes API :

INSTANCES=$(kubectl get nodes -o json | jq -r ".items[].spec.providerID"  | cut -d'/' -f5)
IDS=$(for x in `echo $INSTANCES`; do echo Id=$x ; done | tr '\n' ' ')
echo $IDS

Manually add NLB targets

aws elbv2 register-targets \
  --target-group-arn $(aws elbv2 describe-target-groups --query 'TargetGroups[?TargetGroupName==`kube-dns-nlb`].TargetGroupArn' --output text) \
  --targets $(echo $IDS)

This will add every instance in from the $INSTANCES var as target group for the NLB kube-dns-nlb

Because the service is in NodePort mode, then every instance is able to listen on this port and forward it to the targeted service, in this case the kube-dns service targeting itself the coredns pods.

See what operations can have impact on our targets configuration

Upgrading version for NodeGroups

When upgrading the version of a nodegroup, EKS will start a rolling-upgrade of each instance. this will cause each old instance to disapear from the NLB when it is deleted, and new ones will automatically be added by our Cloudwatch Event + Lambda function

Cleanup

Deleting the target Group

aws elbv2 delete-target-group --target-group-arn $TG_ARN

Deleting the Load Balancer

aws elbv2 delete-load-balancer --load-balancer-arn $LB_ARN