# EKS logging with EFK stack

__Goal__: Set up logging of EKS clusters using EFK stack consisting of:
- AWS hosted ElasticSearch
- FluentD
- Kibana.

The logging should be for _BOTH_ the EKS control plane and pods (via FluentD)

__Problem__: Logging for _EITHER_ control plane _OR_ pods work fine. Logging for _BOTH_ in the same EFK stack does not work.

Note: I obfuscated things like account Id and IP addresses in case this needs to be shared further. I'm sure you understand.

#### Setup description

My environment consists of:
- AWS EC2 Ubuntu instance, acting as a jump server. This is where most of the admin tasks are executed from. The EC2 instance has an IAM role with extensive privileges (though _not_ a blanket "AdminAccess"). This IAM role is called `fo20190822k8s` and has these extensive policies attached:


- An EKS cluster -- named `eksctl-mixed-asgs-20200301-1520`. The cluster has two ASGs (auto-scaling groups): One ASG with on-demand instances, and one ASG with spot instances. In this screenshot, it's the top two ASGs (selected).


### Create a hosted ElasticSearch domain

Instructions adapted from from [EKS Workshop - Logging with EFK](https://eksworkshop.com/intermediate/230_logging/):

For ES (ElasticSearch) domain name I used same name as the K8S cluster (domain name has to be shorter than 28 characters)

aws es create-elasticsearch-domain \
  --domain-name  mixed-asgs-20200301-1520 \
  --elasticsearch-version 6.3 \
  --elasticsearch-cluster-config \
  InstanceType=m5.large.elasticsearch,InstanceCount=3 \
  --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=100 \
  --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["*"]},"Action":["es:*"],"Resource":"*"}]}'

__Important notes__:
1) The domain was created with fully open access. Creating this in an Isengard account created a flurry of tickets flagging this as a security issue. As such access to the ES domain was limited (below).

3) Regardless, the EFK logging problem can be replicated with fully open access, so it does does not _SEEM_ to be an access issue.

Locking down the access to the ES domain is done like this (example adapted from https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html . Account IDs and Source IP obfuscated)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123123123123:role/eksctl-mixed-asgs-20200301-1520-n-NodeInstanceRole-HM3STYCGR6YY",
          "arn:aws:iam::123123123123:role/eksctl-mixed-asgs-20200301-1520-n-NodeInstanceRole-1363S87DRY84T",
          "arn:aws:iam::123123123:role/fo20190822k8s"
        ]
      },
      "Action": "es:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ["*"]
      },
      "Action": "es:*",
      "Condition": {
      "IpAddress": {
        "aws:SourceIp": [
          "123.123.123.123/32"
        ]
      }
    },
      "Resource": "*"
    }
  ]
}

Notes:
- The `fo20190822k8s` IAM is the "Administrator-like" role assumed by the jump server (and others, below).
- The two other IAM roles are the `NodeInstanceRole` that were automatically created when the EKS cluster was created. These IAM roles are attached, respectively, to the EKS worker nodes in the two ASGs for the cluster (above).
- The "Source IP" condition was to allow web acces from my web browser (remote, off-AWS) directly to the Kibana dashboard.

### Enabling EKS cluster control plane logging to CloudWatch

Used the instructions at [EKS docs -- Control plane logging](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html) and at [EKS user guide on GitHub -- Control plane logging](https://github.com/awsdocs/amazon-eks-user-guide/blob/master/doc_source/control-plane-logs.md).

Result: In CloudWatch, a log group with the name `/aws/eks/mixed-asgs-20200301-1520/cluster` was successfully created.

### Installing FluentD for pod log forwarding

Instructions adapted from [EKS workshop - Deploy FluentD](https://eksworkshop.com/intermediate/230_logging/deploy/)

curl https://eksworkshop.com/intermediate/230_logging/deploy.files/fluentd.yml -o eksworkshop-fluentd.yaml

sed -i".bak" -e "s#us-west-2#eu-north-1#g" -e "s#eksworkshop-eksctl#mixed-asgs-20200301-1520#g" -e "s#/eks/#/aws/eks/#g" eksworkshop-fluentd.yaml

__Notes__:
- The YAML config file in the EKS workshop example is _very_ opaque...
- Changing the region from `use-west-2` to `eu-north-1` (where my cluster is located).
- (Matter of cosmetic preference): The default configuration file uses as log prefix `/eks/<cluster-name>`. However, to be consistent with the control plane logging -- which uses `/aws/eks/<cluster-name>` prefix -- adjusting the log prefix as well.

Install it and check status

kubectl apply -f eksworkshop-fluentd.yaml

serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created
configmap/fluentd-config created
daemonset.apps/fluentd-cloudwatch created

kubectl describe daemonset.apps/fluentd-cloudwatch -n kube-system

....
....

__Result__: A second log group was created in CloudWatch, called `/aws/eks/mixed-asgs-20200301-1520/containers`. So overall, I have TWO log groups in Cloud Watch -- one for control plane, and one for pods:


### Forwarding TWO log groups from CloudWatch

__This is the tricky part since it involves Lambda functions__

Following instructions from [EKS workshop -- Configure CloudWatch](https://eksworkshop.com/intermediate/230_logging/configurecwl/) and [AWS Docs -- Streaming CloudWatch Logs Data to Amazon Elasticsearch Service](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html)

Following those instructions:
- From the CloudWatch console, select one of the two log groups.
- "Actions" -> "Stream to Amazon Elastic Service"
- In my case select as "Amazon ES cluster" the ES domain created above.
- As Lambda IAM Execution Role select `fo20190822k8s` (the Administrator-like role. As per above, this role has full access to CloudWatch policy attached).
- Select Common Log format in next screen.

(Note: The log format pattern is inadequate for BOTH control plane logs and pod logs, but it can be adjusted -- somewhat. More on this, later).

- Create the filter, and start streaming logs.

Result:
- A new log group (3rd log group) is created in CloudWatch for the Lambda function thus created.
- The CloudWatch logs are sent to ES domain, and can be seen in Kibana (after configuring Kibana, as per the instructions at [EKS workshop -- Configure CloudWatch](https://eksworkshop.com/intermediate/230_logging/configurecwl/))

#### Problem

Doing those steps for _any_ of the two log groups -- for the control plane OR for pods, doesn't matter -- works. CloudWatch logs are in streamed to ES and can be seen in Kibana.

__HOWEVER__, trying to do the same for both log groups does __NOT__ work.

Note, the result is the same no matter if I create the function for control plane log group and then do the same for the pod log group, or the other way around. That's why below "first log group" or "second log group" can be either of the two log groups.

Things I attempted

#### Alternative 1: Trying repeat the steps above for "the other" log group

Instead of creating another lambda function for the second log group, the following message appears:


Following the same steps as before, it does create a subscription filter for the second log group. However, the Lambda function -- which work fine for the fist log group -- issues only errors for the second log group. From the CloudWatch log for the Lambda function:


#### Alternative 2: In the Lambda GUI try to add another trigger for the second log group.

In the GUI console for the Lambda function that was created, in the "Configuration" tab, under "Designer", it is possible to "Add trigger" and create a forwarder for the 2nd log group (see screenshot below).

Also, in this Lambda GUI it is possible:
- To Enable / Disable any of the two forwarders.
- Adjust the filter pattern (not possible in the "Alternative 1" above).

Result: The function issues the same error as above for the second log group (the one for which the trigger was added after creation).

Disabling - in the "Designer" -- the trigger for the second log group makes the function work fine again (no errors).

Disabling the original trigger / the trigger for the first log group (the one automatically created when the function was created via CloudWatch) and leaving only the manually added trigger issues same error.

This latter case is in this screenshot -- function initially created for the control plane log group, then a 2nd trigger attemped for the pod logs, then the first one disabled to isolate only errors:


__Note__: Disabling / Enabling trigger requires saving / updating the lambda function

### Alternative 3: Try to manually create a separate Lambda function

No matter which log group was chosen as "first log group" -- log group for control plane or pods -- the resulting lambda function code was identical (Node.js code). Only the triggers (???) and possibly something else that I'm missing (???) was different between the two situations.

As such, after creating -- from the CloudWatch GUI -- the lambda function for the first log group, I tried to manually create a Lambda function for the 2nd log group:

- Downloaded the (working) Lambda function for the first log group as a file
- From Lambda GUI -> "Create Function"
- "Use blueprint"
- Search "cloudwatch" in the "Blueprints" search box
- Used "cloudwatch-logs-process-data" as blueprint
- In the "Configure" GUI:
- Provide name
- Use existing IAM role -- same as above
- As "CloudWatch trigger" select the second log group
- Select fiter pattern and name
- Enable filter checkbox

__Result__:
- The Lambda function is created and a CloudWatch log group (4th log group) is created for this function.
- All log entries for the function issue the same error as above.
- This is even if the first function -- the one created via CloudWatch -- has the trigger disabled, or the function is deleted.

Also, manually creating the function using this steps (i.e. not from the CloudWatch GUI) issues the same error even in the absence of any other Lambda function / log streamer.

Questions:

1) What does the message stem from:

> You already have a Lambda function associated with your .. <name>.. Amazon ElasticSearch cluster and this setup will be reusing the Lambda function... <>... to get your log data delivered to the cluster ?

2) How can this be done when "manually" creating a Lambda function. When doing so, are there any other steps needed to successfully stream logs from CloudWatch to ES ?

3) If creating two Lambda functions "associated with the ES cluster" (whatever that means...) is unnecessary and/or impossible, how can the first function be re-used for the 2nd log group

4) (Optional) The Common Log format -- with filter pattern ` [host, ident, authuser, date, request, status, bytes]` seems inadequate for either control plane logs and FluentD logs (slightly different), as fields are not mapped entirely correct. How can that be adjusted so that fields are mapped correctly ? Specify a JSON unmarshalling struct ? Other ?

***

Author: Nick Brandaleone
Date: March 14, 2020 (also known as PI day!)

I was able to get two streams of logs into the ElasticSearch cluster. However, I had to make a few changes for it to work.

1. I created a new lambda function. It had a different name, but nearly the same code as the AWS provided function. I copied the original function into the code editor, and made only minor changes.
2. I manually created a trigger for this new function. The trigger was the second CloudWatch *log_group* we wanted to observe inside the ES cluster. I had to create this **manually**, as it was not possible to use the CW *subscription* feature. The subscription feature only can handle a single lambda function for streaming - since I needed two functions, it was not suitable (although I did use it for the first function/stream).
3. I altered the **index** name of the log stream, inside the lambda function. I found during debugging the second stream was being denied by the ElasticSearch cluster due to an index clash. I do not know enough about ES to determine if there is an easier way to solve this problem. However, by hard-coding the steam into a new index - it worked.
4. I had to go into the ES cluster, and identify the new **index**. Once done, I was able to toggle between the two different indexes, and see the two different *log_groups*.

## Altered lambda function - index name changed
> // index name format: cwl-YYYY.MM.DD
var indexName = [
'kubernetes-' + timestamp.getUTCFullYear(), // year
('0' + (timestamp.getUTCMonth() + 1)).slice(-2), // month
('0' + timestamp.getUTCDate()).slice(-2) // day
].join('.');

## Lambda debugging. Fortunately, there is a toggle in the provided function which helped

> // Set this to true if you want to debug why data isn't making it to
// your Elasticsearch cluster. This will enable logging of failed items
// to CloudWatch Logs.
var logFailedResponses = true; // Set to false normally.

It was by turning on debugging on both the lambda function and the ES cluster, I was able to determine that CloudWatch was sending the logs to the ES cluster. The cluster was responding with a 400 error. By examining the error and some Googling, I was able to confirm the index clash *error*.


Finally, I should admit that while I did install fluentd for log-shipping, I used the simplified *container insights* installation process. This created several log streams, but not the same ones you used from the **eksworkshop**. I am sure our root cause is the same, but our *log_groups* are slightly different.

### Container Insights installation

> curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | sed "s/{{cluster_name}}/cluster-name/;s/{{region_name}}/cluster-region/" | kubectl apply -f -

The above command will create 5 *log groups*, which populate the CloudWatch Dashboards, and provide various logging of the control plane and worker nodes. I have not played around with it enough to know all the details yet.

**Good night**. I hope this is useful.
I personally find the *IAM* credentials to be a huge burden to some of our products. However, security first is not a bad motto.

If you find easier methods please send them my way. I think that *Prometheus* is the superior solution for many reasons - however, ElasticSearch is a powerful product. I am sure many customers will certainly be using it. This error was really an ES configuration issue (i.e. customer responsibility!!!!)

:-)