Securing EKS the AWS way: IRSA & Secrets Manager

## Securing EKS the AWS way: IRSA & Secrets Manager ---  ### About me https://rewanthtammana.com/ Rewanth Tammana is a security ninja, open-source contributor, independent consultant & SME at Uptycs. Previously, Senior Security Architect at Emirates NBD. Passionate about DevSecOps, Application, and Container Security. Added 17,000+ lines of code to Nmap. Holds industry certifications like CKS, CKA, etc. ---  ### About me https://twitter.com/rewanthtammana Speaker & trainer at international security conferences worldwide including Black Hat, Defcon, Hack In The Box (Dubai and Amsterdam), CRESTCon UK, PHDays, Nullcon, Bsides, CISO Platform, null chapters and multiple others. ---  ### About me https://linkedin.com/in/rewanthtammana One of the MVP researchers on Bugcrowd (2018) and identified vulnerabilities in several organizations. Published an IEEE research paper on an offensive attack in Machine Learning and Security. Also, part of the renowned Google Summer of Code program. ---  ### Topics of interest * Eagles view on Kubernetes, managed vs self-hosted * Secrets in EKS * Least privilege access in EKS ---  ### Kubernetes ![](https://hackmd.io/_uploads/ryOjdVC-T.png) [Reference](https://mohan08p.medium.com/simplified-kubernetes-architecture-3febe12480eb) ---  ### Self hosted vs managed Kubernetes ![](https://hackmd.io/_uploads/SyhYK4AW6.png) [Reference](https://www.capgemini.com/us-en/insights/expert-perspectives/run-your-own-or-managed-kubernetes-why-microsoft-aks-may-be-the-right-orchestration-platform-for-you/) ---  ### Kubernetes ![](https://hackmd.io/_uploads/rkkUtVR-6.png) [Reference](https://www.quobyte.com/storage-explained/what-is-kubernetes/) ---  ### EKS/cloud managed service | Component | Managed By | |---|---| | Control Plane | Cloud Provider | | Worker Nodes | End User | ---  ### Securing EKS Different components to secure * IAM * Logging * Secrets Management * Monitoring * Encryption * Misconfigurations * Network Policy * ... * ...... ---  ### Secrets * Secrets are sensitive data like passwords, API keys, and database credentials * In Kubernetes world, how do we securely manage and distribute these secrets? ---  ### Options? * Default Kubernetes Secrets * Vault * Sealed Secrets * ... * ..... ---  ### Default Kubernetes Secrets :Pros: * Easy to use, integrated into Kubernetes. :Cons: * Stored in etcd in base64 format - not truly encrypted * Access control is limited; any pod can access any secret if RBAC isn't strictly enforced ---  ### Default Kubernetes Secrets ![](https://hackmd.io/_uploads/rkrPeayMa.png) ---  ### Vault :Pros: * Highly flexible, supports dynamic secrets :Cons: * Complex setup, requires a separate operational effort * Everything with it is a nightmare ---  ### Vault K8S ![](https://hackmd.io/_uploads/H1uRrS0b6.png) [Reference](https://developer.hashicorp.com/vault/tutorials/kubernetes/agent-kubernetes) ---  ### Sealed Secrets :Pros: * Encrypts secrets outside the cluster, only decrypted inside the cluster :Cons: * Encryption key management can be challenging ---  ![](https://hackmd.io/_uploads/H1_u8S0bT.png) [Reference](https://auth0.com/blog/kubernetes-secrets-management/) ---  ### Other solutions Many other solutions, but often require additional components, increasing the complexity ---  ### AWS Secrets Manager * Managed service by AWS * Encrypts secret data with keys managed by AWS Key Management Service (KMS) * Provides fine-grained control over secrets using AWS IAM * Automatic secret rotation capability ---  ### Why? <style> .reveal { font-size: 36px; } </style> | Why | Reason | |---|---| | Integration | Seamless integration with other AWS services | | Security | Uses AWS KMS for encryption, ensuring high security | | Scalability | Managed service, scales automatically | | Availability | Managed service, highly available | | Cost-effective | Pay for what you use, no overhead of managing another system | | Auditing | Integrated with AWS CloudTrail for monitoring and logging | ---  ### Implementation in EKS * External Secrets * CSI (Container Storage Interface) Driver for AWS ---  ### Implementation in EKS ![](https://hackmd.io/_uploads/S132wrCZp.png) [Reference](https://aws.amazon.com/blogs/containers/leverage-aws-secrets-stores-from-eks-fargate-with-external-secrets-operator/) ---  ### Implementation in EKS * Use IAM roles for service accounts (IRSA) to grant pods the necessary permissions to access secrets * Secrets can be referenced directly in pod definitions or through environment variables * No need to sync secrets across different systems ---  ## Least privilege access * Ensuring least-privilege access in Kubernetes can be complex. * RBAC is decent for Kubernetes specific operations but when Kubernetes has to connect with external resources, it's not the best solution. ---  ## Kubernetes RBAC ![](https://hackmd.io/_uploads/SyYAQJ-GT.png) [Reference](https://engineering.dynatrace.com/blog/kubernetes-security-part-1-role-based-access-control-rbac/) ---  ### Use case An application running on Kubernetes fetching random images from the internet every 30 seconds and uploading them to an S3 bucket. ---  ### Different Ways to Connect with AWS Services * Using IAM User Credentials through Environment Variables * Assign permissions to the EKS worker nodes * IAM Roles for Service Accounts (IRSA) ---  ### Create a policy to allow access to the S3 bucket ```bash echo "{ \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"s3:PutObject\" ], \"Resource\": [ \"arn:aws:s3:::$bucket_name/*\" ] } ] }" > s3-$bucket_name-access.json export policy_name=secure-eks-s3-write-policy export create_policy_output=$(aws iam create-policy --policy-name $policy_name --policy-document file://s3-$bucket_name-access.json) export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn') ``` ---  ### Using IAM User Credentials through Environment Variables ```bash export iam_user=secure-eks-iam-user aws iam create-user --user-name $iam_user aws iam attach-user-policy --user-name $iam_user --policy-arn $policy_arn export results=$(aws iam create-access-key --user-name $iam_user) export access_key=$(echo $results | jq -r '.AccessKey.AccessKeyId') export secret_key=$(echo $results | jq -r '.AccessKey.SecretAccessKey') echo "Access Key: $access_key" echo "Secret Key: $secret_key" ``` ---  ### Using IAM User Credentials through Environment Variables ```bash echo "apiVersion: batch/v1 kind: Job metadata: name: environment-variables-job spec: template: spec: containers: - name: environment-variables-container image: rewanthtammana/secure-eks:v1 env: - name: AWS_REGION value: us-east-1 - name: AWS_ACCESS_KEY value: $access_key - name: AWS_SECRET_KEY value: $secret_key - name: S3_BUCKET_NAME value: $bucket_name restartPolicy: Never" | kubectl apply -f- ``` ---  ### Using IAM User Credentials through Environment Variables ![](https://hackmd.io/_uploads/Bk23m2jWT.png) ---  ### Cons * Secret management is hard * What if someone gets their hand on the authentication details? * What if they use it to exfiltrate the data? * How to differentiate b/w legitimate & malicious requests? ---  ### Assign permissions to the EKS worker nodes ```bash! eks_worker_node_role_name=$(eksctl get nodegroup --cluster $cluster_name -o json | jq -r '.[].NodeInstanceRoleARN' | cut -d '/' -f 2) aws iam attach-role-policy --role-name $eks_worker_node_role_name --policy-arn $policy_arn ``` ---  ### Assign permissions to the EKS worker nodes ```bash echo "apiVersion: batch/v1 kind: Job metadata: name: environment-variables-job spec: template: spec: containers: - name: environment-variables-container image: rewanthtammana/secure-eks:ok-amd64 env: - name: AWS_REGION value: us-east-1 - name: S3_BUCKET_NAME value: $bucket_name restartPolicy: Never" | kubectl apply -f- ``` ---  ### Assign permissions to the EKS worker nodes ![](https://hackmd.io/_uploads/By3QfCsW6.png) ---  ### Pros * No secret management * Ease of use ---  ### Cons * Every single pod on the node will have additional permissions, breaking the least privilege policy * Ideally, only one application in the cluster needs it ---  ### IRSA ---  ### OIDC (OpenID Connect) OIDC is an identity protocol that utilizes the authorization and authentication mechanisms of OAuth 2.0 ---  ### AWS IAM OpenID Connect provider * Helps to keep AWS account secure * No need to distribute or embed long-term security credentials, such as access keys, in your application ---  ### OIDC (OpenID Connect) ![](https://hackmd.io/_uploads/rkTdQAsZ6.png) [Reference](https://youtu.be/CPbvxxslDTU) ---  ### Check IAM OIDC provider status ```bash eksctl get cluster $cluster_name -ojson | jq -r '.[].Tags["alpha.eksctl.io/cluster-oidc-enabled"]' ``` ![](https://hackmd.io/_uploads/BklSdakf6.png) ---  ### Create IAM OIDC provider ```bash eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve ``` ![](https://hackmd.io/_uploads/HJxOYp1Gp.png) ---  ### Create an IAM service account with s3 write policy ```bash eks_service_account=s3-write-service-account eksctl create iamserviceaccount --name $eks_service_account --namespace default --cluster $cluster_name --attach-policy-arn $policy_arn --approve ```  ---  ### Create an IAM service account with s3 write policy ![](https://hackmd.io/_uploads/B1JCNAjba.png) ---  ### Create Kubernetes Job ```bash echo "apiVersion: batch/v1 kind: Job metadata: name: environment-variables-job spec: template: spec: serviceAccountName: $eks_service_account containers: - name: environment-variables-container image: rewanthtammana/secure-eks:ok-amd64 env: - name: AWS_REGION value: us-east-1 - name: S3_BUCKET_NAME value: $bucket_name restartPolicy: Never" | kubectl apply -f- ``` ---  ### Create Kubernetes Job ![](https://hackmd.io/_uploads/rywNORsW6.png) ---  ### DEMO ---  ## CloudFormation, JWT, X509 & more ---  ### Examine CloudFormation of IAM SA ![](https://hackmd.io/_uploads/ryU8uAsWa.png) ---  ### Examine CloudFormation of IAM SA ![](https://hackmd.io/_uploads/rywwuAoZ6.png) ---  ### Examine service account ```bash kubectl get sa $eks_service_account -oyaml ``` ![](https://hackmd.io/_uploads/HJTuOCi-T.png) ---  ### Examine secrets associated with the service account ```bash sa_secret_name=$(kubectl get sa $eks_service_account -ojson | jq -r '.secrets[0].name') kubectl get secrets $sa_secret_name -oyaml ``` ![](https://hackmd.io/_uploads/HkbgYAiWa.png) ---  ### Examine X.509 ca.crt in the secret associated with the service account ```bash kubectl get secret $sa_secret_name -o json | jq -r '.data."ca.crt"' | base64 -d ``` ![](https://hackmd.io/_uploads/HJYrY0oWT.png) ---  ### Check the subject & issuer of the certificate ```bash openssl x509 -in certificate.pem -subject -issuer -noout ``` ![](https://hackmd.io/_uploads/HJ_PtAobp.png) ---  ### Check the expiry of the certificate ```bash openssl x509 -in certificate.pem -dates -noout ``` ![](https://hackmd.io/_uploads/r1h_FRobp.png) ---  ### Examine the token in the secret associated with the service account ```bash kubectl get secret $sa_secret_name -o json | jq -r '.data."token"' | base64 -d ``` ![](https://hackmd.io/_uploads/rk25YCjWp.png) ---  ### Examine the JWT token in jwt.io ![](https://hackmd.io/_uploads/HJg1q0jZa.png) ---  ### Quiz Time We created service accounts & let them assume a role with IRSA. * What about non-EKS clusters? Self-hosted services on AWS? ---  ### Non-EKS clusters on AWS * [Zalando IAM Controller](https://github.com/zalando-incubator/kube-aws-iam-controller) * [Kiam](https://github.com/uswitch/kiam) ---  ### Zalando Using custom Assume role ``` +-------------+ | | +--> | <app-role1> | +-----------------+ +-------------------+ | | | | | | | | +-------------+ | <instance-role> | -- assumes --> | <controller-role> | -- assumes --+ | | | | | +-------------+ +-----------------+ +-------------------+ | | | +--> | <app-role2> | | | +-------------+ ``` ---  ### Recap ![](https://hackmd.io/_uploads/SJ5Iibxf6.png) ---  ### Additional resources [Securing AWS EKS: Implementing Least-Privilege Access with IRSA](https://blog.rewanthtammana.com/securing-aws-eks-implementing-least-privilege-access-with-irsa) ---  ### Thank You! Write to me on: Google: [Rewanth Tammana](https://www.google.com/search?q=rewanth+tammana&sca_esv=575386901&sxsrf=AM9HkKmnGZBGWzwOS8jWMPYp_YgzN4l8og%3A1697865818064&source=hp&ei=WmAzZbizAeHk1e8Pt5ygiA4&iflsig=AO6bgOgAAAAAZTNuas_v-XYaB2vpwWaiYRGIPtXHOTZW&ved=0ahUKEwi4yNaHs4aCAxVhcvUHHTcOCOEQ4dUDCAo&uact=5&oq=rewanth+tammana&gs_lp=Egdnd3Mtd2l6Ig9yZXdhbnRoIHRhbW1hbmEyBhAAGBYYHkjxC1AAWOsKcAB4AJABAJgB3gGgAagSqgEGMC4xMy4xuAEDyAEA-AEBwgIHECMYigUYJ8ICCBAAGIoFGJECwgIREC4YgAQYsQMYgwEYxwEY0QPCAgsQABiABBixAxiDAcICCxAuGIAEGLEDGIMBwgILEC4YigUYsQMYgwHCAg4QABiKBRixAxiDARiRAsICCBAAGIAEGLEDwgILEAAYigUYsQMYgwHCAggQLhixAxiABMICCxAuGIAEGMcBGNEDwgIFEAAYgATCAgUQLhiABMICCxAuGIAEGMcBGK8BwgIHEAAYgAQYCsICDRAuGIAEGLEDGIMBGArCAgoQLhixAxiABBgKwgINEAAYgAQYsQMYgwEYCsICBxAuGIAEGArCAgcQABgNGIAEwgIHEC4YDRiABMICCRAAGA0YgAQYCsICCRAuGA0YgAQYCsICBhAAGB4YDcICCBAAGAUYHhgNwgIIEAAYCBgeGA3CAgoQABgIGB4YDRgK&sclient=gws-wiz) Website: [rewanthtammana.com](https://rewanthtammana.com/) Twitter: [@rewanthtammana](https://twitter.com/rewanthtammana) LinkedIn: [/in/rewanthtammana](https://www.linkedin.com/in/rewanthtammana/)