EKS Pod Identity

## EKS Pod Identity ---   ### About me https://rewanthtammana.com/ Rewanth Tammana is a security ninja, open-source contributor, & independent consultant. Previously, Senior Security Architect at Emirates NBD. Passionate about DevSecOps, Application, and Container Security. Added 17,000+ lines of code to Nmap. Holds industry certifications like CKS, CKA, etc. ---   ### About me https://twitter.com/rewanthtammana Speaker & trainer at international security conferences worldwide including Black Hat, Defcon, Hack In The Box (Dubai and Amsterdam), CRESTCon UK, PHDays, Nullcon, Bsides, CISO Platform, null chapters and multiple others. ---   ### About me https://linkedin.com/in/rewanthtammana One of the MVP researchers on Bugcrowd (2018) and identified vulnerabilities in several organizations. Published an IEEE research paper on an offensive attack in Machine Learning and Security. Also, part of the renowned Google Summer of Code program. ---   ### Topics of interest * Eagles view on Kubernetes, managed vs self-hosted * Least privilege access in EKS * IRSA vs EKS Pod Identity ---   ### Kubernetes ![](https://hackmd.io/_uploads/ryOjdVC-T.png) [Reference](https://mohan08p.medium.com/simplified-kubernetes-architecture-3febe12480eb) ---   ### Self hosted vs managed Kubernetes ![](https://hackmd.io/_uploads/SyhYK4AW6.png) [Reference](https://www.capgemini.com/us-en/insights/expert-perspectives/run-your-own-or-managed-kubernetes-why-microsoft-aks-may-be-the-right-orchestration-platform-for-you/) ---   ### Kubernetes ![](https://hackmd.io/_uploads/rkkUtVR-6.png) [Reference](https://www.quobyte.com/storage-explained/what-is-kubernetes/) ---   ### EKS/cloud managed service | Component | Managed By | |---|---| | Control Plane | Cloud Provider | | Worker Nodes | End User | ---   ### Securing EKS Different components to secure * IAM * Logging * Secrets Management * Monitoring * Encryption * Misconfigurations * Network Policy * ... * ...... ---   ### Create EKS cluster How many use AWS CLI? ---   ### EKSCTL ```bash eksctl create cluster ``` ![image](https://hackmd.io/_uploads/rJTtFWiLa.png) ---   ## Least privilege access * Ensuring least-privilege access in Kubernetes can be complex. * RBAC is decent for Kubernetes specific operations but when Kubernetes has to connect with external resources, it's not the best solution. ---   ## Kubernetes RBAC ![](https://hackmd.io/_uploads/SyYAQJ-GT.png) [Reference](https://engineering.dynatrace.com/blog/kubernetes-security-part-1-role-based-access-control-rbac/) ---   ### Use case An application running on Kubernetes fetching random images from the internet every 30 seconds and uploading them to an S3 bucket. ---   ### How would you do it? * Using IAM User Credentials through Environment Variables  * Assign permissions to the EKS worker nodes  * IAM Roles for Service Accounts (IRSA)  * [NEW - Re:Invent 2023] EKS Pod Identity  ---   ### IRSA ---   ### OIDC (OpenID Connect) OIDC is an identity protocol that utilizes the authorization and authentication mechanisms of OAuth 2.0 ---   ### AWS IAM OpenID Connect provider * Helps to keep AWS account secure * No need to distribute or embed long-term security credentials, such as access keys, in your application ---   ### OIDC (OpenID Connect) ![](https://hackmd.io/_uploads/rkTdQAsZ6.png) [Reference](https://youtu.be/CPbvxxslDTU) ---   ### Check IAM OIDC provider status ```bash eksctl get cluster $cluster_name -ojson | jq -r '.[].Tags["alpha.eksctl.io/cluster-oidc-enabled"]' ``` ![](https://hackmd.io/_uploads/BklSdakf6.png) ---   ### Create IAM OIDC provider ```bash eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve ``` ![](https://hackmd.io/_uploads/HJxOYp1Gp.png) ---   ### Create an IAM service account with s3 write policy ```bash eks_service_account=s3-write-service-account eksctl create iamserviceaccount --name $eks_service_account --namespace default --cluster $cluster_name --attach-policy-arn $policy_arn --approve ```  ---   ### Create an IAM service account with s3 write policy ![](https://hackmd.io/_uploads/B1JCNAjba.png) ---   ### Create Kubernetes Job ```bash echo "apiVersion: batch/v1 kind: Job metadata: name: environment-variables-job spec: template: spec: serviceAccountName: $eks_service_account containers: - name: environment-variables-container image: rewanthtammana/secure-eks:ok-amd64 env: - name: AWS_REGION value: us-east-1 - name: S3_BUCKET_NAME value: $bucket_name restartPolicy: Never" | kubectl apply -f- ``` ---   ### Create Kubernetes Job ![](https://hackmd.io/_uploads/rywNORsW6.png) ---   ## CloudFormation, JWT, X509 & more ---   ### Examine CloudFormation of IAM SA ![](https://hackmd.io/_uploads/ryU8uAsWa.png) ---   ### Examine CloudFormation of IAM SA ![](https://hackmd.io/_uploads/rywwuAoZ6.png) ---   ### Examine service account ```bash kubectl get sa $eks_service_account -oyaml ``` ![](https://hackmd.io/_uploads/HJTuOCi-T.png) ---   ### Examine secrets associated with the service account ```bash sa_secret_name=$(kubectl get sa $eks_service_account -ojson | jq -r '.secrets[0].name') kubectl get secrets $sa_secret_name -oyaml ``` ![](https://hackmd.io/_uploads/HkbgYAiWa.png) ---   ### Examine X.509 ca.crt in the secret associated with the service account ```bash kubectl get secret $sa_secret_name -o json | jq -r '.data."ca.crt"' | base64 -d ``` ![](https://hackmd.io/_uploads/HJYrY0oWT.png) ---   ### Check the subject & issuer of the certificate ```bash openssl x509 -in certificate.pem -subject -issuer -noout ``` ![](https://hackmd.io/_uploads/HJ_PtAobp.png) ---   ### Check the expiry of the certificate ```bash openssl x509 -in certificate.pem -dates -noout ``` ![](https://hackmd.io/_uploads/r1h_FRobp.png) ---   ### Examine the token in the secret associated with the service account ```bash kubectl get secret $sa_secret_name -o json | jq -r '.data."token"' | base64 -d ``` ![](https://hackmd.io/_uploads/rk25YCjWp.png) ---   ### Examine the JWT token in jwt.io ![](https://hackmd.io/_uploads/HJg1q0jZa.png) ---   ### Pros Really good & best way to achieve least privilege access controls ---   ### Cons Cons - Managing it can get tricky and hard ![](https://hackmd.io/_uploads/B1JCNAjba.png) ---   ### EKS Pod Identity ---   ### Enable the add on ```bash eksctl create addon --name eks-pod-identity-agent --cluster $CLUSTER_NAME ``` ![image](https://hackmd.io/_uploads/SkAFoA9Ua.png) ---   ### Daemonset ```bash kubectl get ds -A ``` ![image](https://hackmd.io/_uploads/HymaiAqIa.png) ---   ### Get Pod Identity Association ```bash export CLUSTER_NAME=pod-identity-demo eksctl get podidentityassociation --cluster $CLUSTER_NAME ``` ![image](https://hackmd.io/_uploads/r11o30cLT.png) ---   ### Create Pod Identity Association ```bash export SERVICE_ACCOUNT_NAME=anything eksctl create podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name $SERVICE_ACCOUNT_NAME --permission-policy-arns $policy_arn ``` ![image](https://hackmd.io/_uploads/Bkw4-es8T.png) ---   ### Create AWS Policy ```bash export BUCKET_NAME=random-pod-identity-demo echo "{ \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"s3:PutObject\" ], \"Resource\": [ \"arn:aws:s3:::$BUCKET_NAME/*\" ] } ] }" > s3-$BUCKET_NAME-access.json export POLICY_NAME=pod-identity-bucket-s3-write-policy export create_policy_output=$(aws iam create-policy --policy-name $POLICY_NAME --policy-document file://s3-$BUCKET_NAME-access.json) export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn') echo $policy_arn ``` ---   ### Create AWS Policy ![image](https://hackmd.io/_uploads/SJepn098a.png) ---   ### Create Kubernetes Job ```bash aws s3 mb s3://$BUCKET_NAME --region us-east-1 kubectl create sa $SERVICE_ACCOUNT_NAME echo "apiVersion: batch/v1 kind: Job metadata: name: pod-identity-demo spec: template: spec: serviceAccountName: $SERVICE_ACCOUNT_NAME containers: - name: pod-identity-demo-container image: rewanthtammana/secure-eks:pod-identity-demo env: - name: AWS_REGION value: us-east-1 - name: S3_BUCKET_NAME value: $BUCKET_NAME restartPolicy: Never" | kubectl apply -f- ``` ![image](https://hackmd.io/_uploads/H1pAhA9Ip.png) ---   ### Check job status ```bash kubectl get jobs kubectl get po -l job-name=pod-identity-demo kubectl logs -l job-name=pod-identity-demo ``` ![image](https://hackmd.io/_uploads/Bk_xTC9Lp.png) ---   ### Examine Cloudformation ```bash aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything ``` ![fbce0047-48ab-4636-9a01-3d72e0999a8a](https://hackmd.io/_uploads/S1_3bgoIp.png) ---   ### Examine Role & Policy ```bash export role_name=$(aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything | jq -r '.StackResources[].PhysicalResourceId') echo $role_name aws iam list-attached-role-policies --role-name $role_name ``` ![image](https://hackmd.io/_uploads/HJZAZxsL6.png) ---   ### IRSA vs EKS Pod Identity ![image](https://hackmd.io/_uploads/rywMTC5La.png) ---   ### Inside of Application Pod ![image](https://hackmd.io/_uploads/B1M_TAc8T.png) ---   ### Inside of Pod Identity Webhook ```bash kubectl get ds -n kube-system eks-pod-identity-agent -oyaml ``` ---   ![image](https://hackmd.io/_uploads/HywKT0cL6.png =600x600) ---   ### Inside of Pod Identity Webhook ![f526d4ae-dd65-4b39-9f41-9755385e1ad6-min](https://hackmd.io/_uploads/BJ9p60cUa.png) ---   ### Quiz Time We created service accounts & let them assume a role with IRSA. * What about non-EKS clusters? Self-hosted services on AWS? ---   ### Non-EKS clusters on AWS * [Zalando IAM Controller](https://github.com/zalando-incubator/kube-aws-iam-controller) * [Kiam](https://github.com/uswitch/kiam) ---   ### Zalando Using custom Assume role ``` +-------------+ | | +--> | <app-role1> | +-----------------+ +-------------------+ | | | | | | | | +-------------+ | <instance-role> | -- assumes --> | <controller-role> | -- assumes --+ | | | | | +-------------+ +-----------------+ +-------------------+ | | | +--> | <app-role2> | | | +-------------+ ``` ---   ### Recap ![](https://hackmd.io/_uploads/SJ5Iibxf6.png) ---   ### Additional resources [Securing AWS EKS: Implementing Least-Privilege Access with IRSA](https://blog.rewanthtammana.com/securing-aws-eks-implementing-least-privilege-access-with-irsa) [Rethinking Authentication: AWS ReInvent 2023 Unveils EKS Pod Identity](https://blog.rewanthtammana.com/rethinking-authentication-aws-reinvent-2023-unveils-eks-pod-identity) [Github - Secure EKS](https://github.com/rewanthtammana/secure-eks) ---   ### Thank You! ![linkedin-qrcode](https://hackmd.io/_uploads/B1cGQng4a.png =x400) Google: [Rewanth Tammana](https://www.google.com/search?q=rewanth+tammana&sca_esv=575386901&sxsrf=AM9HkKmnGZBGWzwOS8jWMPYp_YgzN4l8og%3A1697865818064&source=hp&ei=WmAzZbizAeHk1e8Pt5ygiA4&iflsig=AO6bgOgAAAAAZTNuas_v-XYaB2vpwWaiYRGIPtXHOTZW&ved=0ahUKEwi4yNaHs4aCAxVhcvUHHTcOCOEQ4dUDCAo&uact=5&oq=rewanth+tammana&gs_lp=Egdnd3Mtd2l6Ig9yZXdhbnRoIHRhbW1hbmEyBhAAGBYYHkjxC1AAWOsKcAB4AJABAJgB3gGgAagSqgEGMC4xMy4xuAEDyAEA-AEBwgIHECMYigUYJ8ICCBAAGIoFGJECwgIREC4YgAQYsQMYgwEYxwEY0QPCAgsQABiABBixAxiDAcICCxAuGIAEGLEDGIMBwgILEC4YigUYsQMYgwHCAg4QABiKBRixAxiDARiRAsICCBAAGIAEGLEDwgILEAAYigUYsQMYgwHCAggQLhixAxiABMICCxAuGIAEGMcBGNEDwgIFEAAYgATCAgUQLhiABMICCxAuGIAEGMcBGK8BwgIHEAAYgAQYCsICDRAuGIAEGLEDGIMBGArCAgoQLhixAxiABBgKwgINEAAYgAQYsQMYgwEYCsICBxAuGIAEGArCAgcQABgNGIAEwgIHEC4YDRiABMICCRAAGA0YgAQYCsICCRAuGA0YgAQYCsICBhAAGB4YDcICCBAAGAUYHhgNwgIIEAAYCBgeGA3CAgoQABgIGB4YDRgK&sclient=gws-wiz) Website: [rewanthtammana.com](https://rewanthtammana.com/)