<style> img { border-radius: 10px; border: 2px solid #BDC3C7; } </style> ![generate (2)](https://hackmd.io/_uploads/HJvtStmeC.jpg) # 緣由 最近在工作上，例行的檢查AWS **applicaiton loadbalancer** 的monitoring tab時，發現了其中一格神祕的資訊: ![image](https://hackmd.io/_uploads/Sy3YLgN1A.png) 事實上，這個資訊看過多次，但是一直不了解它代表了甚麼意義。查詢了aws 的 **knowledge base**，aws建議可以用**ssl-scan**去檢查attach在loadbalancer上面的certificate。然而，我們的certificate是**ACM** (AWS Certificate Manager)所派發的，基本上不會有問題才是。 > https://repost.aws/knowledge-center/elb-fix-ssl-tls-negotiation-error 接著，我去下載了loadbalancer的access log，但log沒有看到任何關於*Client TLS negotiation errors*的紀錄。 > 例如，access log的欄位解讀: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html 查找了網路上的資訊，發現有人利用AWS的VPC traffic mirroring 來capture loadbalancer的封包，除錯 *Client TLS negotiation errors*的發生原因。基於好奇心，我們來做個實驗吧! # VPC Traffic Mirroring 概念 - 簡單來說，只要是resource具有`ENI` (Elastic Network Interface)，理論上都可以透過traffic mirroring的方式擷取封包。 ![image](https://hackmd.io/_uploads/Hyv8K1Yk0.png) (出處: https://www.qacafe.com/resources/aws-vpc-traffic-mirroring-cloud-packet-capture/) - 決定source以及target的`ENI`。 - Source是applicaiton loadbalancer。 - application loadbalancer的ENI在哪裡可以查到? - Target是同VPC內的某台EC2。 ## VXLAN AWS 利用 VXLAN protocol 來包裝封包: ![image](https://hackmd.io/_uploads/rk-CBZt1R.png) > https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-packet-formats.html ## 取得 application loadbalancer的資訊 ```shell aws elbv2 describe-load-balancers --names mvb-eco-sys-ws-prod --region eu-central-1 ``` ```JSON { "LoadBalancers": [ { "LoadBalancerArn": "arn:aws:elasticloadbalancing:eu-central-1:******************:loadbalancer/app/mvb-eco-sys-ws-prod/741****************", "DNSName": "************.eu-central-1.elb.amazonaws.com", "CanonicalHostedZoneId": "**************", "CreatedTime": "2021-01-13T10:34:40.780000+00:00", "LoadBalancerName": "mvb-eco-sys-ws-prod", "Scheme": "internet-facing", "VpcId": "vpc-***************", "State": { "Code": "active" }, "Type": "application", "AvailabilityZones": [ { "ZoneName": "eu-central-1a", "SubnetId": "subnet-******************", "LoadBalancerAddresses": [] }, { "ZoneName": "eu-central-1b", "SubnetId": "subnet-******************", "LoadBalancerAddresses": [] } ], "SecurityGroups": [ "sg-******************" ], "IpAddressType": "ipv4" } ] } ``` ## 取得applicaiton loadbalancer ENI列表 ```shell aws ec2 describe-network-interfaces --filters Name=description,Values="ELB app/mvb-eco-sys-ws-prod/******************" --query 'NetworkInterfaces[*]' --output json --region eu-central-1 | jq -c '.[].NetworkInterfaceId' ``` > "eni-0a2bf36cdb1c57099" > "eni-00d05507cd1140898" > "eni-080477509901b6a76" > "eni-0e2c5f664c663abc9" > "eni-0905dc9871ecab671" > "eni-0292a0604baf86b91" > "eni-0a1f1ad189baaa7df" > "eni-0a2c8d84bd7953b08" > "eni-05cccf1718fbaf35f" > "eni-06862fbbad18b5c46" > "eni-0c36fed47d6d0de6b" > "eni-081934841f840b09f" > "eni-09cbe68ac9c8f0b44" 這台loadbalancer用量比較大，所以有13張ENI ! ## 什麼 ? 原來application loadbalancer是一排EC2? 其實從以前就覺得奇怪，為什麼loadbalancer跟EC2是放在同一個介面上 (EC2 Dashboard )。原來loadbalancer本質上就是一排的EC2! 透過下面的資料可以觀察到: ```shell aws ec2 describe-network-interfaces --filters Name=description,Values="ELB app/mvb-eco-sys-ws-prod/******************" --query 'NetworkInterfaces[*]' --output json --region eu-central-1 | jq -c '.[]."Association"."PublicDnsName"' ``` > "ec2-3-121-54-139.eu-central-1.compute.amazonaws.com" > "ec2-3-77-159-52.eu-central-1.compute.amazonaws.com" > "ec2-18-153-166-253.eu-central-1.compute.amazonaws.com" > "ec2-3-120-200-244.eu-central-1.compute.amazonaws.com" > "ec2-54-93-93-216.eu-central-1.compute.amazonaws.com" > "ec2-52-28-50-142.eu-central-1.compute.amazonaws.com" > "ec2-3-120-19-81.eu-central-1.compute.amazonaws.com" > "ec2-18-198-21-139.eu-central-1.compute.amazonaws.com" > "ec2-52-57-17-95.eu-central-1.compute.amazonaws.com" > "ec2-3-124-82-62.eu-central-1.compute.amazonaws.com" > "ec2-52-28-107-255.eu-central-1.compute.amazonaws.com" > "ec2-18-157-172-87.eu-central-1.compute.amazonaws.com" > "ec2-52-28-217-154.eu-central-1.compute.amazonaws.com" > # 動手做 順序: > 1. 開啟&設定一台EC2，作為traffic mirroring 的 destination and tcpdump的執行機器 > 2. 設定VPC traffic mirroring ，啟動 mirroring。 > 3. 回到EC2，設定一個vxlan interface，使用tcpdump擷取packets。 ## *EC2 - part 1* 1. 開啟一台EC2，設定好security group。 2. ssh進入EC2，安裝 conntrack and tcpdump。 ```shell sudo iptables -t filter -A INPUT -m state --state NEW\,ESTABLISHED -j ACCEPT && sudo iptables -t filter -A FORWARD -m state --state NEW\,ESTABLISHED -j ACCEPT && sudo iptables -t filter -A OUTPUT -m state --state NEW\,ESTABLISHED -j ACCEPT ``` ```shell /bin/echo "1" > /proc/sys/net/netfilter/nf_conntrack_acct ``` 3. 這個時候應該還沒有任何traffic ![image](https://hackmd.io/_uploads/HJIfhbKyC.png) ## *VPC traffic mirroring Configuration* ### Mirror targets 設定你的EC2為 mirror target ![image](https://hackmd.io/_uploads/ry99o-Y10.png) ### Mirror filters 1. 設定如下圖的 reject & accept，只擷取external traffic的資料。 2. Internal traffic VPC CIDR都去掉。 ![image](https://hackmd.io/_uploads/H1Nrq1t10.png) ### Mirror sessions 1. 從上述的資訊，可以確定你選到正確的ENI: ![螢幕擷取畫面 2024-04-02 164519](https://hackmd.io/_uploads/BJ7eLBK1R.png) ## *EC2 - part 2* 1. 檢查 EC2 上面的 port 4789 是否已經有mirroring來的traffic。 ```shell sudo conntrack -L -p udp ``` ![image](https://hackmd.io/_uploads/HkOr6ZFyA.png) 2. 新增一個VXLAN interface ```shell sudo ip link add ${interface_name} type id ${vni} local ${EC2's local ip address} remote ${source_ip_address} dev ${interface_name_of_EC2_eni} dstport 4789 sudo ip link set ${interface_name} up ``` 3. 開始抓封包 ```shell sudo tcpdump -n -i ${interface_name} -vv -w ${filename}.pcap ``` # 分析pcap 1. 把 \*.pcap下載下來，使用wireshark打開pcap檔。輸入以下的指令做filter: ``` tls.alert_message.level == 2 ``` 2. 輸入完後，可以發現出現了很多**Certificate Unknown - 21**的alert ! ![螢幕擷取畫面 2024-03-29 170916](https://hackmd.io/_uploads/SkVMLZEJC.png) - 打開封包檔: ![image](https://hackmd.io/_uploads/B1BD5etkC.png) 3. follow 某一個TLS stream，可以發現以下的封包: ![螢幕擷取畫面 2024-03-29 171839](https://hackmd.io/_uploads/Byvl_bN1C.png) ## TLS Alert > Error code 21 for TLS > https://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol > ![image](https://hackmd.io/_uploads/rJCo-WFJC.png) # 題外話 - JA3 - 2017年由Saleforces的工程師提出。 > A method for profiling SSL/TLS Clients > https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41/ ## JA3 fingerprint如何得到 ? ![image](https://hackmd.io/_uploads/H1E-pJYyC.png) The field order is as follows: > SSLVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats Example: > 769,47–53–5–10–49161–49162–49171–49172–50–56–19–4,0–10–11,23–24–25,0 MD5 Hash: > 769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0 --> ada70206e40642a3e4461f35503241d5 > 769,4-5-10-9-100-98-3-6-19-18-99,,, --> de350869b8c85de67a350c8d186f11e6 > ## 安裝JA3套件 安裝: ```shell pip install ja3 ja3 --json capture-ws-prod.pcap > capture-ws-prod.json ``` 取得這些有同樣的ja3 fingerprint的request是從哪裡來的 ? ```shell cat capture-ws-prod.json | jq -c '.[] | select(.ja3_digest | contains("a specificed string")) | .source_ip' | sort | uniq > "130.255.x.x" > "145.224.x.x" > "182.173.x.x" > "182.68.x.x" > "188.209.x.x" > ... ``` # 結論 學到了什麼? 1. 事實上，雖然我們嘗試錄了loadbalancer封包，我們無法看切確是什麼error造成的，只知道在某個特定的`JA3 fingerprint`下會有這個error。 2. 原來loadbalancer是一排EC2，隨著用量變大，機器會越來越多 ! 3. `JA3 fringerprint`目前會使用資安領域，例如特定的設備、供應商client的網路足跡；作為資訊安全的一個計分。 4. 我觀察到AWS WAF 也支援這個 JA3 fingerprint: ![image](https://hackmd.io/_uploads/r1aKYWFkR.png) 5. VXLAN是目前資料中心廣泛採用的一種技術，實現實體網路跟抽象網路規劃的decoupling。 > https://info.support.huawei.com/info-finder/encyclopedia/zh/VXLAN.html # Reference 1. https://repost.aws/knowledge-center/elb-fix-ssl-tls-negotiation-error 2. https://sjakthol.github.io/posts/alb-vpc-traffic-mirroring/ 3. https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41/ 4. https://aws.amazon.com/tw/about-aws/whats-new/2023/09/aws-waf-ja3-fingerprint-match/ 6. https://datatracker.ietf.org/doc/html/rfc7348 8. https://docs.aws.amazon.com/zh_tw/vpc/latest/mirroring/traffic-mirroring-how-it-works.html