---
# System prepended metadata

title: 緣由

---


<style>
img {
    border-radius: 10px;
    border: 2px solid #BDC3C7;
}
</style>
![generate (2)](https://hackmd.io/_uploads/HJvtStmeC.jpg)

# 緣由
最近在工作上，例行的檢查AWS **applicaiton loadbalancer** 的monitoring tab時，發現了其中一格神祕的資訊:

![image](https://hackmd.io/_uploads/Sy3YLgN1A.png)

事實上，這個資訊看過多次，但是一直不了解它代表了甚麼意義。查詢了aws 的 **knowledge base**，aws建議可以用**ssl-scan**去檢查attach在loadbalancer上面的certificate。然而，我們的certificate是**ACM** (AWS Certificate Manager)所派發的，基本上不會有問題才是。
> https://repost.aws/knowledge-center/elb-fix-ssl-tls-negotiation-error

接著，我去下載了loadbalancer的access log，但log沒有看到任何關於*Client TLS negotiation errors*的紀錄。

> 例如，access log的欄位解讀: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html


查找了網路上的資訊，發現有人利用AWS的VPC traffic mirroring 來capture loadbalancer的封包，除錯 *Client TLS negotiation errors*的發生原因。基於好奇心，我們來做個實驗吧!
# VPC Traffic Mirroring 概念
- 簡單來說，只要是resource具有`ENI` (Elastic Network Interface)，理論上都可以透過traffic mirroring的方式擷取封包。

![image](https://hackmd.io/_uploads/Hyv8K1Yk0.png)
(出處: https://www.qacafe.com/resources/aws-vpc-traffic-mirroring-cloud-packet-capture/)
- 決定source以及target的`ENI`。
- Source是applicaiton loadbalancer。
    - application loadbalancer的ENI在哪裡可以查到?
- Target是同VPC內的某台EC2。

## VXLAN

AWS 利用 VXLAN protocol 來包裝封包:

![image](https://hackmd.io/_uploads/rk-CBZt1R.png)

> https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-packet-formats.html


## 取得 application loadbalancer的資訊
```shell
aws elbv2 describe-load-balancers --names mvb-eco-sys-ws-prod --region eu-central-1
```
```JSON
{
    "LoadBalancers": [
        {
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:eu-central-1:******************:loadbalancer/app/mvb-eco-sys-ws-prod/741****************",
            "DNSName": "************.eu-central-1.elb.amazonaws.com",
            "CanonicalHostedZoneId": "**************",
            "CreatedTime": "2021-01-13T10:34:40.780000+00:00",
            "LoadBalancerName": "mvb-eco-sys-ws-prod",
            "Scheme": "internet-facing",
            "VpcId": "vpc-***************",
            "State": {
            "Code": "active"
        },
        "Type": "application",
        "AvailabilityZones": [
        {
            "ZoneName": "eu-central-1a",
            "SubnetId": "subnet-******************",
            "LoadBalancerAddresses": []
        },
        {
            "ZoneName": "eu-central-1b",
            "SubnetId": "subnet-******************",
            "LoadBalancerAddresses": []
        }
        ],
            "SecurityGroups": [
            "sg-******************"
        ],
            "IpAddressType": "ipv4"
        }
    ]
}
```


## 取得applicaiton loadbalancer ENI列表
```shell
aws ec2 describe-network-interfaces --filters Name=description,Values="ELB app/mvb-eco-sys-ws-prod/******************" --query 'NetworkInterfaces[*]' --output json --region eu-central-1 | jq -c '.[].NetworkInterfaceId'
```

> "eni-0a2bf36cdb1c57099"
> "eni-00d05507cd1140898"
> "eni-080477509901b6a76"
> "eni-0e2c5f664c663abc9"
> "eni-0905dc9871ecab671"
> "eni-0292a0604baf86b91"
> "eni-0a1f1ad189baaa7df"
> "eni-0a2c8d84bd7953b08"
> "eni-05cccf1718fbaf35f"
> "eni-06862fbbad18b5c46"
> "eni-0c36fed47d6d0de6b"
> "eni-081934841f840b09f"
> "eni-09cbe68ac9c8f0b44"

這台loadbalancer用量比較大，所以有13張ENI !

## 什麼 ? 原來application loadbalancer是一排EC2?
其實從以前就覺得奇怪，為什麼loadbalancer跟EC2是放在同一個介面上 (EC2 Dashboard
)。原來loadbalancer本質上就是一排的EC2!

透過下面的資料可以觀察到:

```shell
aws ec2 describe-network-interfaces --filters Name=description,Values="ELB app/mvb-eco-sys-ws-prod/******************" --query 'NetworkInterfaces[*]' --output json --region eu-central-1 | jq -c '.[]."Association"."PublicDnsName"'
```

> "ec2-3-121-54-139.eu-central-1.compute.amazonaws.com"
> "ec2-3-77-159-52.eu-central-1.compute.amazonaws.com"
> "ec2-18-153-166-253.eu-central-1.compute.amazonaws.com"
> "ec2-3-120-200-244.eu-central-1.compute.amazonaws.com"
> "ec2-54-93-93-216.eu-central-1.compute.amazonaws.com"
> "ec2-52-28-50-142.eu-central-1.compute.amazonaws.com"
> "ec2-3-120-19-81.eu-central-1.compute.amazonaws.com"
> "ec2-18-198-21-139.eu-central-1.compute.amazonaws.com"
> "ec2-52-57-17-95.eu-central-1.compute.amazonaws.com"
> "ec2-3-124-82-62.eu-central-1.compute.amazonaws.com"
> "ec2-52-28-107-255.eu-central-1.compute.amazonaws.com"
> "ec2-18-157-172-87.eu-central-1.compute.amazonaws.com"
> "ec2-52-28-217-154.eu-central-1.compute.amazonaws.com"
> 


# 動手做
順序:
> 1. 開啟&設定一台EC2，作為traffic mirroring 的 destination and tcpdump的執行機器
> 2. 設定VPC traffic mirroring ，啟動 mirroring。
> 3. 回到EC2，設定一個vxlan interface，使用tcpdump擷取packets。

## *EC2 - part 1*
1. 開啟一台EC2，設定好security group。
2. ssh進入EC2，安裝 conntrack and tcpdump。

```shell
sudo iptables -t filter -A INPUT -m state --state NEW\,ESTABLISHED -j ACCEPT && sudo iptables -t filter -A FORWARD -m state --state NEW\,ESTABLISHED -j ACCEPT && sudo iptables -t filter -A OUTPUT -m state --state NEW\,ESTABLISHED -j ACCEPT
```
```shell
/bin/echo "1" > /proc/sys/net/netfilter/nf_conntrack_acct
```
3. 這個時候應該還沒有任何traffic

![image](https://hackmd.io/_uploads/HJIfhbKyC.png)



## *VPC traffic mirroring Configuration*
### Mirror targets
設定你的EC2為 mirror target
![image](https://hackmd.io/_uploads/ry99o-Y10.png)

### Mirror filters
1. 設定如下圖的 reject & accept，只擷取external traffic的資料。
2. Internal traffic VPC CIDR都去掉。

![image](https://hackmd.io/_uploads/H1Nrq1t10.png)

### Mirror sessions
1. 從上述的資訊，可以確定你選到正確的ENI:

![螢幕擷取畫面 2024-04-02 164519](https://hackmd.io/_uploads/BJ7eLBK1R.png)



## *EC2 - part 2*
1. 檢查 EC2 上面的 port 4789 是否已經有mirroring來的traffic。
```shell
sudo conntrack -L -p udp
```
![image](https://hackmd.io/_uploads/HkOr6ZFyA.png)


2. 新增一個VXLAN interface
```shell
sudo ip link add ${interface_name} type 
 id ${vni} local ${EC2's local ip address} remote ${source_ip_address} dev ${interface_name_of_EC2_eni} dstport 4789
sudo ip link set ${interface_name} up
```
3. 開始抓封包
```shell
sudo tcpdump -n -i ${interface_name}  -vv -w ${filename}.pcap
```

# 分析pcap

1. 把 \*.pcap下載下來，使用wireshark打開pcap檔。輸入以下的指令做filter:
```
tls.alert_message.level == 2
```
2. 輸入完後，可以發現出現了很多**Certificate Unknown - 21**的alert !

![螢幕擷取畫面 2024-03-29 170916](https://hackmd.io/_uploads/SkVMLZEJC.png)


- 打開封包檔:
![image](https://hackmd.io/_uploads/B1BD5etkC.png)

3. follow 某一個TLS stream，可以發現以下的封包:

![螢幕擷取畫面 2024-03-29 171839](https://hackmd.io/_uploads/Byvl_bN1C.png)

## TLS Alert
> Error code 21 for TLS
> https://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol
> 

![image](https://hackmd.io/_uploads/rJCo-WFJC.png)



# 題外話 - JA3 
- 2017年由Saleforces的工程師提出。
> A method for profiling SSL/TLS Clients
> https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41/

## JA3 fingerprint如何得到 ?
![image](https://hackmd.io/_uploads/H1E-pJYyC.png)

The field order is as follows:
> SSLVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats

Example:
> 769,47–53–5–10–49161–49162–49171–49172–50–56–19–4,0–10–11,23–24–25,0

MD5 Hash:
> 769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0 --> ada70206e40642a3e4461f35503241d5
> 769,4-5-10-9-100-98-3-6-19-18-99,,, --> de350869b8c85de67a350c8d186f11e6
> 
## 安裝JA3套件

安裝:
```shell
pip install ja3
ja3 --json capture-ws-prod.pcap > capture-ws-prod.json
```

取得這些有同樣的ja3 fingerprint的request是從哪裡來的 ?
```shell
cat capture-ws-prod.json | jq -c '.[] | select(.ja3_digest | contains("a specificed string")) | .source_ip' | sort | uniq

> "130.255.x.x"
> "145.224.x.x"
> "182.173.x.x"
> "182.68.x.x"
> "188.209.x.x"
> ...

```
# 結論
學到了什麼?
 1. 事實上，雖然我們嘗試錄了loadbalancer封包，我們無法看切確是什麼error造成的，只知道在某個特定的`JA3 fingerprint`下會有這個error。
 2. 原來loadbalancer是一排EC2，隨著用量變大，機器會越來越多 !
 3. `JA3 fringerprint`目前會使用資安領域，例如特定的設備、供應商client的網路足跡；作為資訊安全的一個計分。
 4. 我觀察到AWS WAF 也支援這個 JA3 fingerprint:
 ![image](https://hackmd.io/_uploads/r1aKYWFkR.png)
 5. VXLAN是目前資料中心廣泛採用的一種技術，實現實體網路跟抽象網路規劃的decoupling。
    > https://info.support.huawei.com/info-finder/encyclopedia/zh/VXLAN.html

# Reference
1. https://repost.aws/knowledge-center/elb-fix-ssl-tls-negotiation-error
2. https://sjakthol.github.io/posts/alb-vpc-traffic-mirroring/
3. https://engineering.salesforce.com/open-sourcing-ja3-92c9e53c3c41/
4. https://aws.amazon.com/tw/about-aws/whats-new/2023/09/aws-waf-ja3-fingerprint-match/
6. https://datatracker.ietf.org/doc/html/rfc7348
8. https://docs.aws.amazon.com/zh_tw/vpc/latest/mirroring/traffic-mirroring-how-it-works.html