---
# System prepended metadata

title: '事件紀錄: docker 的 container/host 雙向無法連通'
tags: [Container, Network]

---

# 事件紀錄: docker 的 container/host 雙向無法連通
###### tags: `Network` `Container`

## 已知問題



1. ~~ebtables 的 OUPUT 階段 MAC dest 未知, 之後路徑幾乎的亂了~~

這是我弄錯了, 看到 ARP 封包

詳細 Log 在最下面

更新後的路徑, 死在 iptables docker0 路徑不通, 結果走 lo  

補充: arp 解析對於 172.20.168.2 是 incomplete

![](https://i.imgur.com/sAcmhlz.png)


如果 arp 已經可以正常解析 172.20.168.2 時, 路徑如下, 此時觀察 bridge fdb, 沒有路徑可以走

![](https://i.imgur.com/rAxdesH.png)


2. docker 啟動時的網路狀態

錯誤, container 的 veth 進入兩次 forwarding state

```bash
[349716.505623] IPv6: ADDRCONF(NETDEV_CHANGE): veth382c21d: link becomes ready
[349716.505669] docker0: port 1(veth382c21d) entered forwarding state **正常應該進入 blocking state
[349716.505685] docker0: port 1(veth382c21d) entered forwarding state
```

正常

```bash=
[2243171.314113] IPv6: ADDRCONF(NETDEV_CHANGE): vethf8c71c7: link becomes ready
[2243171.314242] docker0: port 1(vethf8c71c7) entered blocking state
[2243171.314247] docker0: port 1(vethf8c71c7) entered forwarding state
```

3. `arp -nv` container ip 顯示 incomplete


> 172.20.168.2                     (incomplete)                              docker0

4. host arping container 沒有回應

```bash=
arping -I docker0 -c 1  172.20.168.2
ARPING 172.20.168.2 from 172.20.168.1 docker0
Sent 1 probes (1 broadcast(s))
Received 0 response(s)
```

host arptables

```bash=
> arptables-save
*filter
:INPUT ACCEPT
:OUTPUT ACCEPT
:FORWARD ACCEPT
```

## 環境資訊

驗證環境

![](https://i.imgur.com/bb70BEP.png)


環境資訊

* Cetnos7
* Docker 20.10.7
* 全部操作都是使用 root 

系統相關參數

```bash=
> uname -a
Linux remote-docker-165 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

> lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.2.1511 (Core)
Release:        7.2.1511
Codename:       Core

> cat /proc/version
Linux version 3.10.0-327.18.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu May 12 11:03:55 UTC 2016

> cat /proc/sys/net/ipv4/ip_forward
1

> sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

> cat /proc/sys/net/ipv4/conf/docker0/forwarding
1

> cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

> sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0

> cat /proc/sys/net/ipv4/conf/docker0/arp_accept
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_ignore
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_notify
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_filter
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_announce
0
```

/etc/docker/daemon.json

```json=
{
    "debug": true
}
```

docker daemon log

> journalctl -u docker.service

Host 網卡

```bash=
> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:90:53:b9 brd ff:ff:ff:ff:ff:ff
    inet 10.62.198.165/22 brd 10.62.199.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe90:53b9/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:52:0e:03:e9 brd ff:ff:ff:ff:ff:ff
    inet 172.20.168.1/24 brd 172.20.168.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:52ff:fe0e:3e9/64 scope link
       valid_lft forever preferred_lft forever
```

Host, 路由

```bash=
> route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.62.196.1     0.0.0.0         UG    100    0        0 ens32
10.62.196.0     0.0.0.0         255.255.252.0   U     100    0        0 ens32
172.20.168.0    0.0.0.0         255.255.255.0   U     0      0        0 docker0
```

啟動測試 container 使用預設 docker0(bridge)

>  docker run -d -p 60000:8000 --name c1 hwchiu/netutils

host 網卡資訊多了 veth70c8f21

```bash=
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:90:53:b9 brd ff:ff:ff:ff:ff:ff
    inet 10.62.198.165/22 brd 10.62.199.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe90:53b9/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:52:0e:03:e9 brd ff:ff:ff:ff:ff:ff
    inet 172.20.168.1/24 brd 172.20.168.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:52ff:fe0e:3e9/64 scope link
       valid_lft forever preferred_lft forever
65: veth70c8f21@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
    link/ether a2:15:23:4a:40:fc brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::a015:23ff:fe4a:40fc/64 scope link
       valid_lft forever preferred_lft forever
```

Bridge 連接的網卡

```bash=
> bridge link
65: veth70c8f21 state UP @(null): <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master docker0 state forwarding priority 32 cost 2
```

bridge forward database

```bash
> bridge fdb
01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:ff:90:53:b9 dev ens32 self permanent
01:00:5e:00:00:fb dev ens32 self permanent
a2:15:23:4a:40:fc dev veth70c8f21 vlan 1 master docker0 permanent
02:42:52:0e:03:e9 dev docker0 vlan 1 master docker0 permanent
a2:15:23:4a:40:fc dev veth70c8f21 master docker0 permanent
33:33:00:00:00:01 dev veth70c8f21 self permanent
01:00:5e:00:00:01 dev veth70c8f21 self permanent
33:33:ff:4a:40:fc dev veth70c8f21 self permanent
```

Container c1 的網卡資訊

```bash=
> docker exec c1 ip addr
[root@remote-docker-165 aspnet_docker]# docker exec c1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
64: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:14:a8:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.168.2/24 brd 172.20.168.255 scope global eth0
       valid_lft forever preferred_lft forever
```

```bash=
> docker exec c1 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.20.168.1    0.0.0.0         UG    0      0        0 eth0
172.20.168.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
```

c1 重啟時 `dmesg` 資訊

```bash=
[349716.192948] device veth382c21d entered promiscuous mode
[349716.193079] IPv6: ADDRCONF(NETDEV_UP): veth382c21d: link is not ready
[349716.193085] docker0: port 1(veth382c21d) entered forwarding state ** 
[349716.193097] docker0: port 1(veth382c21d) entered forwarding state
[349716.193630] docker0: port 1(veth382c21d) entered disabled state
[349716.505623] IPv6: ADDRCONF(NETDEV_CHANGE): veth382c21d: link becomes ready
[349716.505669] docker0: port 1(veth382c21d) entered forwarding state **正常應該進入 blocking state
[349716.505685] docker0: port 1(veth382c21d) entered forwarding state
```

host iptables 規則

```bash=
*filter
:INPUT ACCEPT [2152:323663]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [482:185857]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.20.168.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT

*nat
:PREROUTING ACCEPT [125:13831]
:INPUT ACCEPT [124:13602]
:OUTPUT ACCEPT [24:1640]
:POSTROUTING ACCEPT [24:1640]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.2/32 -o ens32 -j MASQUERADE
-A POSTROUTING -s 172.20.168.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.20.168.2/32 -d 172.20.168.2/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 60000 -j DNAT --to-destination 172.20.168.2:8000
COMMIT
```

host ebtables 規則

```bash=
*nat
:PREROUTING ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
-A PREROUTING --log-level debug --log-prefix "ebtables/nat-PREROUTE" -j CONTINUE
-A OUTPUT --log-level debug --log-prefix "ebtables/nat-OUTPUT" -j CONTINUE
-A POSTROUTING --log-level debug --log-prefix "ebtables/nat-POSTROUTE" -j CONTINUE

*broute
:BROUTING ACCEPT
-A BROUTING --log-level debug --log-prefix "ebtables/broute-BROUTING" -j CONTINUE

*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
-A INPUT --log-level debug --log-prefix "ebtables/filter-INPUT" -j CONTINUE
-A FORWARD --log-level debug --log-prefix "ebtables/filter-FORWARD" -j CONTINUE
-A OUTPUT --log-level debug --log-prefix "ebtables/filter-OUTPUT" -j CONTINUE
```

host ARP

```bash=
> arp -nv
172.20.168.2                     (incomplete)                              docker0
```

## 案例: host ping container

```bash
> ping -c 1 172.20.168.2
PING 172.20.168.2 (172.20.168.2) 56(84) bytes of data.

--- 172.20.168.2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
```

ebtables+iptables 封包流向

```bash=
## (1) ICMP Request
# 1.1 iptable OUTPUT
[345287.767195] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767210] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767218] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767227] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1

# 1.2 iptable POSTROUTE
#     172.20.168.1 >(docker0)>172.20.168.2
[345287.767232] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767238] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1

# ARP
[345287.767259] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767264] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767267] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767284] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345287.767287] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345288.768627] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768710] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768803] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768839] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345288.768842] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345289.770114] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770136] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770147] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770182] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345289.770186] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806

# 1.3 
[345290.771723] iptable/raw-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771753] iptable/mangle-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771772] iptable/filter-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
# 1.4
[345290.771787] iptable/mangle-POSTROUTE IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]

# (2)

[345290.771834] iptable/raw-PREROUTE IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771844] iptable/mangle-PREROUTE IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]

[345290.771854] iptable/mangle-INPUT IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771864] iptable/filter-INPUT IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
```



```bash
> docker exec c1 ethtool --offload eth0 rx off tx off
Cannot set device feature settings: Operation not permitted
```

如果在 ` --privileged` 模式下執行 `ethtool --offload eth0 rx off tx off`, host ping container 一樣不通

```bash=
Actual changes:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ip-generic: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off [requested on]
```

host arp

```bash=
> arp -nv
Address                  HWtype  HWaddress           Flags Mask            Iface
172.20.168.2                     (incomplete)                              docker0
```

container arp

```bash
> docker exec c1 arp -nv
Address                  HWtype  HWaddress           Flags Mask            Iface
172.20.168.1             ether   02:42:52:0e:03:e9   C                     eth0
```

如果 arp 有正常解析

```bash=
> arp -nv
172.20.168.2             ether   02:42:ac:14:a8:02   C                     docker0
```
ebtables+iptables 流向

```bash
## ICMP Send
# iptables OUTPUT
[367546.961508] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961524] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961531] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961541] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1

# iptables POSTROUTE
[367546.961546] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961552] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1

# ebtables OUTPUT
[367546.961602] ebtables/nat-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[367546.961607] ebtables/filter-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800

# ebtables POSTROUTE
[367546.961611] ebtables/nat-POSTROUTE IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800

## ARP
[367546.961669] ebtables/broute-BROUTING IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961673] ebtables/nat-PREROUTE IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961682] ebtables/filter-INPUT IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961695] ebtables/nat-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[367546.961697] ebtables/filter-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[367546.961700] ebtables/nat-POSTROUTE IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806

## Reply
# ebtables BROUTING
[367546.961709] ebtables/broute-BROUTING IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

# ebtables PREROUTE
[367546.961712] ebtables/nat-PREROUTE IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

# iptable PREROUTE
[367546.961723] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=vethe371854 MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=29230 SEQ=1
[367546.961732] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=vethe371854 MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=29230 SEQ=1
```

手動加入 static ARP

```bash=
> arp -s 172.20.168.2 02:42:ac:14:a8:02
```

ebtables+iptables

```bash=
## ICMP Request (SEQ=1) 
[374986.715729] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715749] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715757] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715766] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1

[374986.715772] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715777] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1

[374986.715795] ebtables/nat-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715800] ebtables/filter-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715804] ebtables/nat-POSTROUTE IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715857] ebtables/broute-BROUTING IN=vethcf3805f OUT= MAC source = 

# ARP
02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715861] ebtables/nat-PREROUTE IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715870] ebtables/filter-INPUT IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715879] ebtables/nat-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[374986.715882] ebtables/filter-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[374986.715885] ebtables/nat-POSTROUTE IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806

## ICMP Reply 
[374986.715895] ebtables/broute-BROUTING IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

[374986.715898] ebtables/nat-PREROUTE IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

[374986.715908] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=vethcf3805f MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=20178 SEQ=1
[374986.715918] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=vethcf3805f MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=20178 SEQ=1

## ICMP Request(SEQ=20) 
# 10.62.197.100 >(ens32)> 10.62.198.165
[374998.064041] iptable/raw-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064074] iptable/mangle-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064092] iptable/nat-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20

# 10.62.197.100 >(ens32)> 10.62.198.165
[374998.064112] iptable/mangle-INPUT IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064127] iptable/filter-INPUT IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20

## ICMP Reply
[374998.064158] iptable/raw-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20

[374998.064172] iptable/mangle-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
[374998.064185] iptable/filter-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
[374998.064197] iptable/mangle-POSTROUTE IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
```


## 案例: container ping default gateway

```bash=
## ICMP Request

[351096.316295] ebtables/broute-BROUTING IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806

[351096.316303] ebtables/nat-PREROUTE IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806

[351096.316316] ebtables/filter-INPUT IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806

[351096.316330] ebtables/nat-OUTPUT IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[351096.316333] ebtables/filter-OUTPUT IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806

[351096.316336] ebtables/nat-POSTROUTE IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806

[351096.316347] ebtables/broute-BROUTING IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

[351096.316350] ebtables/nat-PREROUTE IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800

[351096.316364] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
[351096.316379] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
[351096.316388] iptable/nat-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
```