# 事件紀錄: docker 的 container/host 雙向無法連通
###### tags: `Network` `Container`
## 已知問題
1. ~~ebtables 的 OUPUT 階段 MAC dest 未知, 之後路徑幾乎的亂了~~
這是我弄錯了, 看到 ARP 封包
詳細 Log 在最下面
更新後的路徑, 死在 iptables docker0 路徑不通, 結果走 lo
補充: arp 解析對於 172.20.168.2 是 incomplete
如果 arp 已經可以正常解析 172.20.168.2 時, 路徑如下, 此時觀察 bridge fdb, 沒有路徑可以走
2. docker 啟動時的網路狀態
錯誤, container 的 veth 進入兩次 forwarding state
```bash
[349716.505623] IPv6: ADDRCONF(NETDEV_CHANGE): veth382c21d: link becomes ready
[349716.505669] docker0: port 1(veth382c21d) entered forwarding state **正常應該進入 blocking state
[349716.505685] docker0: port 1(veth382c21d) entered forwarding state
```
正常
```bash=
[2243171.314113] IPv6: ADDRCONF(NETDEV_CHANGE): vethf8c71c7: link becomes ready
[2243171.314242] docker0: port 1(vethf8c71c7) entered blocking state
[2243171.314247] docker0: port 1(vethf8c71c7) entered forwarding state
```
3. `arp -nv` container ip 顯示 incomplete
> 172.20.168.2 (incomplete) docker0
4. host arping container 沒有回應
```bash=
arping -I docker0 -c 1 172.20.168.2
ARPING 172.20.168.2 from 172.20.168.1 docker0
Sent 1 probes (1 broadcast(s))
Received 0 response(s)
```
host arptables
```bash=
> arptables-save
*filter
:INPUT ACCEPT
:OUTPUT ACCEPT
:FORWARD ACCEPT
```
## 環境資訊
驗證環境
環境資訊
* Cetnos7
* Docker 20.10.7
* 全部操作都是使用 root
系統相關參數
```bash=
> uname -a
Linux remote-docker-165 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core
> cat /proc/version
Linux version 3.10.0-327.18.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu May 12 11:03:55 UTC 2016
> cat /proc/sys/net/ipv4/ip_forward
1
> sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
> cat /proc/sys/net/ipv4/conf/docker0/forwarding
1
> cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
> sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
> cat /proc/sys/net/ipv4/conf/docker0/arp_accept
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_ignore
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_notify
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_filter
0
> cat /proc/sys/net/ipv4/conf/docker0/arp_announce
0
```
/etc/docker/daemon.json
```json=
{
"debug": true
}
```
docker daemon log
> journalctl -u docker.service
Host 網卡
```bash=
> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:90:53:b9 brd ff:ff:ff:ff:ff:ff
inet 10.62.198.165/22 brd 10.62.199.255 scope global ens32
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe90:53b9/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:52:0e:03:e9 brd ff:ff:ff:ff:ff:ff
inet 172.20.168.1/24 brd 172.20.168.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:52ff:fe0e:3e9/64 scope link
valid_lft forever preferred_lft forever
```
Host, 路由
```bash=
> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.62.196.1 0.0.0.0 UG 100 0 0 ens32
10.62.196.0 0.0.0.0 255.255.252.0 U 100 0 0 ens32
172.20.168.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
```
啟動測試 container 使用預設 docker0(bridge)
> docker run -d -p 60000:8000 --name c1 hwchiu/netutils
host 網卡資訊多了 veth70c8f21
```bash=
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:90:53:b9 brd ff:ff:ff:ff:ff:ff
inet 10.62.198.165/22 brd 10.62.199.255 scope global ens32
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe90:53b9/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:52:0e:03:e9 brd ff:ff:ff:ff:ff:ff
inet 172.20.168.1/24 brd 172.20.168.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:52ff:fe0e:3e9/64 scope link
valid_lft forever preferred_lft forever
65: veth70c8f21@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
link/ether a2:15:23:4a:40:fc brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::a015:23ff:fe4a:40fc/64 scope link
valid_lft forever preferred_lft forever
```
Bridge 連接的網卡
```bash=
> bridge link
65: veth70c8f21 state UP @(null): <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master docker0 state forwarding priority 32 cost 2
```
bridge forward database
```bash
> bridge fdb
01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:ff:90:53:b9 dev ens32 self permanent
01:00:5e:00:00:fb dev ens32 self permanent
a2:15:23:4a:40:fc dev veth70c8f21 vlan 1 master docker0 permanent
02:42:52:0e:03:e9 dev docker0 vlan 1 master docker0 permanent
a2:15:23:4a:40:fc dev veth70c8f21 master docker0 permanent
33:33:00:00:00:01 dev veth70c8f21 self permanent
01:00:5e:00:00:01 dev veth70c8f21 self permanent
33:33:ff:4a:40:fc dev veth70c8f21 self permanent
```
Container c1 的網卡資訊
```bash=
> docker exec c1 ip addr
[root@remote-docker-165 aspnet_docker]# docker exec c1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
64: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:14:a8:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.168.2/24 brd 172.20.168.255 scope global eth0
valid_lft forever preferred_lft forever
```
```bash=
> docker exec c1 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.20.168.1 0.0.0.0 UG 0 0 0 eth0
172.20.168.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
```
c1 重啟時 `dmesg` 資訊
```bash=
[349716.192948] device veth382c21d entered promiscuous mode
[349716.193079] IPv6: ADDRCONF(NETDEV_UP): veth382c21d: link is not ready
[349716.193085] docker0: port 1(veth382c21d) entered forwarding state **
[349716.193097] docker0: port 1(veth382c21d) entered forwarding state
[349716.193630] docker0: port 1(veth382c21d) entered disabled state
[349716.505623] IPv6: ADDRCONF(NETDEV_CHANGE): veth382c21d: link becomes ready
[349716.505669] docker0: port 1(veth382c21d) entered forwarding state **正常應該進入 blocking state
[349716.505685] docker0: port 1(veth382c21d) entered forwarding state
```
host iptables 規則
```bash=
*filter
:INPUT ACCEPT [2152:323663]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [482:185857]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.20.168.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
*nat
:PREROUTING ACCEPT [125:13831]
:INPUT ACCEPT [124:13602]
:OUTPUT ACCEPT [24:1640]
:POSTROUTING ACCEPT [24:1640]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.2/32 -o ens32 -j MASQUERADE
-A POSTROUTING -s 172.20.168.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.20.168.2/32 -d 172.20.168.2/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 60000 -j DNAT --to-destination 172.20.168.2:8000
COMMIT
```
host ebtables 規則
```bash=
*nat
:PREROUTING ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
-A PREROUTING --log-level debug --log-prefix "ebtables/nat-PREROUTE" -j CONTINUE
-A OUTPUT --log-level debug --log-prefix "ebtables/nat-OUTPUT" -j CONTINUE
-A POSTROUTING --log-level debug --log-prefix "ebtables/nat-POSTROUTE" -j CONTINUE
*broute
:BROUTING ACCEPT
-A BROUTING --log-level debug --log-prefix "ebtables/broute-BROUTING" -j CONTINUE
*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
-A INPUT --log-level debug --log-prefix "ebtables/filter-INPUT" -j CONTINUE
-A FORWARD --log-level debug --log-prefix "ebtables/filter-FORWARD" -j CONTINUE
-A OUTPUT --log-level debug --log-prefix "ebtables/filter-OUTPUT" -j CONTINUE
```
host ARP
```bash=
> arp -nv
172.20.168.2 (incomplete) docker0
```
## 案例: host ping container
```bash
> ping -c 1 172.20.168.2
PING 172.20.168.2 (172.20.168.2) 56(84) bytes of data.
--- 172.20.168.2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
```
ebtables+iptables 封包流向
```bash=
## (1) ICMP Request
# 1.1 iptable OUTPUT
[345287.767195] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767210] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767218] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767227] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
# 1.2 iptable POSTROUTE
# 172.20.168.1 >(docker0)>172.20.168.2
[345287.767232] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
[345287.767238] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1
# ARP
[345287.767259] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767264] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767267] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345287.767284] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345287.767287] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345288.768627] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768710] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768803] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345288.768839] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345288.768842] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345289.770114] ebtables/nat-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770136] ebtables/filter-OUTPUT IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770147] ebtables/nat-POSTROUTE IN= OUT=veth70c8f21 MAC source = 02:42:52:0e:03:e9 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[345289.770182] ebtables/broute-BROUTING IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
[345289.770186] ebtables/nat-PREROUTE IN=veth70c8f21 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0806
# 1.3
[345290.771723] iptable/raw-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771753] iptable/mangle-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771772] iptable/filter-OUTPUT IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
# 1.4
[345290.771787] iptable/mangle-POSTROUTE IN= OUT=lo SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
# (2)
[345290.771834] iptable/raw-PREROUTE IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771844] iptable/mangle-PREROUTE IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771854] iptable/mangle-INPUT IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
[345290.771864] iptable/filter-INPUT IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.168.1 DST=172.20.168.1 LEN=112 TOS=0x00 PREC=0xC0 TTL=64 ID=12469 PROTO=ICMP TYPE=3 CODE=1 [SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26375 DF PROTO=ICMP TYPE=8 CODE=0 ID=15565 SEQ=1 ]
```
```bash
> docker exec c1 ethtool --offload eth0 rx off tx off
Cannot set device feature settings: Operation not permitted
```
如果在 ` --privileged` 模式下執行 `ethtool --offload eth0 rx off tx off`, host ping container 一樣不通
```bash=
Actual changes:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ip-generic: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off [requested on]
```
host arp
```bash=
> arp -nv
Address HWtype HWaddress Flags Mask Iface
172.20.168.2 (incomplete) docker0
```
container arp
```bash
> docker exec c1 arp -nv
Address HWtype HWaddress Flags Mask Iface
172.20.168.1 ether 02:42:52:0e:03:e9 C eth0
```
如果 arp 有正常解析
```bash=
> arp -nv
172.20.168.2 ether 02:42:ac:14:a8:02 C docker0
```
ebtables+iptables 流向
```bash
## ICMP Send
# iptables OUTPUT
[367546.961508] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961524] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961531] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961541] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
# iptables POSTROUTE
[367546.961546] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
[367546.961552] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26379 DF PROTO=ICMP TYPE=8 CODE=0 ID=29230 SEQ=1
# ebtables OUTPUT
[367546.961602] ebtables/nat-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[367546.961607] ebtables/filter-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
# ebtables POSTROUTE
[367546.961611] ebtables/nat-POSTROUTE IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
## ARP
[367546.961669] ebtables/broute-BROUTING IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961673] ebtables/nat-PREROUTE IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961682] ebtables/filter-INPUT IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[367546.961695] ebtables/nat-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[367546.961697] ebtables/filter-OUTPUT IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[367546.961700] ebtables/nat-POSTROUTE IN= OUT=vethe371854 MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
## Reply
# ebtables BROUTING
[367546.961709] ebtables/broute-BROUTING IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
# ebtables PREROUTE
[367546.961712] ebtables/nat-PREROUTE IN=vethe371854 OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
# iptable PREROUTE
[367546.961723] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=vethe371854 MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=29230 SEQ=1
[367546.961732] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=vethe371854 MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=29230 SEQ=1
```
手動加入 static ARP
```bash=
> arp -s 172.20.168.2 02:42:ac:14:a8:02
```
ebtables+iptables
```bash=
## ICMP Request (SEQ=1)
[374986.715729] iptable/raw-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715749] iptable/mangle-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715757] iptable/nat-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715766] iptable/filter-OUTPUT IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715772] iptable/mangle-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715777] iptable/nat-POSTROUTE IN= OUT=docker0 SRC=172.20.168.1 DST=172.20.168.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32187 DF PROTO=ICMP TYPE=8 CODE=0 ID=20178 SEQ=1
[374986.715795] ebtables/nat-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715800] ebtables/filter-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715804] ebtables/nat-POSTROUTE IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0800
[374986.715857] ebtables/broute-BROUTING IN=vethcf3805f OUT= MAC source =
# ARP
02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715861] ebtables/nat-PREROUTE IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715870] ebtables/filter-INPUT IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[374986.715879] ebtables/nat-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[374986.715882] ebtables/filter-OUTPUT IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[374986.715885] ebtables/nat-POSTROUTE IN= OUT=vethcf3805f MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
## ICMP Reply
[374986.715895] ebtables/broute-BROUTING IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
[374986.715898] ebtables/nat-PREROUTE IN=vethcf3805f OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
[374986.715908] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=vethcf3805f MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=20178 SEQ=1
[374986.715918] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=vethcf3805f MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 PROTO=ICMP TYPE=0 CODE=0 ID=20178 SEQ=1
## ICMP Request(SEQ=20)
# 10.62.197.100 >(ens32)> 10.62.198.165
[374998.064041] iptable/raw-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064074] iptable/mangle-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064092] iptable/nat-PREROUTE IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
# 10.62.197.100 >(ens32)> 10.62.198.165
[374998.064112] iptable/mangle-INPUT IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
[374998.064127] iptable/filter-INPUT IN=ens32 OUT= MAC=00:50:56:90:53:b9:00:50:56:8a:4d:a7:08:00 SRC=10.62.197.100 DST=10.62.198.165 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17795 DF PROTO=ICMP TYPE=8 CODE=0 ID=6823 SEQ=20
## ICMP Reply
[374998.064158] iptable/raw-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
[374998.064172] iptable/mangle-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
[374998.064185] iptable/filter-OUTPUT IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
[374998.064197] iptable/mangle-POSTROUTE IN= OUT=ens32 SRC=10.62.198.165 DST=10.62.197.100 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=61706 PROTO=ICMP TYPE=0 CODE=0 ID=6823 SEQ=20
```
## 案例: container ping default gateway
```bash=
## ICMP Request
[351096.316295] ebtables/broute-BROUTING IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[351096.316303] ebtables/nat-PREROUTE IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[351096.316316] ebtables/filter-INPUT IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = ff:ff:ff:ff:ff:ff proto = 0x0806
[351096.316330] ebtables/nat-OUTPUT IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[351096.316333] ebtables/filter-OUTPUT IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[351096.316336] ebtables/nat-POSTROUTE IN= OUT=veth382c21d MAC source = 02:42:52:0e:03:e9 MAC dest = 02:42:ac:14:a8:02 proto = 0x0806
[351096.316347] ebtables/broute-BROUTING IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
[351096.316350] ebtables/nat-PREROUTE IN=veth382c21d OUT= MAC source = 02:42:ac:14:a8:02 MAC dest = 02:42:52:0e:03:e9 proto = 0x0800
[351096.316364] iptable/raw-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
[351096.316379] iptable/mangle-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
[351096.316388] iptable/nat-PREROUTE IN=docker0 OUT= PHYSIN=veth382c21d MAC=02:42:52:0e:03:e9:02:42:ac:14:a8:02:08:00 SRC=172.20.168.2 DST=172.20.168.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=12469 DF PROTO=ICMP TYPE=8 CODE=0 ID=1391 SEQ=1
```
Sign in with Wallet
Connect another wallet