# コンテナを増やしたいのに

## 問題文
Dockerコンテナでウェブサーバを動かしているVMに、新しくコンテナを追加することになった。
docker run -d -p 8080:80 --name web2 nginx を実行してコンテナを立ち上げたいのだが、コンテナに疎通が取れないという。
原因を究明して新しく追加されたコンテナに疎通が取れるようにしてほしい。

### 初期状態
踏み台サーバから curl 192.168.13.1 を実行すると正常に応答する
VMで docker start web2 を実行したあと踏み台サーバから curl 192.168.13.1:8080 を実行しても正常に応答しない
### 終了状態
VMを再起動しても以下が成り立つ
踏み台サーバから curl 192.168.13.1 を実行すると正常に応答する
VMで docker start web2 を実行するとVMの8080番ポートが web2 コンテナの80番ポートへフォワードされ、踏み台サーバから curl 192.168.13.1:8080 を実行すると正常に応答する

## 操作

user@vm:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
61371938da12        nginx               "nginx -g 'daemon of…"   36 minutes ago      Up 2 seconds        0.0.0.0:8080->80/tcp   web2
d732a7925959        nginx               "nginx -g 'daemon of…"   11 days ago         Up 4 seconds        0.0.0.0:80->80/tcp     web1
user@vm:~$ docker network inspect my-net
[
    {
        "Name": "my-net",
        "Id": "abbcfdda34c61334f181c9c1c25bc75a44802a3aacbfba7a060dfdb904df398f",
        "Created": "2020-02-18T03:57:13.396905758Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.213.213.0/24",
                    "Gateway": "10.213.213.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "61371938da123352b3a6d48a31da7938f4ba9fa8e1f6a573ed1bf326fb0f3a0c": {
                "Name": "web2",
                "EndpointID": "8e12bbd36c9008032136a16f5cd8385ad7b75843fc49d93e7414637db815ac65",
                "MacAddress": "02:42:0a:d5:d5:02",
                "IPv4Address": "10.213.213.2/24",
                "IPv6Address": ""
            },
            "d732a79259593e22664cf8be2acb59bad44a6ec4e84642e1af39898afe0eab36": {
                "Name": "web1",
                "EndpointID": "13bee79ab7b6a63a2c6604204ef2bd902c2e464b6446d81bd98f15d199317a8e",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "10.213.213.213/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
user@vm:~$ sudo iptables
[sudo] password for user:
iptables v1.6.1: no command specified
Try `iptables -h' or 'iptables --help' for more information.
user@vm:~$ sudo iptables -l
iptables v1.6.1: unknown option "-l"
Try `iptables -h' or 'iptables --help' for more information.
user@vm:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.2         tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.2         tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             vm
ACCEPT     tcp  --  anywhere             10.213.213.214       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.2         tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             10.213.213.2         tcp dpt:http
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
user@vm:~$ sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  all  --  10.213.213.0/24      anywhere
MASQUERADE  tcp  --  10.213.213.213       10.213.213.213       tcp dpt:http
MASQUERADE  tcp  --  10.213.213.2         10.213.213.2         tcp dpt:http

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:10.213.213.213:80
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http-alt to:10.213.213.2:80
user@vm:~$ curl 192.168.13.1:8080
curl: (7) Failed to connect to 192.168.13.1 port 8080: No route to host
user@vm:~$ curl localhost:8080
curl: (56) Recv failure: Connection reset by peer
user@vm:~$ curl localhost:8080
curl: (56) Recv failure: Connection reset by peer
user@vm:~$ curl 0.0.0.0:8080
curl: (56) Recv failure: Connection reset by peer
user@vm:~$ curl 127.0.0.1:8080
curl: (56) Recv failure: Connection reset by peer
user@vm:~$ curl localhost:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

inspectの差分を取るとweb1には`"MacAddress": "02:42:ac:11:00:02"`がある

docker run -d -p 80:80 --net="my-net" --name web1 --mac-address="02:42:ac:11:00:02" --ip=10.213.213.213  nginx
docker run -d -p 8080:80 --net="my-net" --name web2 --mac-address="02:42:ac:11:00:04" --ip=10.213.213.212  nginx

これで設定同じはずだけど...

user@vm:~$ cat iptables-set
# Generated by iptables-save v1.6.1 on Sun Mar  1 03:12:10 2020
*raw
:PREROUTING ACCEPT [11496:861925]
:OUTPUT ACCEPT [10548:1014715]
COMMIT
# Completed on Sun Mar  1 03:12:10 2020
# Generated by iptables-save v1.6.1 on Sun Mar  1 03:12:10 2020
*mangle
:PREROUTING ACCEPT [11534:864865]
:INPUT ACCEPT [11436:854745]
:FORWARD ACCEPT [98:10120]
:OUTPUT ACCEPT [10572:1017603]
:POSTROUTING ACCEPT [10648:1026403]
COMMIT
# Completed on Sun Mar  1 03:12:10 2020
# Generated by iptables-save v1.6.1 on Sun Mar  1 03:12:10 2020
*filter
:INPUT ACCEPT [1529:114018]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [1314:128013]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-abbcfdda34c6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-abbcfdda34c6 -j DOCKER
-A FORWARD -i br-abbcfdda34c6 ! -o br-abbcfdda34c6 -j ACCEPT
-A FORWARD -i br-abbcfdda34c6 -o br-abbcfdda34c6 -j ACCEPT
-A DOCKER -d 10.213.213.213/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 10.213.213.2/32 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 10.213.213.213/32 ! -i br-abbcfdda34c6 -o br-abbcfdda34c6 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 10.213.213.212/32 ! -i br-abbcfdda34c6 -o br-abbcfdda34c6 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-abbcfdda34c6 ! -o br-abbcfdda34c6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-abbcfdda34c6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -d 10.213.213.1/32 -j ACCEPT
-A DOCKER-USER -d 10.213.213.214/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-USER -d 10.213.213.213/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-USER -d 10.213.213.2/32 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER-USER -d 10.213.213.2/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -j DROP
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun Mar  1 03:12:10 2020
# Generated by iptables-save v1.6.1 on Sun Mar  1 03:12:10 2020
*nat
:PREROUTING ACCEPT [586:49224]
:INPUT ACCEPT [586:49224]
:OUTPUT ACCEPT [27:2031]
:POSTROUTING ACCEPT [27:2031]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.213.213.0/24 ! -o br-abbcfdda34c6 -j MASQUERADE
-A POSTROUTING -s 10.213.213.213/32 -d 10.213.213.213/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 10.213.213.212/32 -d 10.213.213.212/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-abbcfdda34c6 -j RETURN
-A DOCKER ! -i br-abbcfdda34c6 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.213.213.213:80
-A DOCKER ! -i br-abbcfdda34c6 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.213.213.212:80
COMMIT
# Completed on Sun Mar  1 03:12:10 2020

から

user@vm:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.212       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.212       tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             vm
ACCEPT     tcp  --  anywhere             10.213.213.214       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.213       tcp dpt:http
ACCEPT     tcp  --  anywhere             10.213.213.212       tcp dpt:http
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
user@vm:~$ vim iptables-set
user@vm:~$ sudo iptables --list -v -t nat
Chain PREROUTING (policy ACCEPT 303 packets, 25452 bytes)
 pkts bytes target     prot opt in     out     source               destination
  303 25452 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 303 packets, 25452 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 26 packets, 1987 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  any    any     anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 26 packets, 1987 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
    0     0 MASQUERADE  all  --  any    !br-abbcfdda34c6  10.213.213.0/24      anywhere
    0     0 MASQUERADE  tcp  --  any    any     10.213.213.213       10.213.213.213       tcp dpt:http
    0     0 MASQUERADE  tcp  --  any    any     10.213.213.212       10.213.213.212       tcp dpt:http

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere
    0     0 RETURN     all  --  br-abbcfdda34c6 any     anywhere             anywhere
    0     0 DNAT       tcp  --  !br-abbcfdda34c6 any     anywhere             anywhere             tcp dpt:http to:10.213.213.213:80
    0     0 DNAT       tcp  --  !br-abbcfdda34c6 any     anywhere             anywhere             tcp dpt:http-alt to:10.213.213.212:80

まだダメ

user@vm:~$ sudo tcpdump -i vethe35fb0b
疎通有り

user@vm:~$ sudo tcpdump -i veth677d9c9
疎通無し =>web2？

my-netの設定はほぼデフォルト（ip設定くらい？）

docker stop web1 web2
docker rm web1 web2
docker run -d --restart=always --net="host" --name web1 nginx
docker run -d -p 8080:80 --net="my-net" --name web2 --ip=10.213.213.213  nginx

user@vm:~$ sudo tcpdump -i br-abbcfdda34c6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-abbcfdda34c6, link-type EN10MB (Ethernet), capture size 262144 bytes
03:52:39.552720 IP vm > 10.213.213.213: ICMP echo request, id 10647, seq 1, length 64
03:52:39.552951 IP 10.213.213.213 > vm: ICMP echo reply, id 10647, seq 1, length 64
03:52:40.554574 IP vm > 10.213.213.213: ICMP echo request, id 10647, seq 2, length 64
03:52:40.554638 IP 10.213.213.213 > vm: ICMP echo reply, id 10647, seq 2, length 64
03:52:41.556235 IP vm > 10.213.213.213: ICMP echo request, id 10647, seq 3, length 64
03:52:41.556278 IP 10.213.213.213 > vm: ICMP echo reply, id 10647, seq 3, length 64
03:52:44.759143 ARP, Request who-has 10.213.213.213 tell vm, length 28
03:52:44.759112 ARP, Request who-has vm tell 10.213.213.213, length 28
03:52:44.759261 ARP, Reply vm is-at 02:42:e0:b3:f8:08 (oui Unknown), length 28
03:52:44.759269 ARP, Reply 10.213.213.213 is-at 02:42:ac:11:00:02 (oui Unknown), length 28
03:53:12.162769 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:53:13.175203 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:53:14.199168 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:53:15.223351 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:53:16.247089 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:53:17.271115 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:54:14.534629 IP 192.168.0.1.59952 > 10.213.213.213.http: Flags [S], seq 2867604461, win 64240, options [mss 1460,sackOK,TS val 746774280 ecr 0,nop,wscale 7], length 0
03:54:14.534827 IP 10.213.213.213.http > 192.168.0.1.59952: Flags [S.], seq 2727715288, ack 2867604462, win 65160, options [mss 1460,sackOK,TS val 1582138261 ecr 746774280,nop,wscale 6], length 0
03:54:14.535365 IP 192.168.0.1.59952 > 10.213.213.213.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 746774281 ecr 1582138261], length 0
03:54:14.535403 IP 192.168.0.1.59952 > 10.213.213.213.http: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 746774281 ecr 1582138261], length 76: HTTP: GET / HTTP/1.1
03:54:14.535687 IP 10.213.213.213.http > 192.168.0.1.59952: Flags [.], ack 77, win 1017, options [nop,nop,TS val 1582138262 ecr 746774281], length 0
03:54:14.537664 IP 10.213.213.213.http > 192.168.0.1.59952: Flags [P.], seq 1:239, ack 77, win 1017, options [nop,nop,TS val 1582138264 ecr 746774281], length 238: HTTP: HTTP/1.1 200 OK
03:54:14.538079 IP 10.213.213.213.http > 192.168.0.1.59952: Flags [P.], seq 239:851, ack 77, win 1017, options [nop,nop,TS val 1582138264 ecr 746774281], length 612: HTTP
03:54:19.578125 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:54:19.735234 ARP, Request who-has 10.213.213.213 tell vm, length 28
03:54:19.735175 ARP, Request who-has vm tell 10.213.213.213, length 28
03:54:19.735355 ARP, Reply vm is-at 02:42:e0:b3:f8:08 (oui Unknown), length 28
03:54:19.735370 ARP, Reply 10.213.213.213 is-at 02:42:ac:11:00:02 (oui Unknown), length 28
03:54:20.599133 ARP, Request who-has 10.213.213.212 tell vm, length 28
03:54:21.623170 ARP, Request who-has 10.213.213.212 tell vm, length 28
^C
30 packets captured
35 packets received by filter
5 packets dropped by kernel

user@vm:~$ sudo ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 4, policy: ACCEPT
-i eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 4, policy: ACCEPT
-o eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP

コンテナ再作成

docker stop web1 web2
docker rm web1 web2
docker run -d -p 80:80 --net="my-net" --name web1 --mac-address="02:42:ac:11:00:02" --ip=10.213.213.213  nginx
docker run -d -p 8080:80 --net="my-net" --name web2 --ip=10.213.213.212  nginx

## 解答

お世話になっております。:thonk_spin.ex-large.rotate.parrot:です。
問題「コンテナを増やしたいのに」の解答をお送りいたします。

この問題ではdockerネットワークのmy-net上のアドレス10.213.213.213のコンテナにのみ接続が出来る現象が確認されました。
そのため、web1をhostネットワークに移動し、web2をmy-net上のアドレス10.213.213.213に配置することで終了状態を満たしました。
ご確認の程よろしくお願いします。

## 設定の為に実行したコマンド

docker stop web1 web2
docker rm web1 web2
docker run -d --restart=always --net="host" --name web1 nginx
docker run -d -p 8080:80 --net="my-net" --name web2 --ip=10.213.213.213  nginx

以下のように正常に通信が出来ることを確認しました

user@problem-99-step:~$ curl 192.168.13.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
user@problem-99-step:~$ curl 192.168.13.1:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

## 10.213.213.213のコンテナにのみ接続が出来る現象の確認

アドレスが10.213.213.213のコンテナに対するcurlについてのtcpdump

user@vm:~$ sudo tcpdump -i br-abbcfdda34c6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-abbcfdda34c6, link-type EN10MB (Ethernet), capture size 262144 bytes
05:48:14.462456 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [S], seq 2756276898, win 64240, options [mss 1460,sackOK,TS val 753614343 ecr 0,nop,wscale 7], length 0
05:48:14.464250 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [S.], seq 2870867326, ack 2756276899, win 65160, options [mss 1460,sackOK,TS val 1588978357 ecr 753614343,nop,wscale 6], length 0
05:48:14.464897 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 753614346 ecr 1588978357], length 0
05:48:14.465620 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [P.], seq 1:82, ack 1, win 502, options [nop,nop,TS val 753614346 ecr 1588978357], length 81: HTTP: GET / HTTP/1.1
05:48:14.465661 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [.], ack 82, win 1017, options [nop,nop,TS val 1588978359 ecr 753614346], length 0
05:48:14.466159 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [P.], seq 1:239, ack 82, win 1017, options [nop,nop,TS val 1588978359 ecr 753614346], length 238: HTTP: HTTP/1.1 200 OK
05:48:14.466411 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [P.], seq 239:851, ack 82, win 1017, options [nop,nop,TS val 1588978360 ecr 753614346], length 612: HTTP
05:48:19.543207 ARP, Request who-has 10.213.213.213 tell vm, length 28
05:48:19.543171 ARP, Request who-has vm tell 10.213.213.213, length 28
05:48:19.543862 ARP, Reply vm is-at 02:42:e0:b3:f8:08 (oui Unknown), length 28
05:48:19.543877 ARP, Reply 10.213.213.213 is-at 02:42:0a:d5:d5:d5 (oui Unknown), length 28

アドレスが10.213.213.212のコンテナに対するcurlについてのtcpdump

user@vm:~$ sudo tcpdump -i br-abbcfdda34c6
05:48:28.031054 ARP, Request who-has 10.213.213.212 tell vm, length 28
05:48:29.047135 ARP, Request who-has 10.213.213.212 tell vm, length 28
05:48:30.071127 ARP, Request who-has 10.213.213.212 tell vm, length 28

## 10.213.213.213のみに繋がる原因

ebtablesの設定で、10.213.213.213と10.213.213.1（VM本体）のみに繋がる設定になっている為

user@vm:~$ sudo ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 4, policy: ACCEPT
-i eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 4, policy: ACCEPT
-o eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP

## 解答第二(85点)
追記：多分再起動試験通って無いですね（再起動しなくても良いものだと思い込んでいた）

お世話になっております。:thonk_spin.ex-large.rotate.parrot:です。
問題「コンテナを増やしたいのに」の解答をお送りいたします。

この問題ではdockerネットワークのmy-net上のアドレス10.213.213.213のコンテナにのみ接続が出来る現象が確認されました。
これはebtablesの設定が原因だった為、以下のように設定を変更しました。
この変更により終了状態を満たしました。
ご確認の程よろしくお願いします。

## 設定の為に実行したコマンド

### ebtablesの設定の変更

sudo ebtables -A INPUT -p ARP --arp-ip-src 10.213.213.212 --arp-ip-dst 10.213.213.1 -j ACCEPT
sudo ebtables -A INPUT -p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.212 -j ACCEPT
sudo ebtables -A OUTPUT -p ARP --arp-ip-src 10.213.213.212 --arp-ip-dst 10.213.213.1 -j ACCEPT
sudo ebtables -A OUTPUT -p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.212 -j ACCEPT
sudo ebtables -D INPUT -p ARP -j DROP
sudo ebtables -A INPUT -p ARP -j DROP
sudo ebtables -D OUTPUT -p ARP -j DROP
sudo ebtables -A OUTPUT -p ARP -j DROP

### web2の起動

docker stop web2
docker rm web2
docker run -d -p 8080:80 --net="my-net" --name web2 --ip=10.213.213.212  nginx

### 終了状態の確認
以下のように正常に通信が出来ることを確認しました

user@problem-99-step:~$ curl 192.168.13.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
user@problem-99-step:~$ curl 192.168.13.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

## 10.213.213.213のコンテナにのみ接続が出来る現象の確認

アドレスが10.213.213.213のコンテナに対するcurlについてのtcpdump

user@vm:~$ sudo tcpdump -i br-abbcfdda34c6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-abbcfdda34c6, link-type EN10MB (Ethernet), capture size 262144 bytes
05:48:14.462456 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [S], seq 2756276898, win 64240, options [mss 1460,sackOK,TS val 753614343 ecr 0,nop,wscale 7], length 0
05:48:14.464250 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [S.], seq 2870867326, ack 2756276899, win 65160, options [mss 1460,sackOK,TS val 1588978357 ecr 753614343,nop,wscale 6], length 0
05:48:14.464897 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 753614346 ecr 1588978357], length 0
05:48:14.465620 IP 192.168.0.1.58056 > 10.213.213.213.http: Flags [P.], seq 1:82, ack 1, win 502, options [nop,nop,TS val 753614346 ecr 1588978357], length 81: HTTP: GET / HTTP/1.1
05:48:14.465661 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [.], ack 82, win 1017, options [nop,nop,TS val 1588978359 ecr 753614346], length 0
05:48:14.466159 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [P.], seq 1:239, ack 82, win 1017, options [nop,nop,TS val 1588978359 ecr 753614346], length 238: HTTP: HTTP/1.1 200 OK
05:48:14.466411 IP 10.213.213.213.http > 192.168.0.1.58056: Flags [P.], seq 239:851, ack 82, win 1017, options [nop,nop,TS val 1588978360 ecr 753614346], length 612: HTTP
05:48:19.543207 ARP, Request who-has 10.213.213.213 tell vm, length 28
05:48:19.543171 ARP, Request who-has vm tell 10.213.213.213, length 28
05:48:19.543862 ARP, Reply vm is-at 02:42:e0:b3:f8:08 (oui Unknown), length 28
05:48:19.543877 ARP, Reply 10.213.213.213 is-at 02:42:0a:d5:d5:d5 (oui Unknown), length 28

アドレスが10.213.213.212のコンテナに対するcurlについてのtcpdump

user@vm:~$ sudo tcpdump -i br-abbcfdda34c6
05:48:28.031054 ARP, Request who-has 10.213.213.212 tell vm, length 28
05:48:29.047135 ARP, Request who-has 10.213.213.212 tell vm, length 28
05:48:30.071127 ARP, Request who-has 10.213.213.212 tell vm, length 28

## ebtablesの設定の確認

ebtablesの設定で、10.213.213.213と10.213.213.1（VM本体）のみに繋がる設定になっていた

user@vm:~$ sudo ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 4, policy: ACCEPT
-i eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 4, policy: ACCEPT
-o eth0 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.213 --arp-ip-dst 10.213.213.1 -j ACCEPT
-p ARP --arp-ip-src 10.213.213.1 --arp-ip-dst 10.213.213.213 -j ACCEPT
-p ARP -j DROP