---
# System prepended metadata

title: isucon10 final 環境準備

---

# isucon10 final 環境準備　

https://github.com/matsuu/aws-isucon
https://github.com/matsuu/vagrant-isucon/tree/master/isucon10-final-standalone

## server
aws ec2 t2.large
1: `13.231.182.10` アプリケーション (xsuportal)
2: `52.198.229.213` 実ベンチマーカー

usename: `ubuntu`

全員分のsshキーを追加済み

## 改変まとめ
- `benchmarker/main.go`
```diff=88
- agent.DefaultTLSConfig.InsecureSkipVerify = false
+ agent.DefaultTLSConfig.InsecureSkipVerify = true
```
- `benchmarker/scenario/benchmarker.go`
```diff=276
- tlsConfig = grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, ""))
+ // tlsConfig = grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, ""))
```
- `/etc/hosts` に追記
```
172.31.20.0 app1.isucon.unipota.me
172.31.17.1 app2.isucon.unipota.me
172.31.45.139 bench.isucon.unipota.me
```
- `/home/isucon/env` @ app.t.isucon.dev
```diff
MYSQL_HOSTNAME=localhost
MYSQL_PORT=3306
MYSQL_USER=isucon
MYSQL_PASS=isucon
MYSQL_DATABASE=xsuportal
+BENCHMARK_SERVER_HOST=app.t.isucon.dev
+BENCHMARK_SERVER_PORT=50051
```

## setup
- セットアップ手順
    - https://github.com/matsuu/vagrant-isucon/tree/master/isucon10-final-standalone
    - https://github.com/isucon/isucon10-final
- `sudo -i -u isucon`
- サービスの起動
    ```
    sudo systemctl start envoy
    sudo systemctl start xsuportal-api-golang.service
    sudo systemctl start xsuportal-web-golang.service
    ```
- benchmarker
    - 別サーバーからベンチマーカーを動かそうとすると、
    ```
    x509: cannot validate certificate for 172.31.33.9 because it doesn't contain any IP SANs
    ```
    のようなエラーが出るので修正する
    - 自己署名証明書のせい？ 
    - `benchmarker/main.go:88`
    ```diff
    - agent.DefaultTLSConfig.InsecureSkipVerify = false
    + agent.DefaultTLSConfig.InsecureSkipVerify = true
    ```
- `/etc/hosts`
    ```
    13.231.182.10 app.t.isucon.dev   
    52.198.229.213 bench.t.isucon.dev
    ```
- ec2イメージ作成時証明書とってるところ
    - https://github.com/matsuu/aws-isucon/blob/main/isucon10-final/provision.sh#L21
    - [How do I use SANs with openSSL instead of common name? - Stack Overflow](https://stackoverflow.com/questions/64814173/how-do-i-use-sans-with-openssl-instead-of-common-name)
      - `openssl` コマンドに `-addext 'subjectAltName = DNS:app.t.isucon.dev' -addext 'subjectAltName = DNS:bench.t.isucon.dev'` を渡せばできそう？
      - ```
        openssl req -x509 -subj "/CN=*.t.isucon.dev" \
         -addext "subjectAltName = DNS:app.t.isucon.dev, DNS:bench.t.isucon.dev" \
         -sha256 -nodes -days 3650 -newkey rsa:2048 \
         -keyout secrets/san-key.pem -out secrets/san-cert.pem
        ```
          - `req: Use -help for summary.` と言われるのでコマンドの使い方が何か間違ってそう
      - [Know about SAN Certificate and How to Create With OpenSSL](https://geekflare.com/san-ssl-certificate/)
        - > SAN stands for “Subject Alternative Names” and this helps you to have a single certificate for multiple CN (Common Name).
- systemd-runに実行時の環境変数を渡してあげる
    - -E GODEBUG=x509ignoreCN=0
    ```
    sudo systemd-run \
      --working-directory=/home/isucon/benchmarker \
      --pipe \
      --wait \
      --collect \
      --uid=$(id -u)\
      --gid=$(id -g) \
      --slice=benchmarker.slice \
      --service-type=oneshot \
      -E GODEBUG=x509ignoreCN=0 \
      -p AmbientCapabilities=CAP_NET_BIND_SERVICE \
      -p CapabilityBoundingSet=CAP_NET_BIND_SERVICE \
      -p LimitNOFILE=2000000 \
      -p TimeoutStartSec=110s \
        ~isucon/benchmarker/bin/benchmarker \
        -exit-status \
        -tls \
        -target app.t.isucon.dev:443 \
        -host-advertise bench.t.isucon.dev \
        -push-service-port 1001 \
        -tls-cert /etc/ssl/private/tls-cert.pem \
        -tls-key /etc/ssl/private/tls-key.pem
    ```
    

```
07:03:41.881460 ISUCON10 benchmarker e858b2588a199f9c7407baacf48b53126b8aeed6+dirty
07:03:41.881569 ===> PREPARE
07:03:41.975203 ERR: prepare: critical: http: Post "https://app.t.isucon.dev:443/initialize": x509: certificate signed by unknown authority
07:03:41.975267 ===> SCORE
Count:

(0 * 1.0) + 0 - 0(err: 0, timeout: 0)
Pass: false / score: 0 (0 - 0)
Fail reason: Critical error
```

- `benchmarker/main.go:88`
    ```diff
    - agent.DefaultTLSConfig.InsecureSkipVerify = false
    + agent.DefaultTLSConfig.InsecureSkipVerify = true
    ```
```
07:05:33.772142 ISUCON10 benchmarker e858b2588a199f9c7407baacf48b53126b8aeed6+dirty
07:05:33.772247 ===> PREPARE
07:05:33.971151 ERR: prepare: critical: http-server-error: 不正な HTTP ステータスコード: 503 (POST: /initialize)
07:05:33.971226 ===> SCORE
Count:
```

```shell
$ journalctl -u xsuportal-web-golang.service | grep /initialize
May 08 05:55:10 ip-172-31-42-155 .x[1894]: {"time":"2021-05-08T05:55:10.3458802Z","id":"","remote_ip":"127.0.0.1","host":"localhost:9292","method":"POST","uri":"/initialize","user_agent":"benchmarker-initializer","status":200,"error":"","latency":787534298,"latency_human":"787.534298ms","bytes_in":34,"bytes_out":21}
May 08 05:55:18 ip-172-31-42-155 .x[1894]: {"time":"2021-05-08T05:55:18.878274989Z","id":"","remote_ip":"127.0.0.1","host":"localhost:9292","method":"POST","uri":"/initialize","user_agent":"benchmarker-initializer","status":200,"error":"","latency":357342096,"latency_human":"357.342096ms","bytes_in":34,"bytes_out":21}
```

- Envoy
    - config: `/etc/envoy/config.yaml`
    - log: `/var/log/envoy/access.log`

```shell
isucon@ip-172-31-39-27:~$ curl --insecure https://13.231.182.10
<!doctype html><html lang="ja" class="has-background-grey-lighter"><head><title>XSUCON Portal</title><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="xsu:api-base-url" content="/"/><link href="/packs/vendor.css" rel="stylesheet"></head><body><div id="app"></div><script src="/packs/vendor.js"></script><script src="/packs/audience.js"></script></body></html>
isucon@ip-172-31-39-27:~$ curl --insecure https://app.t.isucon.dev/
upstream connect error or disconnect/reset before headers. reset reason: connection failureisucon
```

```=
curl -vk https://app.t.isucon.dev
*   Trying ::1:443...
* TCP_NODELAY set
* Connected to app.t.isucon.dev (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.t.isucon.dev
*  start date: May  1 03:54:40 2021 GMT
*  expire date: Apr 29 03:54:40 2031 GMT
*  issuer: CN=*.t.isucon.dev
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5643cf49d820)
> GET / HTTP/2
> Host: app.t.isucon.dev
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 503
< content-length: 91
< content-type: text/plain
< date: Sat, 08 May 2021 07:39:23 GMT
< server: envoy
<
* Connection #0 to host app.t.isucon.dev left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure
```

`/etc/hosts` の記載を間違えていた :kan:

- AWS
 ![](https://i.imgur.com/IKYfIL5.png)


webapp/golang/cmd/xsuportal/main.go
```go=188
    host := util.GetEnv("BENCHMARK_SERVER_HOST", "localhost")
    port, _ := strconv.Atoi(util.GetEnv("BENCHMARK_SERVER_PORT", "50051"))
```
これはxsucon側のbenchmarker? ←はい


- サーバー１の/home/isucon/envにBENCHMARK_SERVER_HOSTを設定
    - BENCHMARK_SERVER_HOST=app.t.isucon.dev
    - `sudo systemctl daemon-reload`
    - `sudo systemctl restart xsuportal-web-golang.service`

```
08:38:28.014903 ISUCON10 benchmarker e858b2588a199f9c7407baacf48b53126b8aeed6+dirty
08:38:28.015092 ===> PREPARE
08:38:37.792012 Language: go
08:38:37.792034 HTTP: https://app.t.isucon.dev:443/(tls=true)
08:38:37.792040 gRPC: localhost:50051
08:38:37.792120 ===> LOAD
08:38:37.792148 LOAD INFO
  Registration Open at: 2021-05-08 08:38:38 +0000 UTC
  Contest Start at: 2021-05-08 08:38:48 +0000 UTC
  Contest Freeze at: 2021-05-08 08:39:28 +0000 UTC
  Contest Ends at: 2021-05-08 08:39:38 +0000 UTC
08:38:48.134783 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.147122 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.165834 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.192635 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.204953 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.218591 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.233141 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.273034 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.279541 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
08:38:48.294305 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"
```

```
mysql> select * from benchmark_jobs;
+----+---------+--------+----------------------+--------+-----------+-----------------+--------+--------+------------+-------------+----------------------------+----------------------------+
| id | team_id | status | target_hostname      | handle | score_raw | score_deduction | reason | passed | started_at | finished_at | created_at                 | updated_at                 |
+----+---------+--------+----------------------+--------+-----------+-----------------+--------+--------+------------+-------------+----------------------------+----------------------------+
|  1 |      10 |      0 | xsu-contestant-00010 | NULL   |      NULL |            NULL | NULL   |   NULL | NULL       | NULL        | 2021-05-08 08:45:37.050763 | 2021-05-08 08:45:37.050763 |
|  2 |       2 |      0 | xsu-contestant-00002 | NULL   |      NULL |            NULL | NULL   |   NULL | NULL       | NULL        | 2021-05-08 08:45:37.050891 | 2021-05-08 08:45:37.050891 |
```

エラー内容
```
08:45:48.073575 ERR: load: invalid-response: 期待より古い内容のリーダーボードが返却されています(GET /api/contestant/dashboard)
08:45:48.074067 OLDER LEADERBOARD: 
  2021-05-08 08:45:48.047563828 +0000 UTC requested at
  2021-05-08 08:45:37.000182 +0000 UTC latest finish
  2021-05-08 08:45:35.000182 +0000 UTC allowed cache time
  1970-01-01 00:00:00 +0000 UTC leadeboard max time
  2021-05-08 08:46:17 +0000 UTC frozen time
  2021-05-08 08:45:48.07405842 +0000 UTC now time
```

↑のエラーを出す実ベンチマーカーのコード (benchmarker/scenario/verify.go)
```go=315
  AdminLogger.Printf("OLDER LEADERBOARD: \n  %s requested at\n  %s latest finish\n  %s allowed cache time\n  %s leadeboard max time\n  %s frozen time\n  %s now time\n", requestedAt, sLatestMarkedAt, allowedMaxTime.Add(-cacheTime), maxMarkedAt, contest.ContestFreezesAt, time.Now().UTC())
  if allowCache {
      return errorInvalidResponse("期待より古い内容のリーダーボードが返却されています(GET /api/audience/dashboard)")
  } else {
      return errorInvalidResponse("期待より古い内容のリーダーボードが返却されています(GET /api/contestant/dashboard)")
  }
```

webapp/golang/cmd/xsuportal/main.go
```golang
func (*ContestantService) EnqueueBenchmarkJob(e echo.Context) error {
// (略)
	_, err = tx.Exec(
		"INSERT INTO `benchmark_jobs` (`team_id`, `target_hostname`, `status`, `updated_at`, `created_at`) VALUES (?, ?, ?, NOW(6), NOW(6))",
		team.ID,
		req.TargetHostname,
		int(resourcespb.BenchmarkJob_PENDING),
	)
```

https://github.com/fullstorydev/grpcurl

```shell
$ go/bin/grpcurl app.t.isucon.dev:50051 xsuportal.proto.services.bench.BenchmarkQueue/ReceiveBenchmarkJob
Failed to dial target host "app.t.isucon.dev:50051": tls: first record does not look like a TLS handshake
$ go/bin/grpcurl -plaintext app.t.isucon.dev:50051 xsuportal.proto.services.bench.BenchmarkQueue/ReceiveBenchmarkJob
Error invoking method "xsuportal.proto.services.bench.BenchmarkQueue/ReceiveBenchmarkJob": failed to query for service descriptor "xsuportal.proto.services.bench.BenchmarkQueue": server does not support the reflection API
```

https://github.com/fullstorydev/grpcurl#proto-source-files
> To use grpcurl on servers that do not support reflection, you can use .proto source files.

↑にあるように `.proto` を指定すると server2 から grpcurl でアクセスできる

```shell
$ go/bin/grpcurl -plaintext -proto proto/xsuportal/services/bench/receiving.proto app.t.isucon.dev:50051 xsuportal.proto.services.bench.BenchmarkQueue/ReceiveBenchmarkJob
{
  "jobHandle": {
    "jobId": "1",
    "handle": "CiLAofVx0z13pdtnaH7G8A==",
    "targetHostname": "xsu-contestant-00008",
    "contestStartedAt": "2021-05-08T13:59:49Z",
    "jobCreatedAt": "2021-05-08T13:59:49.054096Z"
  }
}
```

[isucon10-final/webapp/tools at master · isucon/isucon10-final](https://github.com/isucon/isucon10-final/tree/master/webapp/tools)

/etc/hosts
```
#public
#13.231.182.10 app.t.isucon.dev
#52.198.229.213 bench.t.isucon.dev

#private
172.31.42.155 app.t.isucon.dev
172.31.39.27 bench.t.isucon.dev
```

`gRPC: app.t.isucon.dev:443` にしてみる

```
13:30:28.134857 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"
```

証明書を元のやつに戻してみる
```
13:42:24.377349 ERR: load: benchmarker-receive: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0"
```

ベンチマーカー実行時の `-tls` オプションを消す
```
13:44:50.817450 ===> PREPARE
13:44:50.820464 ERR: prepare: critical: http: Post "http://app.t.isucon.dev:443/initialize": EOF
```

`-target 13.231.182.10` にする
```
13:46:12.458471 ===> PREPARE
13:46:12.461016 ERR: prepare: critical: http-server-error: 不正な HTTP ステータスコード: 301 (POST: /initialize)
```

実ベンチマーカー側の gRPC client の TLS を無効にしたら正の点数が出た :tada:

https://github.com/isucon/isucon10-final/blob/master/benchmarker/scenario/benchmarker.go#L276
```diff
- tlsConfig = grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, ""))
+ // tlsConfig = grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, ""))
```

- 実ベンチマーカーに渡す `-tls` を消すだけではだめだった
  - `web` (HTTP) はTLS有効、 `api` (gRPC) はTLS無効にしないといけなかった


[grpc-go/grpc-auth-support.md at master · grpc/grpc-go](https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-auth-support.md#enabling-tls-on-a-grpc-server)
> Enabling TLS on a gRPC server
```go
creds, err := credentials.NewServerTLSFromFile(certFile, keyFile)
server := grpc.NewServer(grpc.Creds(creds))
```

~/webapp/golang/cmd/benchmark_server/main.go を書き換えて TLS を有効にすれば良いのでは？

```golang=284
        creds, err := credentials.NewServerTLSFromFile("/home/isucon/secrets/san-cert.pem", "/home/isucon/secrets/san-key.pem")
        if err != nil {
                log.Fatalf("Failed to generate credentials %v", err)
        }
        server := grpc.NewServer(grpc.Creds(creds))
```

これで実ベンチマーカーを走らせたら `"transport: authentication handshake failed: x509: certificate signed by unknown authority"` と怒られてダメだった :sad_parrot:

むしろ本番でどうしてこれで動いていたのかが気になる…………

## メモ
oisu-

journalctlの使い方
https://qiita.com/aosho235/items/9fbff75e9cccf351345c

[そもそも RPC ってなんだ - Qiita](https://qiita.com/il-m-yamagishi/items/8709de06be33e7051fd2)

[内部実装から理解するgRPC - Qiita](https://qiita.com/immrshc/items/075fcbf22b5f5e48f9ec)