卒研 - HackMD

# 卒研 > 方針: go-kawakami.oit.ac.jp のコピーを作り、それをベースに改良する。 ## 作るものと欲しい機能順次追加するかも 1. ライブ画像配信用に、別ポートで立ち上げるwebサーバ - 想定どおりの要求の場合はAccessLogを書かない 2. 既存のApacheを置き換える、WAF(ウェブアプリケーションファイアウォール)機能つきwebサーバ - URLをチェックし、怪しい要求が来たら対処する - rejectする - dropする - やたら大きなファイルを送りつけて仕返しする(笑) --- ### 1/14報告分 - ホワイトリスト自動作成UI - ドキュメントルート以下を全探索してホワイトリストに登録する - shellスクリプトでもなんでも可 - リスト外ファイル並びにクエリのやたら長いアクセスをブロック - 短時間で複数回のアクセスがあった場合は一定時間そのIPアドレスに対して何も返さない - Python ベンチマーク用クライアント作成 - HTTPのkeepaliveに対応したもの - できれば、Websocketに対応したものも合わせて - 結果次第でwebsocketかHTTPか決定 ### 1/22報告分 - 実験し、うまく行かなかった場合その理由を考察することが大事 - ファイルオープンでの処理負荷に着目し、その部分を改良するプログラムを制作した - しかし処理負荷は軽減できなかった - このことより、負荷はファイルオープンによって発生しているものではなく、HTTPのリクエスト処理に負荷がかかっていると思われる - Websocketを使用し、HTTPのリクエスト・レスポンスの手順を廃することで負荷の軽減を達成することができた(書ければ) ## 2週間目標 ### 10/29 or 31報告予定分 - [x] go-kawakami.oit.ac.jp のコピーサーバーを作る - [x] Node.jsで、最低限要求に応答できるwebサーバを作ってみる。 #### 追加でやったこと - [x] Node.jsで、特定のファイルを返すwebサーバを作る ### 11/25〜29報告予定分 - [x] エラーが発生しても落ちないためを行の追加 (furukawa, ikegami) - go-kawakamiのnoderun以下にあるファイルを参照 - [x] 返すファイルを変更しやすいようにプログラム書き換え (furukawa, ikegami) - 設定ファイルを作るもしくは、行数が短ければプログラムコードの上の方 - [ ] Webサーバ用ベンチマークテストツールを探してみて実行 (furukawa, ikegami) - GoogleのV8エンジンは素晴らしいらしい - [ ] Node.jsのガベージコレクションの頻度などについて調査 (ikegami, furukawa) - ガベージコレクション実行中はプログラムが長時間停止してしまうらしい ### 12/9〜13報告予定分 - [ ] Webサーバ用ベンチマークテストツールを探してみて実行(続き) - **a1秒間に500アクセス以上できるかどうか** - [ ] HTTPS対応(Let's Encrypt) - HTTPと混在するとchromeが弾くようになるため ## memo ``` 150.89.255.21 - - [03/Feb/2020:23:41:59 +0900] "GET /img/usage.jpg HTTP/1.1" 200 57627 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:41:59 +0900] "GET /img/usage_en.jpg HTTP/1.1" 200 55206 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:41:59 +0900] "GET /img/howitworks_en.jpg HTTP/1.1" 200 69505 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:41:59 +0900] "GET /img/a1.jpg HTTP/1.1" 200 89552 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "POST /msg/today-msg.txt HTTP/1.1" 200 340 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "GET /ramdsk/camera.jpg?page=1580740912233 HTTP/1.1" 200 43749 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "POST /msg/top-msg.txt HTTP/1.1" 200 281 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "POST /msg/today-msg.txt HTTP/1.1" 200 340 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "GET /ramdsk/hourly.json?_=1580740910958 HTTP/1.1" 404 491 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 150.89.255.21 - - [03/Feb/2020:23:42:00 +0900] "GET /ramdsk/daily.json?_=1580740910959 HTTP/1.1" 404 491 "http://150.89.255.61/index.html?ja" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" ``` ``` 196.52.84.16 - - [08/Jan/2020:06:09:50 +0900] "gamagaeru.com" ":443" "GET /admin/ HTTP/1.1" 301 67 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:09:52 +0900] "www.gamagaeru.com" ":443" "GET /admin/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:09:55 +0900] "gamagaeru.com" ":443" "GET /shop/admin/ HTTP/1.1" 301 72 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:09:56 +0900] "www.gamagaeru.com" ":443" "GET /shop/admin/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:09:57 +0900] "gamagaeru.com" ":443" "GET /store/admin/ HTTP/1.1" 301 73 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:09:58 +0900] "www.gamagaeru.com" ":443" "GET /store/admin/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:00 +0900] "gamagaeru.com" ":443" "GET /magento/admin/ HTTP/1.1" 301 75 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:01 +0900] "www.gamagaeru.com" ":443" "GET /magento/admin/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:03 +0900] "gamagaeru.com" ":443" "GET /magento2/admin/ HTTP/1.1" 301 76 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:04 +0900] "www.gamagaeru.com" ":443" "GET /magento2/admin/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:05 +0900] "gamagaeru.com" ":443" "GET /pub/errors/503.php HTTP/1.1" 301 79 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:06 +0900] "www.gamagaeru.com" ":443" "GET /pub/errors/503.php HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:07 +0900] "gamagaeru.com" ":443" "GET /shop/pub/errors/503.php HTTP/1.1" 301 84 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:08 +0900] "www.gamagaeru.com" ":443" "GET /shop/pub/errors/503.php HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:09 +0900] "gamagaeru.com" ":443" "GET /store/pub/errors/503.php HTTP/1.1" 301 85 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:10 +0900] "www.gamagaeru.com" ":443" "GET /store/pub/errors/503.php HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:11 +0900] "gamagaeru.com" ":443" "GET /magento/pub/errors/503.php HTTP/1.1" 301 87 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:12 +0900] "www.gamagaeru.com" ":443" "GET /magento/pub/errors/503.php HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:13 +0900] "gamagaeru.com" ":443" "GET /magento2/pub/errors/503.php HTTP/1.1" 301 88 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:14 +0900] "www.gamagaeru.com" ":443" "GET /magento2/pub/errors/503.php HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:15 +0900] "gamagaeru.com" ":443" "GET / HTTP/1.1" 301 61 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:16 +0900] "www.gamagaeru.com" ":443" "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:19 +0900] "gamagaeru.com" ":443" "GET /shop/ HTTP/1.1" 301 66 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:20 +0900] "www.gamagaeru.com" ":443" "GET /shop/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:21 +0900] "gamagaeru.com" ":443" "GET /store/ HTTP/1.1" 301 67 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:22 +0900] "www.gamagaeru.com" ":443" "GET /store/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:23 +0900] "gamagaeru.com" ":443" "GET /magento/ HTTP/1.1" 301 69 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:24 +0900] "www.gamagaeru.com" ":443" "GET /magento/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:25 +0900] "gamagaeru.com" ":443" "GET /magento2/ HTTP/1.1" 301 70 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 196.52.84.16 - - [08/Jan/2020:06:10:27 +0900] "www.gamagaeru.com" ":443" "GET /magento2/ HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" ``` ```jsonld= [ { addr: '::ffff:192.0.2.27', unixtime: 1580718039, times: 1 }, { addr: '2001:db8:dead:beef:573:196:1791:1605', unixtime: 1580717973, times: 7 } ] ``` ```jsonld= [ { addr: '::ffff:192.0.2.27', unixtime: 1580718150 }, { addr: '2001:db8:dead:beef:573:196:1791:1605', unixtime: 1580718165 } ] ``` ```javascript= // 一定間隔で画像データ取得 setInterval(function(){ //指定した間隔(ミリ秒)で無限ループ fs.readFile(config.path, function(err,data){ imgdata = data; // 取得した画像データをメモリで保持 }); },200); //ミリ秒 // リクエスト処理 https.createServer(options, function(req, res){ res.writeHead(200, {'Content-Type' : 'image/jpeg'}); res.end(imgdata); // 保持している画像データで応答 }).listen(config.port); ``` ``` process.on('uncaughtException', function(err) { //avoid exitting var etime = moment().format("YYYY-MM-DD HH:mm:ss "); fs.appendFile('./log/error.log', etime + '\r\n' + err + '\r\n'); }); ``` ``` ::ffff:192.0.2.21 - [2020/02/04:03:27:29(9H)] "GET /170417/Dist/images/chap0501.png HTTP/1.1" 404 - "https://www61.example.com:3443/rule.html?hoge=fuga" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" 2001:db8:dead:beef:573:196:1791:1605 user [2020/02/04:03:30:42(9H)] "GET /hoge.txt?p=fug HTTP/1.1" 200 7 "-" "curl/7.58.0" [2020/02/04:04:04:31(9H)] 2001:db8:dead:beef:573:196:1791:1605 Add [2020/02/04:04:04:33(9H)] ::ffff:192.0.2.21 Blocked ``` ## apachebench ### apache ``` This is ApacheBench, Version 2.3 <$Revision: 1807734 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking www61.gamagaeru.com (be patient) Server Software: Apache/2.4.41 Server Hostname: www61.gamagaeru.com Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256 TLS Server Name: www61.gamagaeru.com Document Path: /ramdsk/camera.jpg Document Length: 46158 bytes Concurrency Level: 100 Time taken for tests: 2529.370 seconds Complete requests: 100000 Failed requests: 50000 (Connect: 0, Receive: 0, Length: 50000, Exceptions: 0) Keep-Alive requests: 50000 Total transferred: 2324412450 bytes HTML transferred: 2309948000 bytes Requests per second: 39.54 [#/sec] (mean) Time per request: 2529.370 [ms] (mean) Time per request: 25.294 [ms] (mean, across all concurrent requests) Transfer rate: 897.43 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 23 184.0 8 7449 Processing: 0 2504 2503.9 5002 5074 Waiting: 0 1 2.7 1 65 Total: 0 2527 2533.2 5012 12460 Percentage of the requests served within a certain time (ms) 50% 5012 66% 5035 75% 5042 80% 5045 90% 5053 95% 5059 98% 5067 99% 5072 100% 12460 (longest request) ``` ### node ``` This is ApacheBench, Version 2.3 <$Revision: 1807734 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking www61.gamagaeru.com (be patient) Server Software: CosyLab Server Hostname: www61.gamagaeru.com Server Port: 3100 SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128 TLS Server Name: www61.gamagaeru.com Document Path: /ramdsk/camera.jpg Document Length: 46158 bytes Concurrency Level: 100 Time taken for tests: 478.951 seconds Complete requests: 100000 Failed requests: 0 Keep-Alive requests: 0 Total transferred: 4627600000 bytes HTML transferred: 4615800000 bytes Requests per second: 208.79 [#/sec] (mean) Time per request: 478.951 [ms] (mean) Time per request: 4.790 [ms] (mean, across all concurrent requests) Transfer rate: 9435.49 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 46 462 20.1 456 630 Processing: 2 17 2.6 17 79 Waiting: 1 9 1.8 9 55 Total: 67 479 20.8 474 656 Percentage of the requests served within a certain time (ms) 50% 474 66% 478 75% 483 80% 486 90% 503 95% 521 98% 539 99% 557 100% 656 (longest request) ``` ```javascript= IncomingMessage { (中略) httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: false, headers: { host: '[2001:2f8:1024:b3f0:82ee:73ff:feb9:ca64]:3000', connection: 'keep-alive', pragma: 'no-cache', 'cache-control': 'no-cache', dnt: '1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36', accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', referer: 'http://[2001:2f8:1024:b3f0:82ee:73ff:feb9:ca64]:3000/', 'accept-encoding': 'gzip, deflate', 'accept-language': 'ja,en-US;q=0.9,en;q=0.8' cookie: '_ga=GA1.1.475259271.1580449298; _gid=GA1.1.156268885.1580449298' }, (中略) _dumped: false } ```