Archipelago GM - API Limiter Adjustment

# Archipelago GM - API Limiter Adjustment ## Meta Information - Report Date: `June 21, 2019` - Reported By: `Vinsensius Angelo - Tech` - URL: [http://bit.ly/2Rs4LD1](http://bit.ly/2Rs4LD1) ## Proteus Callback Problem - New Relic Report ([reference](https://rpm.newrelic.com/accounts/908709/applications/91783725/filterable_errors#/show/1a67efd6-9356-11e9-92fe-0242ac110009_0_4378/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart&filters=%5B%7B%22key%22%3A%22error.class%22%2C%22value%22%3A%22werkzeug.exceptions%3ATooManyRequests%22%2C%22like%22%3Afalse%7D%5D)) - End-point: `https://gm.yapulsa.com/api/v1/transaction/callback/proteus_out` - HTTP Method: `POST` - Rate Limit (per Remote IP Address): `100 / minute; 10 / second` - Time Interval: `Jun 20, 6:45 PM` until `Jun 20, 7:22 PM` - Error Count: `239` - Salmon Reporting - End-point: `https://gm.yapulsa.com/api/v1/transaction/callback/proteus_out` - HTTP Method: `POST` - Client ([reference](http://salmon.sepulsa.id/app/kibana#/doc/filebeat-*/filebeat-2019.06.20/nginx-access?id=AWt2IxaUPEVRnTOyrl1d&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5m,mode:quick,to:now)))) - Remote IP Address: `3.0.135.232` - Agent: `axios/0.18.0` - Request Behaviour - Weekday \ Time Interval Sample: `2019-06-19 00:00:00` - `2019-06-19 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-18T17:00:00.000Z',mode:absolute,to:'2019-06-19T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Hit Total: `3,956` - Peak Hours: - `2019-06-19 10:30:00` until `2019-06-19 11:30:00` (min. `93 Hits`; max. `137 Hits`) - `2019-06-19 18:00:00` until `2019-06-19 21:00:00` (min. `185 Hits`; max. `234 Hits`) - Request per minute: - Max. `3 Hits` - `2019-06-19 19:29:00` until `2019-06-19 19:30:00` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-19T12:29:00.000Z',mode:absolute,to:'2019-06-19T12:30:00.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Weekend \ Time Interval Sample: `2019-06-16 00:00:00` - `2019-06-16 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-15T17:00:00.000Z',mode:absolute,to:'2019-06-16T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Hit Total: `3,795` - Peak Hours: - `2019-06-16 10:30:00` until `2019-06-16 11:30:00` (min. `91 Hits`; max. `116 Hits`) - `2019-06-16 18:00:00` until `2019-06-16 21:00:00` (min. `144 Hits`; max. `210 Hits`) - Request per minute: - Max. `2 Hits` - `2019-06-19 19:36:37` until `2019-06-19 19:36:38` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-16T12:36:30.000Z',mode:absolute,to:'2019-06-16T12:37:00.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) ## BPA Kraken v1 Callback - Salmon Reporting - End-point: `https://callback.yapulsa.com/api/v1/transaction/callback/kraken` - HTTP Method: `POST` - Client ([reference](http://salmon.sepulsa.id/app/kibana#/doc/filebeat-*/filebeat-2019.06.08/nginx-access?id=AWs4B0ZdPEVRnTOygFg2&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5m,mode:quick,to:now)))) - Remote IP Address: `52.74.47.66` - Agent: `Kraken` - Request Behaviour - Weekday \ Time Interval Sample: `2019-06-20 00:00:00` - `2019-06-20 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-19T17:00:00.000Z',mode:absolute,to:'2019-06-20T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Hit Total: `23,662` - Peak Hours: - `2019-06-20 09:00:00` until `2019-06-20 11:30:00` (min. `651 Hits`; max. `804 Hits`) - `2019-06-20 17:00:00` until `2019-06-20 20:30:00` (min. `858 Hits`; max. `1,396 Hits`) - Request per minute: - Max. `59 Hits` - `2019-06-20 19:32:00` until `2019-06-20 19:32:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-20T12:32:00.000Z',mode:absolute,to:'2019-06-20T12:32:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Weekend \ Time Interval Sample: `2019-06-16 00:00:00` - `2019-06-16 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-15T17:00:00.000Z',mode:absolute,to:'2019-06-16T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) - Hit Total: `20,470` - Peak Hours: - `2019-06-16 09:00:00` until `2019-06-16 11:30:00` (min. `560 Hits`; max. `684 Hits` - `2019-06-16 17:00:00` until `2019-06-16 20:30:00` (min. `784 Hits`; max. `1,079 Hits`) - Request per minute: - Max. `39 Hits` - `2019-06-16 19:30:00` until `2019-06-16 19:30:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-16T12:30:00.000Z',mode:absolute,to:'2019-06-16T12:30:30.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram)) ## Current Situation's Pros and Cons ### Pros - API Max Throttle help stability of GM system: - Avoid server resources usage violation. - Reduce security breach / brute-force attack (based on fraud/spammer report last 2 month). - Help to limit AlphaPay Transaction to avoid double request due to technical issue on client-apps (3 / minute; 1 second). - Callback from Third-Party using public API URL since 2017 (there is no security-checking or IP Address whitelist about client source for each HTTP request). ### Cons - Callback from `Canopus V2` and `Serpul - Proteus Out` can't be moved to `https://callback.yapulsa.com` due to some reason as below: - Beta / Sanity for related callback. - There are some error-log report due to instable/inconsistence of releated feature. - Portal Client App (PAIMIN) - Heavily use API endpoints to load data from GM (planned to be shutdown on July or August 2019) - Vulnerable to any hacker activity that affected to security issues on few API end-points (reported for last 2 months) ## Temporary Solutions - GM End-point: `/api/v1/transaction/callback/{push_type}` - Hosts: - `https://gm.yapulsa.com` - `https://gmapi.yapulsa.com` - `https://api.yapulsa.com` - `https://callback.yapulsa.com` - HTTP Method: `POST` - Rate Limit (per Remote IP Address): - Last configured: `100 / minute; 10 / second` - Current situation: `6000 / minute; 100 / second` - GM Another End-point due to Portal Client App (PAIMIN) access: - New Relic Report ([reference](https://rpm.newrelic.com/accounts/908709/applications/91783725/filterable_errors#/show/313df923-9355-11e9-92fe-0242ac110009_4379_8962/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart&filters=%5B%7B%22key%22%3A%22error.class%22%2C%22value%22%3A%22werkzeug.exceptions%3ATooManyRequests%22%2C%22like%22%3Afalse%7D%5D)) - Hosts: `https://gm.yapulsa.com` - HTTP Method: `GET|POST|PUT|PATCH|DELETE|OPTIONS` - Rate Limit (per `Remote IP Address` or per `user ID`): - Last configured: `60 / minute` - Current situation: `1000 / minute` - After Portal app shutdown: `200 / minute` or `300 / minute` - Each Callback from third-party need to be moved from `https://gm.yapulsa.com` into `https://callback.yapulsa.com`. - Risks: - Beta / Sanity testing for third-party integrations cannot be properly tested.