---
# System prepended metadata

title: Archipelago GM - API Limiter Adjustment

---

# Archipelago GM - API Limiter Adjustment

## Meta Information
- Report Date: `June 21, 2019`
- Reported By: `Vinsensius Angelo - Tech`
- URL: [http://bit.ly/2Rs4LD1](http://bit.ly/2Rs4LD1)

## Proteus Callback Problem
- New Relic Report ([reference](https://rpm.newrelic.com/accounts/908709/applications/91783725/filterable_errors#/show/1a67efd6-9356-11e9-92fe-0242ac110009_0_4378/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart&filters=%5B%7B%22key%22%3A%22error.class%22%2C%22value%22%3A%22werkzeug.exceptions%3ATooManyRequests%22%2C%22like%22%3Afalse%7D%5D))
	- End-point: `https://gm.yapulsa.com/api/v1/transaction/callback/proteus_out` 
	- HTTP Method: `POST`
	- Rate Limit (per Remote IP Address): `100 / minute; 10 / second`
	- Time Interval: `Jun 20, 6:45 PM` until `Jun 20, 7:22 PM`
	- Error Count: `239`

- Salmon Reporting
	- End-point: `https://gm.yapulsa.com/api/v1/transaction/callback/proteus_out` 
	- HTTP Method: `POST`
	- Client ([reference](http://salmon.sepulsa.id/app/kibana#/doc/filebeat-*/filebeat-2019.06.20/nginx-access?id=AWt2IxaUPEVRnTOyrl1d&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5m,mode:quick,to:now))))
		- Remote IP Address: `3.0.135.232` 
		- Agent: `axios/0.18.0`
	- Request Behaviour
		- Weekday \ Time Interval Sample: `2019-06-19 00:00:00` - `2019-06-19 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-18T17:00:00.000Z',mode:absolute,to:'2019-06-19T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
			- Hit Total: `3,956`
			- Peak Hours:
				- `2019-06-19 10:30:00` until `2019-06-19 11:30:00` (min. `93 Hits`; max. `137 Hits`)
				- `2019-06-19 18:00:00` until `2019-06-19 21:00:00` (min. `185 Hits`; max. `234 Hits`)
 
			- Request per minute:
				- Max. `3 Hits` - `2019-06-19 19:29:00` until `2019-06-19 19:30:00` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-19T12:29:00.000Z',mode:absolute,to:'2019-06-19T12:30:00.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
		- Weekend \ Time Interval Sample: `2019-06-16 00:00:00` - `2019-06-16 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-15T17:00:00.000Z',mode:absolute,to:'2019-06-16T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
			- Hit Total: `3,795`
			- Peak Hours:
				- `2019-06-16 10:30:00` until `2019-06-16 11:30:00` (min. `91 Hits`; max. `116 Hits`)
				- `2019-06-16 18:00:00` until `2019-06-16 21:00:00` (min. `144 Hits`; max. `210 Hits`)
 
			- Request per minute:
				- Max. `2 Hits` - `2019-06-19 19:36:37` until `2019-06-19 19:36:38` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-16T12:36:30.000Z',mode:absolute,to:'2019-06-16T12:37:00.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:gm.yapulsa.com),query:(match:(beat.name:(query:gm.yapulsa.com,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:4871','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:args.status_code,negate:!f,value:'422'),query:(match:(args.status_code:(query:422,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fproteus_out,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*proteus_out')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))

				
## BPA Kraken v1 Callback
- Salmon Reporting
	- End-point: `https://callback.yapulsa.com/api/v1/transaction/callback/kraken` 
	- HTTP Method: `POST`
	- Client ([reference](http://salmon.sepulsa.id/app/kibana#/doc/filebeat-*/filebeat-2019.06.08/nginx-access?id=AWs4B0ZdPEVRnTOygFg2&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-5m,mode:quick,to:now))))
		- Remote IP Address: `52.74.47.66` 
		- Agent: `Kraken`
	- Request Behaviour
		- Weekday \ Time Interval Sample: `2019-06-20 00:00:00` - `2019-06-20 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-19T17:00:00.000Z',mode:absolute,to:'2019-06-20T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
			- Hit Total: `23,662`
			- Peak Hours:
				- `2019-06-20 09:00:00` until `2019-06-20 11:30:00` (min. `651 Hits`; max. `804 Hits`)
				- `2019-06-20 17:00:00` until `2019-06-20 20:30:00` (min. `858 Hits`; max. `1,396 Hits`)
 
			- Request per minute:
				- Max. `59 Hits` - `2019-06-20 19:32:00` until `2019-06-20 19:32:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-20T12:32:00.000Z',mode:absolute,to:'2019-06-20T12:32:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
		- Weekend \ Time Interval Sample: `2019-06-16 00:00:00` - `2019-06-16 23:59:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-15T17:00:00.000Z',mode:absolute,to:'2019-06-16T16:59:59.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))
			- Hit Total: `20,470`
			- Peak Hours:
				- `2019-06-16 09:00:00` until `2019-06-16 11:30:00` (min. `560 Hits`; max. `684 Hits`
				- `2019-06-16 17:00:00` until `2019-06-16 20:30:00` (min. `784 Hits`; max. `1,079 Hits`)
 
			- Request per minute:
				- Max. `39 Hits` - `2019-06-16 19:30:00` until `2019-06-16 19:30:59` ([reference](http://salmon.sepulsa.id/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-06-16T12:30:00.000Z',mode:absolute,to:'2019-06-16T12:30:30.000Z'))&_a=(columns:!(_source),filters:!(('$$hashKey':'object:6204','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:beat.name,negate:!f,value:yapulsav2-api3),query:(match:(beat.name:(query:yapulsav2-api3,type:phrase)))),('$$hashKey':'object:7372','$state':(store:appState),meta:(alias:!n,disabled:!t,index:'filebeat-*',key:type,negate:!f,value:yapulsav2-local),query:(match:(type:(query:yapulsav2-local,type:phrase)))),('$$hashKey':'object:1544','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:request,negate:!f,value:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken),query:(match:(request:(query:%2Fapi%2Fv1%2Ftransaction%2Fcallback%2Fkraken,type:phrase)))),('$$hashKey':'object:2202','$state':(store:appState),meta:(alias:!n,disabled:!f,index:'filebeat-*',key:type,negate:!f,value:nginx-access),query:(match:(type:(query:nginx-access,type:phrase))))),index:'filebeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))),vis:(aggs:!((params:(field:levelname,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=filebeat-*&type=histogram))	


## Current Situation's Pros and Cons 
### Pros
- API Max Throttle help stability of GM system:
	- Avoid server resources usage violation.
	- Reduce security breach / brute-force attack (based on fraud/spammer report last 2 month).
	- Help to limit AlphaPay Transaction to avoid double request due to technical issue on client-apps (3 / minute; 1 second).
- Callback from Third-Party using public API URL since 2017 (there is no security-checking or IP Address whitelist about client source for each HTTP request).

### Cons
- Callback from `Canopus V2` and `Serpul - Proteus Out` can't be moved to `https://callback.yapulsa.com` due to some reason as below:
	- Beta / Sanity for related callback.
	- There are some error-log report due to instable/inconsistence of releated feature.
- Portal Client App (PAIMIN)
	- Heavily use API endpoints to load data from GM (planned to be shutdown on July or August 2019)
	- Vulnerable to any hacker activity that affected to security issues on few API end-points (reported for last 2 months)


## Temporary Solutions
- GM End-point: `/api/v1/transaction/callback/{push_type}` 
	- Hosts:
		- `https://gm.yapulsa.com`
		- `https://gmapi.yapulsa.com`
		- `https://api.yapulsa.com`
		- `https://callback.yapulsa.com`
	- HTTP Method: `POST`
	- Rate Limit (per Remote IP Address): 
		- Last configured: `100 / minute; 10 / second`
		- Current situation: `6000 / minute; 100 / second`
- GM Another End-point due to Portal Client App (PAIMIN) access:
	- New Relic Report ([reference](https://rpm.newrelic.com/accounts/908709/applications/91783725/filterable_errors#/show/313df923-9355-11e9-92fe-0242ac110009_4379_8962/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart&filters=%5B%7B%22key%22%3A%22error.class%22%2C%22value%22%3A%22werkzeug.exceptions%3ATooManyRequests%22%2C%22like%22%3Afalse%7D%5D))
	- Hosts: `https://gm.yapulsa.com`
	- HTTP Method: `GET|POST|PUT|PATCH|DELETE|OPTIONS` 
	- Rate Limit (per `Remote IP Address` or per `user ID`): 
		- Last configured: `60 / minute` 
		- Current situation: `1000 / minute`
		- After Portal app shutdown: `200 / minute` or `300 / minute`
	
- Each Callback from third-party need to be moved from `https://gm.yapulsa.com` into `https://callback.yapulsa.com`.
- Risks:
	- Beta / Sanity testing for third-party integrations cannot be properly tested.