---
# System prepended metadata

title: Cosmed Sale Research 20200815

---

# Cosmed Sale Research 20200815

## 現場反饋
- 開賣後分類頁仍顯示尚未開賣、售完後分類頁未顯示已售完
    - 兩層 CDN 導致 delay 4~6min
- 商品頁持續顯示“開賣通知我"
- 收藏 load 不出來
- iOS 首頁白頁
- 監控到的 error / latency 下降


## 觀察重點

```
比較基準

墊腳石 8/12 搶購：
- time 0812 17:50 ~ 0812 18:10
- max RPM 93k

康是美 8/15 搶購：
- time 0815 10:50 ~ 0815 11:10
- max RPM 75k
```

- [ ] RealTimeData 回來的慢、fail
- [ ] 收藏頁回來的慢、fail
- [ ] 首頁回來的慢、fail
- [x] 是否因為調整 timeout 時間, 導致產生更多 timeout error. 還是該 timeout 的不管 5s, 20s 都會 timeout
- [x] error 是否有變多
- [ ] 加 CDN 會有幫助嗎


## 要拿出來討論的事項整理
- [ ] timeout error 變多的情況推論 (p100 -> p90, 10x timeout)





## 研究內容

### error 是否有變多

> 研究人員: `@tomaz`

- 兩天比較
  - 康是美 8/15 搶購: RPM max 75k, error sum 1500 * 10 = 15000
  - 墊腳石 8/12 搶購: RPM max 93k, error sum 2819
- 數量多出了很多，但是佔比差不多
- 展開原因也發現分佈比例類似



**康是美 8/15 搶購 - error 數量**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-15T02:50:00.000Z  
end-time: 2020-08-15T03:10:00.000Z  
query-string:
```
filter ( type = "ERROR" and tag = "OperationLoggingApolloServerPlugin" )
| stats count() as cnt by data.operationName
| sort cnt desc

```
-------------------------------------------
|        data.operationName        | cnt  |
|----------------------------------|------|
| iOS_salePageRealTimeData         | 740  |
| android_salePage_realtime_info   | 614  |
| iOS_salePageAdditionalInfo       | 27   |
| cms_shopCategory_default_orderby | 27   |
| iOS_salePageInfo                 | 27   |
| cms_shopCategory                 | 24   |
| android_salePage                 | 18   |
| android_salePage_extra           | 8    |
| iOS_shopCategory                 | 6    |
| <no-operation>                   | 3    |
| cms_shopCategory_promotion_list  | 2    |
| android_getSalePageInit          | 2    |
| android_searchHotKeywords        | 1    |
| cms_layoutTemplate_spCatAd_list  | 1    |
| total                            | 1500 |
-------------------------------------------


**康是美 8/15 搶購 - 展開原因**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-15T02:50:00.000Z  
end-time: 2020-08-15T03:10:00.000Z  
query-string:
```
filter ( type = "ERROR" and tag = "OperationLoggingApolloServerPlugin" )
| parse err.message /text: (?<reason>.+)/
| stats count() as cnt by data.operationName, data.error.type, reason
| sort cnt desc

```
------------------------------------------------------------------------------
|        data.operationName        | data.error.type |     reason      | cnt |
|----------------------------------|-----------------|-----------------|-----|
| iOS_salePageRealTimeData         | Error           | request timeout | 732 |
| android_salePage_realtime_info   | Error           | request timeout | 608 |
| cms_shopCategory_default_orderby | Error           | request timeout | 24  |
| cms_shopCategory                 | Error           | request timeout | 19  |
| iOS_salePageAdditionalInfo       | Error           | request timeout | 18  |
| iOS_salePageInfo                 | Error           | request timeout | 18  |
| android_salePage                 | PythiaError     |                 | 9   |
| iOS_salePageAdditionalInfo       | PythiaError     |                 | 9   |
| iOS_salePageInfo                 | PythiaError     |                 | 9   |
| android_salePage                 | Error           | request timeout | 9   |
| iOS_salePageRealTimeData         | PythiaError     |                 | 8   |
| android_salePage_extra           | Error           | request timeout | 7   |
| android_salePage_realtime_info   | PythiaError     |                 | 6   |
| iOS_shopCategory                 | Error           | request timeout | 6   |
| cms_shopCategory                 | PythiaError     |                 | 5   |
| cms_shopCategory_default_orderby | PythiaError     |                 | 3   |
| <no-operation>                   | Error           | request timeout | 3   |
| android_getSalePageInit          | PythiaError     |                 | 2   |
| cms_shopCategory_promotion_list  | Error           | request timeout | 2   |
| android_searchHotKeywords        | Error           | request timeout | 1   |
| android_salePage_extra           | PythiaError     |                 | 1   |
| cms_layoutTemplate_spCatAd_list  | Error           | request timeout | 1   |
------------------------------------------------------------------------------


**墊腳石 8/12 搶購 - error 數量**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-12T09:50:00.000Z  
end-time: 2020-08-12T10:10:00.000Z  
query-string:
```
filter ( type = "ERROR" and tag = "OperationLoggingApolloServerPlugin" )
| stats count() as cnt by data.operationName
| sort cnt desc
```

-------------------------------------------
|        data.operationName        | cnt  |
|----------------------------------|------|
| iOS_salePageRealTimeData         | 1216 |
| android_salePage_realtime_info   | 913  |
| iOS_salePageAdditionalInfo       | 182  |
| iOS_salePageInfo                 | 180  |
| cms_shopCategory                 | 122  |
| cms_shopCategory_default_orderby | 104  |
| android_salePage_extra           | 33   |
| android_salePage                 | 30   |
| iOS_shopCategory                 | 14   |
| iOS_leftMenu                     | 11   |
| android_getSalePageInit          | 8    |
| CouponListPage                   | 2    |
| cms_shopCategory_promotion_list  | 1    |
| iOS_couponList                   | 1    |
| iOS_appAnnouncementList          | 1    |
| CouponList                       | 1    |
| total                            | 2819 |
-------------------------------------------


**墊腳石 8/12 搶購 - 展開原因**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-12T09:50:00.000Z  
end-time: 2020-08-12T10:10:00.000Z  
query-string:
```
filter ( type = "ERROR" and tag = "OperationLoggingApolloServerPlugin" )
| parse err.message /text: (?<reason>.+)/
| stats count() as cnt by data.operationName, data.error.type, reason
| sort cnt desc

```
-------------------------------------------------------------------------------
|        data.operationName        | data.error.type |     reason      | cnt  |
|----------------------------------|-----------------|-----------------|------|
| iOS_salePageRealTimeData         | Error           | request timeout | 1048 |
| android_salePage_realtime_info   | Error           | request timeout | 892  |
| iOS_salePageInfo                 | PythiaError     |                 | 158  |
| iOS_salePageAdditionalInfo       | PythiaError     |                 | 158  |
| iOS_salePageRealTimeData         | PythiaError     |                 | 157  |
| cms_shopCategory                 | PythiaError     |                 | 103  |
| cms_shopCategory_default_orderby | PythiaError     |                 | 83   |
| android_salePage_extra           | PythiaError     |                 | 22   |
| android_salePage                 | PythiaError     |                 | 22   |
| cms_shopCategory_default_orderby | Error           | request timeout | 21   |
| android_salePage_realtime_info   | PythiaError     |                 | 21   |
| cms_shopCategory                 | Error           | request timeout | 19   |
| iOS_salePageAdditionalInfo       | Error           | request timeout | 13   |
| iOS_salePageAdditionalInfo       | Error           | request failed  | 11   |
| iOS_salePageInfo                 | Error           | request timeout | 11   |
| iOS_salePageRealTimeData         | Error           | request failed  | 11   |
| android_salePage_extra           | Error           | request timeout | 11   |
| iOS_salePageInfo                 | Error           | request failed  | 11   |
| iOS_leftMenu                     | PythiaError     |                 | 10   |
| iOS_shopCategory                 | Error           | request timeout | 8    |
| android_salePage                 | Error           | request timeout | 7    |
| iOS_shopCategory                 | PythiaError     |                 | 6    |
| android_getSalePageInit          | PythiaError     |                 | 6    |
| CouponListPage                   | Error           | request timeout | 2    |
| android_getSalePageInit          | Error           | request timeout | 2    |
| iOS_couponList                   | Error           | request timeout | 1    |
| iOS_appAnnouncementList          | Error           | request timeout | 1    |
| android_salePage                 | Error           | request failed  | 1    |
| iOS_leftMenu                     | Error           | request timeout | 1    |
| cms_shopCategory_promotion_list  | Error           | request timeout | 1    |
| CouponList                       | Error           | request timeout | 1    |
-------------------------------------------------------------------------------


### 是否因為 timeout 時間縮短了，導致 timeout error 大幅提升
> 研究人員: `@tomaz`

- 平時的 timeout error 在修改前、修改後並沒有太大差異 
  (平時不會 timeout 的都能在 5s 內回應, 超過 5s 的就會 timeout 了，不管等 5s, 20s 都一樣)
- 搶購時 timeout error 倍數上升，但 api 分佈比例類似
  - 墊腳石 2823
  - 康是美 3245 * 10 = 32450
- 推論: **修改前 p100 的人會 timeout, 修改後 p90 的人會 timeout, 大約 10x 人數**

- 那得到了什麼
  - 墊腳石 latency p95 7s, avg 1.2s
  - 康是美 latency p95 4.2s, avg < 300ms



**墊腳石 timeout error**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-12T09:50:00.000Z  
end-time: 2020-08-12T10:10:00.000Z  
query-string:
```
filter ( data.status = "timeout" and type = "ERROR" )
| parse data.url /^https?:\/\/(?<service>.+?)\/(?<path>([^\.]*?\/){0,3}).*?$/
| stats count() as _count by service, path
| sort _count desc
```
--------------------------------------------------------------------------------------
|     service      |                          path                          | _count |
|------------------|--------------------------------------------------------|--------|
| webapi.91mai.com | webapi/SalePage/GetSalePageRealTimeData/               | 1925   |
| webapi.91mai.com | webapi/TraceSalePageList/                              | 229    |
| webapi.91mai.com | webapi/SalePageV2/GetSalePageV2Info/                   | 188    |
| webapi.91mai.com | webapi/ShopCategory/GetSalePageList/                   | 75     |
| webapi.91mai.com | webapi/Shop/                                           | 55     |
| webapi.91mai.com | webapi/LayoutTemplateData/GetLayoutTemplateData/       | 55     |
| webapi.91mai.com | webapi/SearchV2/                                       | 36     |
| webapi.91mai.com | webapi/SalePage/GetSalePageHotListByShopCategoryId/    | 27     |
| webapi.91mai.com | webapi/SalePageV2/                                     | 25     |
| webapi.91mai.com | webapi/APPNotification/                                | 23     |
| webapi.91mai.com | webapi/shop/getCustomizedBrandIdentityDisplaySettings/ | 22     |
| webapi.91mai.com | webapi/shop/getForcedLogoutVersionList/                | 21     |
| webapi.91mai.com | webapi/ShopCategory/GetPromotionList/                  | 20     |
| webapi.91mai.com | webapi/SalePageV2/GetSalePageAdditionalInfo/           | 18     |
| webapi.91mai.com | webapi/ShopStaticSetting/                              | 15     |
| webapi.91mai.com | webapi/AppAnnouncement/                                | 15     |
| webapi.91mai.com | webapi/AppNotification/GetMobileAppSettings/           | 14     |
| webapi.91mai.com | webapi/Shop/GetShopCategoryListV3/                     | 12     |
| webapi.91mai.com | webapi/AppAnnouncement/getAppAnnouncementList/         | 10     |
| webapi.91mai.com | webapi/ecoupon/                                        | 9      |
| webapi.91mai.com | webapi/Shop/GetShopintroduction/                       | 8      |
| api2.91mai.com   | o2o/api/coupon/                                        | 7      |
| webapi.91mai.com | webapi/PromotionV2/GetList/                            | 5      |
| webapi.91mai.com | webapi/Activity/                                       | 4      |
| webapi.91mai.com | webapi/HotSaleRanking/GetHotSaleRankingList/           | 2      |
| webapi.91mai.com | webapi/shop/getShopContractSetting/                    | 2      |
| webapi.91mai.com | webapi/InfoModule/                                     | 1      |
--------------------------------------------------------------------------------------



**康是美 timeout error**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-15T02:50:00.000Z  
end-time: 2020-08-15T03:10:00.000Z  
query-string:
```
filter ( data.status = "timeout" and type = "ERROR" )
| parse data.url /^https?:\/\/(?<service>.+?)\/(?<path>([^\.]*?\/){0,3}).*?$/
| stats count() as _count by service, path
| sort _count desc
```
---------------------------------------------------------------------------------------------------
|            service            |                          path                          | _count |
|-------------------------------|--------------------------------------------------------|--------|
| webapi.91mai.com              | webapi/SalePage/GetSalePageRealTimeData/               | 1324   |
| webapi.91mai.com              | webapi/TraceSalePageList/                              | 1182   |
| d38tzu0atxk400.cloudfront.net | webapi/SalePageV2/GetSalePageV2Info/                   | 152    |
| d38tzu0atxk400.cloudfront.net | webapi/ShopCategory/GetSalePageList/                   | 84     |
| webapi.91mai.com              | webapi/Shop/                                           | 63     |
| webapi.91mai.com              | webapi/SalePage/GetSalePageHotListByShopCategoryId/    | 57     |
| webapi.91mai.com              | webapi/SearchV2/                                       | 35     |
| webapi.91mai.com              | webapi/SalePageV2/                                     | 32     |
| webapi.91mai.com              | webapi/APPNotification/                                | 31     |
| webapi.91mai.com              | webapi/shop/getCustomizedBrandIdentityDisplaySettings/ | 31     |
| d38tzu0atxk400.cloudfront.net | webapi/ShopCategory/GetPromotionList/                  | 30     |
| webapi.91mai.com              | webapi/ShopStaticSetting/                              | 30     |
| webapi.91mai.com              | webapi/AppAnnouncement/                                | 29     |
| webapi.91mai.com              | webapi/shop/getForcedLogoutVersionList/                | 23     |
| d38tzu0atxk400.cloudfront.net | webapi/LayoutTemplateData/GetLayoutTemplateData/       | 22     |
| webapi.91mai.com              | webapi/Activity/                                       | 21     |
| webapi.91mai.com              | webapi/AppNotification/GetMobileAppSettings/           | 19     |
| webapi.91mai.com              | webapi/AppAnnouncement/getAppAnnouncementList/         | 19     |
| webapi.91mai.com              | webapi/ecoupon/                                        | 17     |
| webapi.91mai.com              | webapi/Shop/GetShopintroduction/                       | 15     |
| d38tzu0atxk400.cloudfront.net | webapi/SalePageV2/GetSalePageAdditionalInfo/           | 13     |
| webapi.91mai.com              | webapi/HotSaleRanking/GetHotSaleRankingList/           | 7      |
| webapi.91mai.com              | webapi/shop/getShopContractSetting/                    | 3      |
| d38tzu0atxk400.cloudfront.net | webapi/Shop/GetShopCategoryListV3/                     | 3      |
| webapi.91mai.com              | webapi/InfoModule/                                     | 1      |
| api2.91mai.com                | o2o/api/coupon/                                        | 1      |
| webapi.91mai.com              | webapi/PromotionV2/GetList/                            | 1      |
---------------------------------------------------------------------------------------------------


**平時的 timeout error (改後)**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-14T12:50:00.000Z  
end-time: 2020-08-14T13:10:00.000Z  
query-string:
```
filter ( data.status = "timeout" and type = "ERROR" )
|stats count()
#| parse data.url /^https?:\/\/(?<service>.+?)\/(?<path>([^\.]*?\/){0,3}).*?$/
#| stats count() as _count by service, path
#| sort _count desc
```
-----------
| count() |
|---------|
| 47      |
-----------


**平時的 timeout error (改前)**  
region: ap-northeast-1  
log-group-names: /eks-ap-northeast-1-tw-91app-io/prod-bff  
start-time: 2020-08-07T12:50:00.000Z  
end-time: 2020-08-07T13:10:00.000Z  
query-string:
```
filter ( data.status = "timeout" and type = "ERROR" )
|stats count()
#| parse data.url /^https?:\/\/(?<service>.+?)\/(?<path>([^\.]*?\/){0,3}).*?$/
#| stats count() as _count by service, path
#| sort _count desc
```
-----------
| count() |
|---------|
| 59      |
-----------