---
tags: cofacts, analytics, incidents
GA: UA-98468513-3
---

# Cofacts Incidents with Analytics

This document records known incidents that may impact the number reportings in:
- [Cofacts analytics](https://cofacts.tw/analytics)
- [Cofacts Data](https://github.com/cofacts/opendata)
- [Cofacts API](https://api.cofacts.tw)

Note: These information are provided in best effort and may not be comprehensive. Please use the open data at your own risk and make conclusions carefully.

Time below are in Taiwan time (UTC+8).

## Incidents

### 2017/03 ~ 2017/04 garbage articles in DB

There are 1045 non-sense articles inserted into DB by 6 LINE user IDs.

You can find the list of UserId and articles here: https://docs.google.com/spreadsheets/d/1uqhbl2s_vktPAIJiKiu9L0yTD30Kk_B50W1OpDGhrU4/edit#gid=1409443958

Related discussion: https://hackmd.io/@mrorz/SkHt8JZ6l?type=view

### 2018/07/11 New articles requires "reason" to submit

In order to reduce the number of inadequate articles being submitted to database, we are enforcing the user to submit a "reason" why they think this is a rumor before letting the message enter the database.

- Initial report on reduction of new messages: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/https%3A%2F%2Fhackmd.io%2Fs%2Frkj-S9m4m
- Revamp reason submission to LIFF: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/%2Fs%2FSyN-sb8b4

### 2018/09/22 ~ 2019/5/20 Dropped search count for long messages in LINE bot

In 2019/5/20 we [fixed a bug](https://github.com/cofacts/rumors-line-bot/commit/3c4b5236bb3cb6262aa873b6d78b676235fcb9fa) that would drop Google Analytics events with category `Article` & action `Selected` for long messages [(more than 800 characters)](https://github.com/cofacts/rumors-line-bot/issues/97).

Therefore, the article selected event for long messages may be dropped after we [start sending content to Google Analytics](https://github.com/cofacts/rumors-line-bot/pull/99) and before the bug is fixed.

### 2018/10/21 Messages with URL causes chatbot to crash

In 2018/10/21, `ListArticle` API would return error if the searched message contains URLs. This affects websites and Cofacts LINE bot.

Detail: https://g0v.hackmd.io/xd8i1oR-Q9OEFQjOCPOplQ#Incident-1-LINE-bot-does-not-respond-when-URL-is-attached

### 2020/03/06 - 03/16 API server downtime

Minor API & website downtimes (< 2 hours each, recovers automatically)

Detail: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/%2F%40mrorz%2FSJsDsS1I8

### 2020/03/09 Disk full issue

Cofacts server disk is full during 08:00 ~ 13:46. Therefore, no new articles and replies can be submitted at those times.

Detail: https://g0v.hackmd.io/@johnson/BkjIIk8B8#39-Disk-full-website-amp-line-bot-%E2%80%9Cindex-is-read-only%E2%80%9D-error

### 2021/10/03 ~ 2021/10/05 LINE bot events dropped


The LINE bot server lost connection to Google Analytics due to wrong DNS entry on production server.

In the mean time, [all events from LINE bot](https://github.com/cofacts/rumors-line-bot#google-analytics-events-table) (excepts those with event category being `LIFF`) cannot be sent to Google Analytics.

Related info: https://g0v.hackmd.io/zlWWCpDTQ5iADHNw7Thpog#LINE-bot-event-%E9%80%81%E4%B8%8D%E9%80%B2-Google-Analytics

### 2021/11/19 11:00 - 14:40 server under DoS attack

During the time period, all access to API, website and LINE bot is extremely slow and is possibly dropped out.

Related info: https://g0v.hackmd.io/YaQbzua8SWmkm2DNSGD-eQ?view#Server-%E9%81%AD%E5%8F%97%E6%94%BB%E6%93%8A%E7%B4%80%E9%8C%84

### 2023/06/05 - 06/06 DDoS attack

During the time period, it is possible that the user count and page view count of Cofacts website is affected.

The attack managed to deny service for about an hour. During that time, all traffic to Cofacts server are denied.

https://g0v.hackmd.io/dkyaltHuSp6jPPw9oL4Wig?view#DDoS-report

### 2023/10/06 - 2023/10/10 MongoDB on chatbot failed

Chatbot users are not able to select and rate fact-checking replies in this period.


### 2023/10/15 22:00 - 10/16 09:40 Chatbot not accessible

The root cause is a misconfiguration of nginx server as we migrate production English and Japanese production chatbot to Google Cloud Run.

During the incident, Taiwanese Mandarin version of the Cofacts chat bot was inaccessible.

### 2023/10/19 02:00 - 12:09 API not accessible

After the release of https://github.com/cofacts/rumors-api/releases/tag/release%2F20231019 , the IP of the API server changed.
However, [nginx would cache IP addresses](https://github.com/docker/compose/issues/2003#issuecomment-139022760), thus nginx cannot correctly pass requests to the API server.

This impacts API server, thus the website and the chatbot cannot access any of the content from Cofacts database.