HackMD
Cofacts Incidents with Analytics
This document records known incidents that may impact the number reportings in:
Note: These information are provided in best effort and may not be comprehensive. Please use the open data at your own risk and make conclusions carefully.
Time below are in Taiwan time (UTC+8).
Incidents
2017/03 ~ 2017/04 garbage articles in DB
There are 1045 non-sense articles inserted into DB by 6 LINE user IDs.
You can find the list of UserId and articles here: https://docs.google.com/spreadsheets/d/1uqhbl2s_vktPAIJiKiu9L0yTD30Kk_B50W1OpDGhrU4/edit#gid=1409443958
Related discussion: https://hackmd.io/@mrorz/SkHt8JZ6l?type=view
2018/07/11 New articles requires "reason" to submit
In order to reduce the number of inadequate articles being submitted to database, we are enforcing the user to submit a "reason" why they think this is a rumor before letting the message enter the database.
- Initial report on reduction of new messages: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/https%3A%2F%2Fhackmd.io%2Fs%2Frkj-S9m4m
- Revamp reason submission to LIFF: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/%2Fs%2FSyN-sb8b4
2018/09/22 ~ 2019/5/20 Dropped search count for long messages in LINE bot
In 2019/5/20 we fixed a bug that would drop Google Analytics events with category Article & action Selected for long messages (more than 800 characters).
Therefore, the article selected event for long messages may be dropped after we start sending content to Google Analytics and before the bug is fixed.
2018/10/21 Messages with URL causes chatbot to crash
In 2018/10/21, ListArticle API would return error if the searched message contains URLs. This affects websites and Cofacts LINE bot.
2020/03/06 - 03/16 API server downtime
Minor API & website downtimes (< 2 hours each, recovers automatically)
Detail: https://g0v.hackmd.io/@mrorz/cofacts-meeting-notes/%2F%40mrorz%2FSJsDsS1I8
2020/03/09 Disk full issue
Cofacts server disk is full during 08:00 ~ 13:46. Therefore, no new articles and replies can be submitted at those times.
2021/10/03 ~ 2021/10/05 LINE bot events dropped
The LINE bot server lost connection to Google Analytics due to wrong DNS entry on production server.
In the mean time, all events from LINE bot (excepts those with event category being LIFF) cannot be sent to Google Analytics.
Related info: https://g0v.hackmd.io/zlWWCpDTQ5iADHNw7Thpog#LINE-bot-event-送不進-Google-Analytics
2021/11/19 11:00 - 14:40 server under DoS attack
During the time period, all access to API, website and LINE bot is extremely slow and is possibly dropped out.
Related info: https://g0v.hackmd.io/YaQbzua8SWmkm2DNSGD-eQ?view#Server-遭受攻擊紀錄
2023/06/05 - 06/06 DDoS attack
During the time period, it is possible that the user count and page view count of Cofacts website is affected.
The attack managed to deny service for about an hour. During that time, all traffic to Cofacts server are denied.
https://g0v.hackmd.io/dkyaltHuSp6jPPw9oL4Wig?view#DDoS-report
2023/10/06 - 2023/10/10 MongoDB on chatbot failed
Chatbot users are not able to select and rate fact-checking replies in this period.
2023/10/15 22:00 - 10/16 09:40 Chatbot not accessible
The root cause is a misconfiguration of nginx server as we migrate production English and Japanese production chatbot to Google Cloud Run.
During the incident, Taiwanese Mandarin version of the Cofacts chat bot was inaccessible.
2023/10/19 02:00 - 12:09 API not accessible
After the release of https://github.com/cofacts/rumors-api/releases/tag/release%2F20231019 , the IP of the API server changed.
However, nginx would cache IP addresses, thus nginx cannot correctly pass requests to the API server.
This impacts API server, thus the website and the chatbot cannot access any of the content from Cofacts database.
2024/05/19 17:30 - 20:07 DDoS attack
Cofacts service were under DDoS attack during this timeframe.
Website, API and chatbots are all unavailable.
We activated Cloudflare WAF rules to defend the attack.
For details, see https://g0v.hackmd.io/EjQAz1rERUKWfTy1M-cynA?both#20240519-DDoS-攻擊
2024/10/01 13:01 - 21:21 LINE bot not accesible
The root cause is a misconfiguration on firewall as we try to adjust the firewall settings to defend off malicious requests.
Taiwanese Mandarin version of the Cofacts chat bot was inaccessible during the period.
2024/11/24 17:00 - 2024/11/25 14:27 TW LINE bot not accessible
Due to high server load, LINE bot server is terminated around 2024/11/24 17:00 and is not restarted until 11/25 14:27.
