# 20220307 Storage Usage Spike in Kerjaansvc

## Overview

Error raised on export excel file endpoint and generating payslip for agent (PDF).

## Time and Duration

| Time | Event | Actor |
| -------- | -------- | -------- |
| 2023-03-02 11:49:00 | First occurence of disk utilization spike | - |
| 2023-03-06 11:50:00 | Disk utilization spiked to 100% | Hasbi Asshidiq |
| 2023-03-06 12:42:00 | [Bugbuster report](https://staffinc-co.slack.com/archives/C015UUA1K8F/p1678081347073959) submitted, attendance recap bulk | Ajeng Putri Kinasih |
| 2023-03-06 13:00:00 | Restart the service, disk utilization down to 0% then spiked to 40% | Bramanto Leksono, Hasbi Asshidiq |
| 2023-03-06 14:53:00 | [Bugbuster report](https://staffinc-co.slack.com/archives/C015UUA1K8F/p1678089206768079) submitted, payslip | Wahyu Widarti |
| 2023-03-06 16:30:00 | Temporary workaround for payslip export | Bramanto Leksono |
| 2023-03-06 17:00:00 | Access SSH, found and export core dump file to S3 | Febry Nugroho, Firyal Fakhrilhadi |

## Impact

Segmentation fault raised by `uvicorn.workers.UvicornWorker` (webserver used) causing Linux kernel to write core dump files on each segmentation fault occurence, which takes huge storage spaces (more than 700MB per file). Error no directory to write temporary file follow-up such condition since server container storage space is limited to 20GB.

### Impacted Service / Application

1. Kerjaansvc, export XLS/CSV file and PDF generator function
2. Client Portal, error shown on accessing export file functions

## Root Cause

//TODO: inspect segmentation fault log from monitor.sampingan.co

## Resolution

1. Restart kerjaansvc instance on high disk utilization rate (above 80%)
3. Install [`faulthandler`](https://docs.python.org/3/library/faulthandler.html) to ensure segmentation fault stack trace can be logged.
4. Export and inspect system log via https://monitor.sampingan.co
5. Resolve issue raised and logged in system log

## References

1. [Core dump file](https://wiki.archlinux.org/title/Core_dump)
2. [Kerjaansvc sample core dump file #1](https://s3.console.aws.amazon.com/s3/object/kerjaan2-dev?region=ap-southeast-1&prefix=core.1419)
3. [Kerjaansvc sample core dump file #2](https://s3.console.aws.amazon.com/s3/object/kerjaan2-dev?region=ap-southeast-1&prefix=core.1418)
4. [Kerjaansvc sample core dump file #3](https://s3.console.aws.amazon.com/s3/object/kerjaan2-dev?region=ap-southeast-1&prefix=core.22)
5. [Sentry error log on XLS export](https://sentry.sampingan.co/sampingan/kerjaansvc/issues/26252/?query=is%3Aunresolved)