---
# System prepended metadata

title: Incident report

---

# Incident report

## What happened?

Between 2022-08-17 11:13 and 11:58 (UTC+2) we overwrote some files in the job pool processing S3 bucket. These files were meant to be processed by the central job advert store microservice which failed because of our modifications.


### Details
We used a function to determine the destination s3 path as

```python
def __get_destination_s3_path(self):
    return self.source_path.replace('lensadl-bi-job-advert-management', get_bucket())

```
which originally didn't contain `lensadl-bi-job-advert-management` (should have been `lensadl-job-feed-processing-production`), so it returned the original path. The lambda funtion has full access to S3, so it overwrote the original files.

### Immediate actions
- We destroyed the stack at about 12:15, but fortunately we pushed a newer version around 11:55 which stopped the malfuntion. 


## Affected components
- Job Feed Processing (ForcePush)
- Central Job Advert Store (ForcePush)
- Job Pool Archive (DE)

## Causes

- There is no staging SNS topic, which actually sends messages, so we used the _production_ SNS to develop scripts.
- Lambda function has full access to S3.

## Prevent further cases

- Restrict S3 access to read-only for `lensadl-job-feed-processing-production`.
- Explicitly construct S3 destination.
- There could be a staging SNS topic with actual messages.