# Incident report ## What happened? Between 2022-08-17 11:13 and 11:58 (UTC+2) we overwrote some files in the job pool processing S3 bucket. These files were meant to be processed by the central job advert store microservice which failed because of our modifications. ### Details We used a function to determine the destination s3 path as ```python def __get_destination_s3_path(self): return self.source_path.replace('lensadl-bi-job-advert-management', get_bucket()) ``` which originally didn't contain `lensadl-bi-job-advert-management` (should have been `lensadl-job-feed-processing-production`), so it returned the original path. The lambda funtion has full access to S3, so it overwrote the original files. ### Immediate actions - We destroyed the stack at about 12:15, but fortunately we pushed a newer version around 11:55 which stopped the malfuntion. ## Affected components - Job Feed Processing (ForcePush) - Central Job Advert Store (ForcePush) - Job Pool Archive (DE) ## Causes - There is no staging SNS topic, which actually sends messages, so we used the _production_ SNS to develop scripts. - Lambda function has full access to S3. ## Prevent further cases - Restrict S3 access to read-only for `lensadl-job-feed-processing-production`. - Explicitly construct S3 destination. - There could be a staging SNS topic with actual messages.