# LWS outage 2020-01-20 ## Facts * Workers offline (reporting crashed, even after redeploy) * bumping RAM helped, although they don't use all the RAM * AMQP large amount of unprocessed messages * Little visibility * Poor UI in admin tool makes some information unclear * Unusued queues > You have 147 queues without any consumers. * Some queues have a very large backlog > We recommend less than 10 000 messages in all queues. You have 5 queues with more than 10 000 messages. ## Suggestions * Delete empty queues without subscribers * When is the queue created? * We believe when the event is queued it should create * Purge queues without subscribers * Question, can we save the data * Delete several queues * BambooHR update supplier only contains supplier_id we have this data and iut's for one customer account * Intercom is a deleted integration * We no longer process events for created tasks for the one customer using BambooHR, it's accepted tasks ## Outcomes * purged intercom queues * removed SupplierClaimed event-handling from LWS (42k messages unprocessed) * removed IntercomException base exception class * unknown what is causing `evt-services-supplier_updated--bamboohr_provisioner.handle_supplier_updated` 500,000 messages got queued ([the code for that was removed in this commit](https://github.com/kalohq/lws/commit/7cc82a27d6b1f4c440cb88ad6605656dd5c0cbb8) on Jan 4, 2019) * linked Sentry issue https://sentry.io/organizations/kalo/issues/1024140461/?project=270183&query=is%3Aunresolved ## Engineers * Lewis Cowles * Tom Finill