---
# System prepended metadata

title: LWS outage 2020-01-20

---

# LWS outage 2020-01-20

## Facts

* Workers offline (reporting crashed, even after redeploy)
    * bumping RAM helped, although they don't use all the RAM
* AMQP large amount of unprocessed messages
    * Little visibility
    * Poor UI in admin tool makes some information unclear
    * Unusued queues
      > You have 147 queues without any consumers.
    * Some queues have a very large backlog
      >  We recommend less than 10 000 messages in all queues. You have 5 queues with more than 10 000 messages.

## Suggestions

* Delete empty queues without subscribers
    * When is the queue created?
        * We believe when the event is queued it should create
* Purge queues without subscribers
    * Question, can we save the data
* Delete several queues
    * BambooHR update supplier only contains supplier_id we have this data and iut's for one customer account
    * Intercom is a deleted integration
    * We no longer process events for created tasks for the one customer using BambooHR, it's accepted tasks


## Outcomes

* purged intercom queues
* removed SupplierClaimed event-handling from LWS (42k messages unprocessed)
* removed IntercomException base exception class
* unknown what is causing 
`evt-services-supplier_updated--bamboohr_provisioner.handle_supplier_updated` 500,000 messages got queued ([the code for that was removed in this commit](https://github.com/kalohq/lws/commit/7cc82a27d6b1f4c440cb88ad6605656dd5c0cbb8) on  Jan 4, 2019)
* linked Sentry issue https://sentry.io/organizations/kalo/issues/1024140461/?project=270183&query=is%3Aunresolved

## Engineers

* Lewis Cowles
* Tom Finill
