# iOS users removal `#73342`

Date: 2022-07-15

Contributors:

* Martin Beránek (Ackee DevOps)

### People involved:

* Tomáš Hejátko (Ackee DevOps)
* Karina Balagazova (Ackee BE)
* Pavel Švagr (Ackee BE)
* Jaroslav Šmolík (Ackee BE)
* Jiří Šmolík (Ackee BE)
* Rostislav Babacek (Ackee iOS)
* Marek Elznic (Ackee PM)
* Michal Vlasák (Ackee BE)

### Summary:

Thirty-nine users got removed from the application by login in to iOS app version and calling delete actions on themselves.

#### Impact:

Thirty-nine users lost all of their data in the application. Restore of the database took multiple days. The impact on the user experience of deleted users was significant.

#### Root cause:

Largely unknown.

Based on the investigation: user passwords were guessed correctly multiple times. Possible weak passwords might be a root cause (it can't be excluded now).

See RPS of authentication API call:


##### Possible issue with the explanation:

* The attacker had to have list of users of the application,
* Passwords had to be very weak, probably with the same internal logic (e.g.: anicepassword12, anicepassword32, ...).

##### Symptoms supporting the root-cause:

* only users with usernames & passwords were removed (or it was not proven otherwise),
* only one IP address was used for the [`deleteMe`](https://gitlab.ack.ee/Backend/artdeal-userapi/-/blob/development/src/app/controllers/resolvers/userResolver.ts#L97) requests: [178.197.224.137](https://ipinfo.io/178.197.224.137)
* IP has not been widely use in [the past 60 days](https://console.cloud.google.com/logs/query;query=%22178.197.224.137%22;timeRange=P60D;cursorTimestamp=2022-07-14T00:25:29.950207Z?project=artdeal-production) in the application before the incident
* iOS application used for the incident was released [2 weaks before](https://gitlab.ack.ee/iOS/artdeal/-/pipelines/337759) the incident, `user-agent` used: `ArtDeal/3012 CFNetwork/1331.0.7 Darwin/21.4.0`
* During the incident, there was a 29% password guess error rate:

| Method | Requests | Errors|
|-------|-------|-------|
| google.cloud.identitytoolkit.v1.AuthenticationService.SignInWithPassword | 167 | 29.34% |
* Currently, only constraint to password is their length, which is 6 characters (e.g. password 123456 is ok by the app)
* User also called multiple other backend endpoints, e.g. `userMe`, `editMe`, `getMessageToken` suggesting possible testing scenarion done on production instead of stage/development

##### Correlated logs:

What's probably interesting is that IP was used in the cluster of requests at [13.7.2022 at 8 am](https://cloudlogging.app.goo.gl/3Qy7H6wUhpTMzW5j9).

There is no direct way to map requests from load balancer to the application. But, during that time, only requests from the IP [178.197.224.137](https://ipinfo.io/178.197.224.137) and [Cloud Scheduler](https://cloud.google.com/scheduler) IP were used. Only user using the application during that time was `[e-mail]`.

From the log entry, we can also see his address and phone number:

{
    __typename: "Address"
    city: "Sankt Moritz "
    country: "CH"
    email: "[e-mail]"
    id: "13"
    isBuyerDefaultBilling: false
    isBuyerDefaultShipping: false
    isSellerBilling: false
    isSellerDefaultShipping: true
    label: null
    name: "001.Gallery "
    note: ""
    phone: "[phone]"
    postalCode: "7500"
    street: "  Via Maistra 46"
}

See [google maps](https://goo.gl/maps/a2reGKqjQW1QHNYZ7) if interested, the phone number has no entry on [whosenumber.info](https://whosenumber.info/41796941891) page.

The user has been [first active](https://cloudlogging.app.goo.gl/ZfZ8mLK5uiLVfpzw7) at 22.6 with a few requests here & there. The user has been in the application since `Dec 21, 2021` (visible in [Firebase](https://console.firebase.google.com/u/0/project/artdeal-production/authentication/users)).

### Resolution:

No direct fix of the root cause is currently known. An additional investigation based on other data is required. Currently, we don't know.

### Detection:

By customer during morning [email checkup](https://ackee.slack.com/archives/C02DNJNL9PX/p1657780568660699). Large number of users deleted their account.

### Action items:

* Backend logs should be improved to be more readable. Currently, the JSON output tends to [be too large](https://console.cloud.google.com/logs/query;cursorTimestamp=2022-07-14T01:49:03.348733Z;query=timestamp%3D%222022-07-14T01:49:03.348733Z%22%0AinsertId%3D%2262cf760f0005523dda609197%22;timeRange=2022-07-13T23:15:00.000Z%2F2022-07-14T01:51:00.000Z?project=artdeal-production) for the Google Monitoring to process correctly.
* Firebase authentication audits logs might be enabled to make better visibility in the possible upcoming issues. Since this is not a default setup, Firebase support ticket 10188798 has been raised.
* Backup of the [firebase accounts](https://firebase.google.com/docs/cli/auth#authexport) once a day helps with recovery.
* Consider database [PITR](https://cloud.google.com/sql/docs/postgres/backup-recovery/pitr) backup to help with smaller [RTO](https://www.enterprisestorageforum.com/management/rpo-and-rto-understanding-the-differences/)
* Backup tools for Mux is currently uknown

### Lessons Learned

#### What went well:

* Database data could be restored from the day before
* Firebase accounts could have been recreated with different passwords
* GCS bucket data could be restored due to versioning being enabled (that is not the default setup)
* [GetStream](https://ackee.slack.com/archives/C02DNJNL9PX/p1657882223854429) users were only marked as deleted, not deleted permanently

#### What went wrong:

* There is no audit of user activity from Firebase
* There is no password hardening enforcement
* Backup of the database was day old
* [Mux](https://ackee.slack.com/archives/C02DNJNL9PX/p1657882348775659?thread_ts=1657882223.854429&cid=C02DNJNL9PX) data were lost
* [Major release](https://ackee.slack.com/archives/C02DNJNL9PX/p1657919486706459) was planned on 16.7.2022, which created a lot of informational fog

#### Where we got lucky:

* Only thirty-nine users got removed
* Only users with usernames & passwords got removed
* Attack was done by a single IP address once, never repeated in the next days
* Database data could be restored from the day before thanks to Karina creating a backup before deploying to production

### Timeline:

* 14.7.2022 8:36 - Issue reported in the [Slack channel](https://ackee.slack.com/archives/C02DNJNL9PX/p1657780568660699) by customer
* 14.7.2022 10:30 - Suspected [CRON](https://ackee.slack.com/archives/C02DNJNL9PX/p1657782525781399?thread_ts=1657780568.660699&cid=C02DNJNL9PX) was investigated, but it is not even in the development [environment during given time](https://ackee.slack.com/archives/C02DT1C8G4C/p1657784125980579?thread_ts=1657780702.774909&cid=C02DT1C8G4C)
* 14.7.2022 14:30 - list of [users removed from the database](https://ackee.slack.com/archives/C02DNJNL9PX/p1657801817994099?thread_ts=1657797771.292569&cid=C02DNJNL9PX) has been published
* 15.7.2022 17:19 - GCS data were restored
* 15.7.2022 10:30 - [No additional users were removed](https://ackee.slack.com/archives/C02DNJNL9PX/p1657877400206039) suggesting removed users had to have some attribute in common
* 19.7.2022 13:22 - database users' UUID were [restored in database](https://ackee.slack.com/archives/C03DPTEJ1TL/p1658229770711829?thread_ts=1658154099.536709&cid=C03DPTEJ1TL)

### Supporting information:

* Logs with user deletation https://cloudlogging.app.goo.gl/np2Eq3uVXdCKrji29
* Logs from whole incident https://cloudlogging.app.goo.gl/En4dKsFeDGBUpJYR8