Backend Archtecture Notes

# Backend Archtecture Notes ## Background Jobs Currently we rely a lot on background jobs in order to integration and synchronize our backend with third-party services (RevenueCat, SendGrid, etc...). However, most of these jobs would not be required, should we change the architecture and make our API the single source of truth for our business logic: - [ ] Revenue Cat hidden behind a /subscriptions endpoint - [ ] No data manipulation directly from Adminium Here're a few good practices that we should consider implementing: - [ ] Background jobs should be idempotent/transactional, “retry’able” and MUST handle integration errors with third-party services (SendGrid, RevenueCat, etc…); - [ ] Our API should be the single source of truth for business logic and should hide ALL integrations with third-party services such as RevenueCat, SendGrid, etc... - [ ] No data should be manipulated directly in the database, instead, an admin page should be provided, and this one should rely on our API to implement the required features. - [ ] Recurrent background jobs that need to process huge amounts of data should be used as last resort since these ones require a lot of infrastructure resources (Recalculating numerology fields for all users every year for instance). ### Subscriptions: Revenue Cat - [] Rely on webhooks to update subscriptions in Heroku in "real-time"; - [] Rely on RevenueCat's export to fix existing SendGrid entries - - [ ] There should be a subscriptions endpoint and the backend should be the one talking to revenue cat, and not the client. This will: - [ ] Centralize the subscription logic in one place (our backend) as opposed to across multiple clients; - [ ] Multiple clients (iOS, android, web, etc) will be able to reuse the same API; - [ ] Allows us to switch from revenucat to any other provider in the future without having to change any of our clients; - [ ] Allow the backend to track subscriptions in realtime as opposed to having nightly builds processing a huge number of users; Teledipity relies on Sendgrid to deliver its marketing emails and newsletter. Every user in our system is added to one or more contact lists in Sendgrid, based on their personal year (PY) information and whether or not they are a PREMIUM account (paid user). That's because users will get different content depending on their personal year number and whether or not they are paid users. Teledipity as of this writing has 11 personal years, they are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, and 22. Each personal year, ends up having two contact lists in Sendgrid, one for FREE users, and another for PREMIUM users, adding up to 22 contact lists for personal years. Finally, there are two more contact lists, one used to communicate with ALL premium users, and another to track user who have cancelled their premium membership, giving us a total of 24 contact lists in SendGrid. ## Integration use cases In order to keep the user experience accurate, it's important to keep SendGrid's contact lists up-to-date with the information we have in our records (Heroku's database), so a few integration points are required: ### 1. Creating a new user (Critical) When a new user is created in Teledipity, a SendGrid contact must be created so that they can receive our newsletter and marketing emails. <details> <summary>Example: Free account</summary> **Given** that a new user is created in Teledipity with personal year 3 **When** the user is a FREE account **Then** their SendGrid contact is created and added to the FREE PY3 list </details> <details> <summary>Example: Premium account</summary> **Given** that a new user is created in Teledipity with personal year 3 **When** the user is a PREMIUM account **Then** their SendGrid contact is created and added to the PREMIUM PY3 list **And** the contact is also added to the "ALL PREMIUM users" list </details> ### 2. Updating an existing user When updates are made to user fields that are part of their SendGrid contact info, then the SendGrid contact must be updated as well. There are two scenarios to account for: #### 2.1. Changes to `personal_year` (Critical) When a user's personal year changes, its contact in SendGrid must be moved to a new contact list, corresponding to the new personal year value. <details> <summary>Example: Free account</summary> **Given** that a FREE user with personal year of 3 **When** that user's personal year changes to say 4 **Then** the corresponding SendGrid contact should be `removed` from the FREE PY3 list **And** that contact should be added to the FREE PY4 list </details> <details> <summary>Example: Premium account</summary> **Given** that a PREMIUM user with personal year of 3 **When** that user's personal year changes to say 4 **Then** the corresponding SendGrid contact should be `removed` from the PREMIUM PY3 list **And** that contact should be added to the PREMIUM PY4 list </details> #### 2.2. Changes to fields other than `personal_year` (Critical?) When a user is updated in our records without changing their "personal year (PY)", if the information updated is part of the SendGrid contact, then the contact must be updated in SendGrid, in this case there's no need to change any of the contact lists the user currently belongs to. Teledipity as of this writing uses the following user fields to create a SendGrid contact: email full_name preferred_name life_path expression soul_urge personality first_cycle birth_day personal_year personal_month overbalanced_energy_1 overbalanced_energy_2 overbalanced_energy_3 balanced_energy_1 balanced_energy_2 balanced_energy_3 underbalanced_energy_1 underbalanced_energy_2 underbalanced_energy_3 mobile_number telnum admin accepted_terms first_name last_name locale **Confirm with Andrew if we need to keep that same list or if we can narrow it down** <details> <summary>Example: Updating user info</summary> **Given** that an existing user in Teledipity **When** the user updates for instance, their name and date of birth **Then** the corresponding SendGrid contact must be updated with the new values **And** no contact list needs to be updated </details> ### 3. Unsubscribing a user (Non-critical) When users unsubscribe from marketing emails, their contact must be deleted from **all** SendGrid lists. That is because SendGrid charge us per contact. <details> <summary>Example</summary> **Given** a PREMIUM account whose personal year is 3 **When** the user unsubscribes from marketing emails **Then** the user's SendGrid contact should be removed from the PREMIUM PY3 list **And** removed from the **ALL PREMIUM users** list </details> ### 4. Premium subscription (critical) When a FREE account is upgraded to PREMIUM, the SendGrid contact must be moved from their FREE contact lists to the corresponding PREMIUM contact lists. <details> <summary>Example</summary> **Given** a FREE user whose personal year is 3 **When** they upgrade to PREMIUM **Then** their SendGrid contact must be moved from the FREE PY3 contact list to the PREMIUM PY3 contact list **And** that contact must be added to the "ALL PREMIUM users" contact list </details> ### 5. Premium cancellation (Non-critical) When a PREMIUM acocunt becomes FREE through cancellation, payment failure (SUBSCRIBED BY ADMIN goes from Y to N), or premium subscription expired by admin (`subscribed_by_admin = true`), the SendGrid contact must be moved from the PREMIUM contact lists to their corresponding FREE contact lists. <details> <summary>Example</summary> **Given** a PREMIUM account whose personal year is 3 **When** they cancel their PREMIUM subscription or fail to pay for the subscription **Then** their SendGrid contact must be moved from the PREMIUM PY3 contact list to the FREE PY3 contact list **And** that contact must be removed from the "ALL PREMIUM users" contact list **And** that contact must be added to the "CANCELLED MEMBERSHIP" contact list </details> ### 6. Deleting a user (Non-critical) When a user is deleted from Teledipity, their corresponding SendGrid contact must be deleted as well and removed from any contact list. Users can be deleted in a few ways: 1. Users delete their own account from the settings screen on the mobile app; 2. Users delete their account from the website (wordpress?), which triggers a webhook call to Teledipity's API which takes care of deleting the user from our records; 3. Users are marked by the admin to be deleted via Adminium (`will_be_deleted = true`) and a recurrent job takes care of deleting these users, at the moment this job is scheduled to run every 10 min. Currently, the backend will perform one API request per user and does not leverage any batch deletion APIs. In all three scenarios above, the corresponding SendGrid contacts should be deleted. <details> <summary>Example: Free account</summary> **Given** an existing FREE user with personal year of 3 **When** that user is deleted from our records **Then** the corresponding SendGrid contact should be `removed` from the FREE PY3 list **And** removed from the CANCELLED MEMBERSHIP list if present </details> <details> <summary>Example: Premium account</summary> **Given** an existing PREMIUM user with personal year of 3 **When** that user is deleted from our records **Then** the corresponding SendGrid contact should be `removed` from the PREMIUM PY3 list **And** removed from the "ALL PREMIUM users" list </details> #### Suggested Implementation Given that user deletion is considered a non-critical path, deleting users in batches can likely simplify this use case while reducing the amount of hits to the SendGrid API. Example: 1. User clicks the delete button on the app, or user is marked to be deleted by the admin, or a webhook call from the website tells our backend to mark the user to be deleted, e.g. `will_be_deleted = true`; 2. Users marked to be deleted have the auth tokens revoked and are "locked out of the app", e.g. sign in as a user marked to be deleted will fail with possibly a 404 error (Not Found); 3. Once a day (or as often as needed) a scheduled job collects all users marked to be deleted and: 1. Delete the corresponding SendGrid contacts in batches of 30k (limit imposed by SendGrid); 2. Delete the batched users from our database. This solution centralizes the logic for deleting users in one place which will allow us to implement better error handling, and reduce the amount of Jobs required to handle the user deletion use case. # Managing SendGrid Contact Lists - Add contacts to specific lists when: - A new user is created - User's personal year changes - User purchases a premium subscription - User cancels their premium subscription - Remove contacts from specific lists when: - User's personal year changes - User purchases a premium subscription - User cancels their premium subscription # Managing SendGrid Contacts - Create a contact when: - A new user is created - Remove a contact when: - A user is deleted - Update a contact when: - the corresponding user is updated affecting any of the fields belonging to the contact # Requirements ## Enabling and Disabling SendGrid Integration It's important to be able to disable certain integration points for various occasions, example: Jan 1st issues, data maintenace at the DB level, etc... This mechanism should affect all integration points, except for deleting contacts. ## Error Handling And Retries Jobs should be idempotent as much as possible and **must** handle SendGrid errors so they can be automatically retried eventually. ## SendGrid Updates ### Adhoc SendGrid Updates The simplest way to keep SendGrid synchronized with our backend, is to enqueue jobs as soon as possible to update a given contact. However, in certain occasions we may hit the rate limits restrictions imposed by SendGrid, in which case, we can implement our own rate limit mechanism at the active job level via https://github.com/nickelser/activejob-traffic_control. This gem essentially extends Rails' active job (backed by sidekiq) so that we can control how many instances of the same job can run cuncurrently and how many are allowed to run within the rate limit period, once the rate limit is hit, new jobs are enqueuend to be executed later, given us visibility to these jobs and ability to retry them when needed. Implementation Sketch: ```rb class User < ApplicationRecord after_create :create_send_grid_contact after_destroy :delete_send_grid_contact after_update :update_send_grid_contact private def create_send_grid_contact SendGridJobs::CreateContactJob.perform_later(self) end def delete_send_grid_contact SendGridJobs::DeleteContactJob.perform_later(self.email) end def update_send_grid_contact if saved_change_to_personal_year? || just_cancelled_premium_membership? || just_purchesed_premium_membership? SendGridJobs::RefreshContactListsJob.perform_later(self) else SendGridJobs::UpdateContactInfoJob.perform_later(self) end end end ``` ### SendGrid Batch Updates Via Recurrent Jobs Updates to SendGrid could happen via recurrent jobs, running every few minutes if possible, allowing us to make larger updates to SendGrid in a single API request. The downside is that the solution becomes more complex, and we'd have to start tracking when these updates should happen (user creation, updates, membership status changes, etc...). If job activity is generally within acceptable rate limit threasholds and only in a few occasions can be problematic, I'd try to stick to a simpler solution. ### Recurrent Jobs Updates #### User will_be_deleted = true Admin users can mark users to be deleted by updating the `will_be_deleted` column directly in the database. A recurrent job (every 10 min?) should take care of deleting the corresponding contacts from SendGrid. #### Expired subscribed_by_admin Admin users can "promote" users to a premium subscription with an expiration date. A recurrent job (every day?) should take care of refreshing these users contact lists in SendGrid when their subscription expires. #### Jan 1st: Numerology Recalculation For cases where we need to process a large amount of data, such as Jan 1st when users change their personal year and need to be moved to new SendGrid contact lists, we can rely on SendGrid batch update API so that we make as fewer API calls as possible. The idea would be to, every Jan 1st: 1. Remove all contacts from SendGrid, this is basically a single API call to SendGrid. This returns a job ID. 2. Use the job ID to check with SendGrid every 10 seconds whether the job is completed; 3. Once the job completes, enqueue one job per batches of 5k users in the DB to re-compute their numerology. This will allow us to leverage as many workers as we can, without backing up Sidekiq too much and making the process quicker; 1. Each job that completes recalculating the users numerology, should then proceed and re-create the SendGrid contacts for all users in the batch using the SendGrid batch API, ensuring we hit SendGrid's API as little as possible. 4. Enqueue a job to recreate # Questions 1. Do we need to continue to support the `TURN_OFF_SENDGRID_LISTS` env variable? **YES**. This can be useful in many occasions, it's a safe guard we want to keep. 2. Look into how revenuecat affects SendGrid integrations and vice-versa 3. What user information should be part of a SendGrid contact? *We only need `preferred name, full name, personal year, and locale`. all others dont treally need to be sent and can be taken off the new integration* Currently it's a pretty long list of fields: - email - full_name - preferred_name - life_path - expression - soul_urge - personality - first_cycle - birth_day - personal_year - personal_month - overbalanced_energy_1 - overbalanced_energy_2 - overbalanced_energy_3 - balanced_energy_1 - balanced_energy_2 - balanced_energy_3 - underbalanced_energy_1 - underbalanced_energy_2 - underbalanced_energy_3 - mobile_number - telnum - admin - accepted_terms - first_name - last_name - locale