# Deployment process improvements - quasi SaaS-ification

:::info
Status: **WiP**
> [color=#b508cc] Please add your comments more or less in this manner.
:::

# Goals
- speed-up launch time (from Go-for-it decision to having a battle-ready instance wiped and clean)
- simplify procedure both from tech side and client/business side (gathering information)
- automate where possible
:::info
**Danish**
In addition our concrete goals are:
> [color=#b508cc] Bringing the time to launch instance from on the order of couple of days to less than 10 minutes. (Not withstanding the time needed to launch the apps on their respective platforms).

> [color=#b508cc] Store state in stateful storage rather than google forms and git repositories.

> [color=#b508cc] Handle customisations dynamically on run-time rather than on build-time as much as possible
:::

:::info
**we assume spaceOS stays more or less as it is in terms of architecture (not-quite-SaaS)**
:::

# Tech perspective - status quo

## Steps of launch procedure - status quo, big picture
**[Customer service]**
- 0. Configuring Google Play / Apple Developer accounts (if there aren't any)
- 1. Gather information from the client

**[Backend team]**

- 2. Create EC2 Instance for PROD
- 3. Configure CloudFlare DNS Settings
- 4. Create S3 Bucket + set its initial state
- 5. Create Jenkins jobs
- 6. Create Config files
- 7. Create GIT branches
- 8. Add STG Stack to Staging Server
- 9. Add database for PROD in AWS RDS (database + user) & set initial state of Database (PROD and STG)
- 10. Configure Google integration
- 11. Configure Google Analytics
- 12. Configure SMTP Email settings (often done as part of step 6. )
- 13. Mailers

**[Frontend team]**

- 14. Frontend Branding

**[Mobile team]**
- 15. Implementing branding on mobile apps
- 16. Releasing apps for both stores

**[Tester/QA]**
- Soft, manual checkup if everything is in accordance with requirements

(all those steps are currently done pretty much manually)

# Change proposals for Phase #1

:::info
**disclaimer:** there's definitely a backend lean to our proposal, which we are not trying to hide :smiley:
:::

### Gathering information from the client

:::warning
**Problem:**
Gathering client requirements is currently done via Deployment form, such as this one:
[Singapore Press Holdings Depl. Form](https://docs.google.com/spreadsheets/d/16NOUF1K6cV3BlyMbf5xialGJVEfP0XIzbdsjd6_rhFk/edit#gid=1416116433)

There are three things wrong with the form:
- it - mostly -cannot be directly mapped to config values of our instances
- has a tendency to get messy, (erm... not to mention the looks)
- since it's not linked to any structurized data source, it can't be really used for automation
:::

:::success
**Change proposal:**

a minimalist Rails instance will be created (hereinafter referred to as **Automator**) to:
- provide a form, validated, with intention to be used by our Customer service (Phase #1), and Client representatives (Phase #2)
- provide a database layer, to hold all relevant instance metadata in one place (finally)
- provide a basis for automation scripts that would eventually produce subsequent instances for new clients

**why Rails and not just a bunch of shell/py scripts?**
- cause we'd really like to have a DB layer to it and have all data in 1 place instead of having them scattered across 20+ spreadsheets.
- and be able to produce a GUI for those like Oksana, and Weronika
- and have scripts in one damn place
- and (be able to quickly leverage powers of Rails should we wish it (sending emails, Slack integration, async jobs, controlling via API etc., if not now, then sometime in the future)
:::

:::info
**Danish**
The user experience and expectation is wrong. We need something that acts like a SAAS even if it isn't one:

> [color=#b508cc] The form needs to be completely handled transparently of Customer Services Representative. That means it must be designed to be self explantory and people should be able to self-signup by just going to spaceos.io.

> [color=#b508cc] It must be able to handle setting an instance up as a trial(demo). Which means upgrading a system from a trial to a paid instance should be possible.

> [color=#b508cc] All the settings defined in the automator should be changeable later in the running system itself.
:::

==(No changes in main API codebase required)==

### Create EC2 Instance for PROD
(right now - done manually)

:::success
**Change proposal:**
- There's an AWS Image of production instance which serves as a base for all subsequent releases
- We'd use Ruby AWS gem, AWS CLI, to create a new EC2 instance in AWS
- (Script would be built on top of Automator).
:::

::: info
**Danish**
> [color=#b508cc] Why not use Kubernetes or Container Engine? We are already deploying containers. We can save a lot of money on underutilized instances by just switching to the container engine or kubernetes.
:::

==(No changes in main API codebase required)==

### Configure CloudFlare DNS Settings

(right now - done manually)

:::success
**Change proposal:**
- We'd use Vanilla HTTP requests and CloudFlare REST API to create DNS entries for PROD and STG instances
- (Script would be built on top of Automator).
:::

==(No changes in main API codebase required)==

### Create S3 Bucket + set its initial state
(right now - done manually)

:::success
**Change proposal:**
- We'd use Ruby AWS gem, AWS CLI, to create a new S3 bucket, and to fill it with initial data.
- (Script would be built on top of Automator).
:::

::: info
**Danish**
> [color=#b508cc] We should use just one bucket with subfolders for different deployments. Different buckets doesn't give us any benefit while just increasing the complexity.
> > [color=blue] JK: noted. I'm all for it. :+1:
:::

### Creating Jenkins jobs
the following jobs are created
- `wms_web_build-<new_instance_name>`
- `wms_web_release-<new_instance_name>`
- `wms_core_release-<new_instance_name>`
- `wms_web_release-stg-<new_instance_name>`
- `wms_core_release-stg-<new_instance_name>`
the following jobs are modified
- `wms_core_build-stg` (has to be ammended with `wms_core_release-stg-<new_instance_name>` job reference )
- `wms_web_build-stg` (has to be ammended with `wms_web_release-stg-<new_instance_name>` job reference )

:::success
**Change proposal:**
- We'd make a template of each of those jenkins job as a XML ERB template, then fill it with instance/client specific information from the form, and push it to Jenkins via its REST API
- (Script would be built on top of Automator).
:::

### Create Config files (config.yml, nginx.conf)
(right now - done manually)
:::success
**Change proposal:**
- Similarly, we'd also make templates for that files, and then send them over to the right GIT branches.
- (Script would be built on top of Automator).
:::

### Creating GIT branches
the following branches are created in `docker` repository (holding config):
- `<new_instance_name>`,
- `dev-<new_instance_name>`

:::success
**Change proposal:**
- Script would create those branches locally via the shell, and then push it to remote git server
- (Script would be built on top of Automator).
:::

::: info
**Danish**
> [color=#b508cc] Let's move the state into the db as much as we can. Keeping it in repos is too finicky and prone to abuse.
> > [color=blue] **JK:** Couldn't agree with you more. I'd definitely like to have all this damn config to be in the DB ASAP. It is only a matter of deciding what goes into Phase 1, and what can to be done later.
:::

### Add STG Stack to Staging Server
:::success
**Change proposal:**
We'd launch a shell script on staging server that:
- uploads STG docker-compose.yml
- creates and launches STG Stack
(Script would be built on top of Automator).
:::

::: info
**Danish**
> [color=#b508cc] Why do we need a special docker compose for staging?
> > [color=blue] **JK:** There are some **minor** differences between STG stack and PROD stack. E.g. in STG, Postgres is Dockerized, whereas in PROD it is not. If we are turning it all upside down one way or another we probably can have STG db's in Amazon, similarly to PRODs.
:::

### Add database for PROD in AWS RDS (database + user) & set initial state of Database (PROD and STG)
:::success
**Change proposal:**
- Automator would connect to remote PROD Postgres Server, and:
- create new USER and (PROD) DATABASE
- seed DB with initial state taken from Automator (raw SQL)
(w/o using `backup` container, just by pumping up init dump directly to DB)
:::

### Configure Google integration
- adding callback urls and such to Google Cloud project settings UI manually (I've found no API for that)

::: info
**Danish**
> [color=#b508cc] What do we need it for? Google Cal? **JK: In short: yes**
> Why isn't it just a setting that can be changed in the UI after launch? **JK: It can be set after launch, alright - but not in spaceOS - you have to register callback URLs in Google interface.**
> What about Outlook/Office365 Cal? **JK: In spring '18 our colleague who's no longer here added Outlook/Office365 Cal integration to the backend; due to some changes in priorities and what not, it was never implemented in the Frontend and/or properly tested. As it is now - forgotten piece of code that we left hoping we'd come back to it one day.**
:::

### Configure Google Analytics
- setting tracker ID manually (AFAIK Google Tracking ID cannot be generated with API)

> [color=#b508cc] @Danish, can Google Tag Manager help us in any way in that regard?

@TODO check: https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/profiles/insert

### Configure SMTP Email settings

:::warning
**Problem:**
A common problem is that clients are somewhat slow in passing their credentials, and account settings, which slows down the process. It happened several times that provided data were lacking, and inaccurate causing additional time to be invested in troubleshooting problems which were not of our making.

It is also a habit of certain customers of ours (I won't be mentioning any names :) to change passwords for those emails (or disabling Gmail `less_secure_mode`) without any prior warning, causing spaceOS to cease sending emails properly.
:::

:::success
**Change proposal:**
By default we should ship spaceOS for every new customer with SMTP settings of a new generated account in our (sub)domain, to speed-up the process. Only when they are ready and willing we'd change it to their designated email.

**Option #1** (more elegant):
- We create a mail server for new spaceOS instances/clients (either crude Dockerized email server, or based on **Amazon SES**)
- once there is a new deployment, a new email account is created such as `<new_instance_name>@clients.spaceos.io`
- **Amazon SES** seems to be a nice option since we'd be getting 62.000 emails/month for free, and $0.10 for every subsequent 1k of emails. In that way self-hosted email server makes little sense.

**Option #2** (somewhat crude) :
- For new instances, by default SMTP settings are fixed to use one account, say `[e-mail]`
- Sent emails are either not saved at all, or erased frequently
- instances have different `reply_to` addresses set in the config.
:::

::: info
**Danish**
> [color=#b508cc] Are these emails the kind that someone needs to respond to?
> > [color=blue] JK: No, not really. Confirmations of bookings, buying stuff in a cafe/shop, reminders, password reset emails - stuff like that. Clicking a link in the body is a maximum of interaction required.
:::

### Mailers
:::warning
**Problem:**
Right now some of the data of clients (address, social account addresses are hardcoded in the footers
:::

:::success
**Change proposal:**
Config would be ehanced so that it'd be possible to declare those values (and header img) in Deployment form.
They'd be then included in the instance config allowing email templates to be fully dynamic

==**That will require some changes to API;**==
**prereqisites: Config (enhanced) in the DB**

:::

### Web Branding (TBC)
**(@TODO: consult with Web devs)**

:::warning
**Problem:**
Right now web is built with parameter which indicates client branding. Branding is tailored to each client and then chosen in build time.
:::

:::success
**Change proposal:**
- Web would be built without branding
- Backend would hold color palette in the config. Pallette would be defined in the form, by Customer Service (Phase #1) and/or Client (Phase #2)
- Backend would serve common images/assets (header, icons etc.) on constant url's (Web wouldn't even know concept of client)
- Backend would serve a (dynamically prep'd) css file with color definitions reflecting color pallette of client's choosing (**TBC**)

==**That will require changes to API;**==
**prereqisites: 1) Config (enhanced) in the DB, 2) color palette in the config**
:::

::: info
**Danish**
> [color=#b508cc] No need to serve different files from backend. Backend can remain "dumb" w.r.t css needed. Frontend can request the file it needs based on the settings.
> > [color=blue] That's pretty much what I was trying to convey.
:::

# Desired flow

- Customer Success Person gets a green light
- Fills the (new) Deployment form provided by Automator
- Backend dev verifies if form data are complete and correct (apart from form validation).
- (Phase #1) Backend dev can launch all subsequent steps of creating instance with a few clicks (each step initiated with a single click, at least in Phase #1)
- Mobile app flow remains the same

::: info
**Danish**
> [color=#b508cc] This desired flow is too operations heavy. We should remove any need for human intervention here.
:::

# Change proposals for Phase #2

- Automator, apart from having forms for **Customer Service** (enhanced, more complete) would have forms for Customer (somewhat limited, secured, maybe more straight-forward in terms of GUI)

- Backend config would be migrated from YAML files to DB. Dynamic changes of the config (such as changing color palette) would be possible without restarting machine.

# Topics for further study:

- would we be better off with Jenkins Pipelines, and JenkinsFile(s) [About Jenkinsfile](https://jenkins.io/doc/book/pipeline/jenkinsfile/)?

- Building with Jenkins is kinda slow. Should we add workers or consider some Cloud-based CI tool

- Is AWS preventing subfolders in S3? What would be the consequences of it.

## Task list

### Those that can be done without touching backend API codebase

- building Automator (DB for config, forms for client and customer service person,
- automation scripts:
- init new DB
- create git branches
- create jenkins jobs
- configure Cloudflare DNS

### Those that require changes in backend API codebase

- transition to config-from-DB
- making email templates fully dynamic
- migration to common S3? (copy stuff each time from scratch, or have it separated into common assets and per-client assets?)

## Questions & Answers

::: info
**Danish**
> [color=#b508cc] What I am also missing here is the flow for upgrades. How do we plan on keeping all the instances with latest code?
> >[color=blue] **JK:** IMHO, it should not be a big problem as far as backend and web are concerned. We could easily do it now, should there be a decision to do it. Since every release has its own jenkins jobs, We'd create an umbrella Jenkins job that basically does one thing only - fires multiple others that do the per-client-upgrade. Since the setup isn't ideal, it would have some drawbacks - update for some instance could fail - but it'd get the job done for now.
>
> > [color=blue] Mobile apps can have a hard time though. Since we're operating across multiple Google/Apple Dev accounts belonging to our clients it's difficult to coordinate multiple updates in one time.

:::