# Questions from Twilio

[TOC]

We're currently discussing with Twilio and they have questions regarding our tech. If you could provide simple answers (some sentences, not a long story) or pointers to documentation that would be great. Meeting with Twilio is scheduled at 6pm, 22th of October (tomorrow ;-)

## 1. Please provide an overview of the technology stack and how it all works together:
### a. Product Architecture
Some of our APIs documentation: https://allo-media.github.io/live-api-doc/

Each product needs a "pipeline" out of some of our platform's sub-systems:
- **Connect:** how the audio enters our platform
- VoIP (phone calls, MRCPv2)

Infra with 3 tiers:
- ISBC: to interconnect VoIP providers (SFR, voxbeam, voxbone, COLT)
- SCR: to route calls into infra (routing all sip, cdr)
- ASBC + proxy: to interconnect our customers (SIP, MRCPv2, SIPREC)

MRCPv2:
- Unimrcp + our own module (rust)
- Websocket streams
- SFTP
- **Enrich:** how conversations are qualified, transcribed, and enhanced with metadata
- Transcription
- Transformation (text2num, punctuation, etc…)
- Named entities recognition
- Redaction/Anonymization
- web navigation prior to phone call (call tracking)
- classification by theme (lead scoring, customer effort, quality monitoring...)
- **Activate:** how we can restitute enriched data to the customer
- Rest API
- Third-parties integrations (google analytics, salesforce, callminer, viavoo...)
- UI for users to dig into data (Scribr) and export them

Each subsystem is made of scalable, loosely coupled and fine grained services connected to an event bus (Event Driven Architecture). See blog post here: https://www.allo-media.net/en/tech/architecture/2020/02/24/eda-architecture.html

### b. Cloud Infrastructure/Design
#### IP Network

- 2 datacenters hosted by OVH (separated by 500kms)
- OSPF between each DCs
- Infra is active / active
- VPN IPSEC interco with our customers (static route or BGP)
- VPN OpenVPN for admins connections
- DMZ for publics servers

### c. Data stores and data pipelines
- PostgreSQL
- Elasticsearch
- Redis
- RabbitMQ

Some small systems are using MySQL.

### d. UI / API design and key technologies used

Our tech stack is described here: https://stackshare.io/companies/allo-media

You can get the API documentation here: https://allo-media.github.io/live-api-doc/

## 2. Data overview

### a. What languages do you support?

French, English, Spanish (baseline)

## b. What public data do you use for model training?

French = none
English = parts of Fisher & Switchboard
Spanish = Fisher & Callhome

## c. What proprietary data do you use for model training?

French = Manually transcribed data from our customers (around 1k hours)
English = same (but only 60 hours)
Spanish = none

## d. What type of agreements do you have in place with customers to use their data for model training?

## 3. Architecture and technology

### a. What is the tenancy model?

We currently use a multi-tenant model. However we are planning to create a dedicated tenant for customers who require PCI DSS compliance.

### b. What tools does your dev team use - how and why?

https://stackshare.io/companies/allo-media

Frontend development:
- Elm
- Vue.js

Backend development:
- Python (web, datascience, NLP)
- Elixir (Massive and scalable Network I/O, live API using websockets)
- Rust (high performance and critical components)
- C++ (legacy or external code integration)
- Java (legacy code)
- PHP (legacy code)
- Nodejs (managing many simultaneous requests)

Devops:
- Puppet
- Ansible
- Docker

## 4. Software development lifecycle -
### a. Description of the software development and delivery process

We follow an agile methodology with two weeks sprints. A sprint starts with a sprint planning on the first monday and ends with a 1h company-wide demo and a retrospective between team members. We do three synchronous "standup meeting" during the week and two asynchronous (i.e in a written form).

We use Clubhouse for project management, with stories to track small features and epics for larger cross-project features.

A story starts in the backlog, it is then moved to a "Ready for dev" status after sprint planning. The developer moves the story to the "In development" status when working on the story. They create a pull request in Github with a link to the story and move the story to "Ready for review" status. After the pull request is reviewed and validated, the story is merged and automatically deployed to a staging environment as part of the CI build process. The product manager can then validate the story and move it to "Ready for deploy". Depending on the criticity of the story, we can then either deploy it to production immediately or deploy a set of stories alltogether.

We write blueprints before implementing any feature. The feature is discussed on the blueprint (Github PRs) and once the Blueprint is ready, it goes into our backlog for next sprints.

### b. How does the software process work for small / medium / large projects?

## 5. Deployment / Production
### a. Infrastructure / Security
- Nextcloud : /Documents Technique/Clients/PAQS/AlloMedia PAQS.pdf
- Wiki : [Doc Exploitation IT](https://allowiki.allo-media.net/books/sysadmin/page/document-dexploitation-%28en%29)
### b. How does code get to production, what’s the process?

All code is deployed automatically to a staging environment which is a replica of the production environment (without customer data). It is then tested and after being successfully verified, can be deployed to production by doing a push to the `production` branch in GitHub.

The `production` branch is then build in CI, and we then create a docker image and store it in Docker Hub. The image is then deployed in the following ten minutes to the production environment. For customer-facing applications or critical internal services, we do a zero downtime rolling deployment. For internal services we can update the service in place as most of them are asynchronous services relying on a message broker (RabbitMQ) through our event-driven architecture.

### c. How are you planning to handle failure? [ backup / redundancy]
We try to avoid failure in the first place and to ensure high availability through redundancy, both at the infrastructure and software level.
Most services are deployed in two geographically distinct datacenters, the deployment is automated through Ansible playbooks and all applications can be deployed in a matter of minutes on a newly created environment.

Discuss
* monitoring
* install and recovery automation
* Zendesk?
* backup policy
* …
## 6. Discuss near-term and long-term R&D roadmap
## 7. Discuss developed intellectual property and licensed software

## 8. Decoder and NLU stack overview, training pipelines, and key technologies used.

ASR = kaldi + promising experiments on wav2letter for E2E ASR
punctuation = seq2seq transformer models (pytorch)
NER = CRF in sklearn + promising experiments on flair (https://github.com/flairNLP/flair)
Call classif = MLP in sklearn

## 9. Security / Incident Management
### a. Provide details on the last incident and explain how you solved
### b. Talk through a resolved security incident and explain