# The notion of design emergence：Evolving your software - Tan Yeong Sheng (陈永胜)

{%hackmd yEJ_rTxzTWa2K450RqmT8w %}

>

本講題將主要闡述『如何演變我們的軟體產品？』

前言

新加坡每五年，以大型計劃從教育推動產業
新加坡將學府的專注力從生物科技轉向軟體產業

[Full PDF Slide Deck](https://drive.google.com/file/d/1L2m_Vtsm0a2Luae-DZoxi5vlEgrqN0Xy/view?usp=sharing)

> "The Best Architecture requirements, and **designs emerge** from **self-organizing teams**."

## Emergent design

Software Craft and product building is Knowledge Creation + Knowledge Preservation


How many of you would like to see your software product survive beyond your career?

Products like Altavista Search had a shelf life span of about 3 to 5 yrs.
Google Search has survived for more than a decade.
Key difference between them is how the product evolved and remained relevant.

## When do you have best knowledge?

What is the most thing tool that helps you to make the timely right choice?
- The relevant and right Decision and Choice rely heavily on having as much knowledge as possible, as soon as possible, when you need to make the decision and choice.

Knowledge vs Time - Your knowledge of your software product over time
- Hypothetically it's a linear increment
- Reality: at the begining it's possible to be flat, and as it grows, you would realize there are more things to be integrated and learnt about your business domain or product market fit.
- We should strive for: (interact with your product earlier, get it into your users hands sooner) to increase the knowledge as soon as possbile, that comes from early feedback.

### Traditional vs Emergent

Traditionally, the waterfall method is like Singapore government's economic plan and how that trickles into the education system reforms and structure: upfront, comprehensive but slow, far off tangent with emerging societal needs with emerging global challenges.
There is no absolute right or wrong when a decision or choice is made. It is all about opportunity cost. Any decision made at a certain point of time, has only the benefit of just in time contextual relevancy of information available.

The emergent method (like scrum) is iterative, taking small steps every time (Inspect and Adapt). It is cheap and easy to pivot and change your solution, design and technical decisions. Use iterative tiny product development cycles to gather faster feedback sooner to better sense the market, sense the competion, sense the technology evolution.

Traditional:
- Design (with **all** decisions) $\rightarrow$ Initial development

Emergent:

Design(decide an initial direction) $\rightarrow$ Design/Code $\rightarrow$ Reflect $\rightarrow$ Design/code $\rightarrow$

### Make it work $\rightarrow$ Make it right $\rightarrow$ Make it fast

* The primitive whole isn't just a bunch of unassembled parts.
* It's an "under-developed whole" and something that does work.

> Big design up front is dumb, but doing no design up front is even dumber

When do we choose, how do we choose?

### Motivation $\rightarrow$ Skills $\rightarrow$ Timing

* Motivation:
- Is there a market for the platform
- Is there a need for the platform
* Skills
- Can you hire
- Is there a big enough community to seek help
- Example: [Slang](https://news.efinancialcareers.com/us-en/282097/slang-goldman-sachs), [A+](https://news.efinancialcareers.com/us-en/3001925/morgan-stanley-a-programming-language)

## What upfront decision are hardest to change?

I started in the semiconductor industry building equipment controllers and enjoyed controlling robots. With the semiconductor industry downturn in the early 2000 in Singapore, I was moved to pure software industry, building applications to rejuvenate legacy Mainframe finance and manufacturing execution systems. My choice back in 2002 was to equip myself with Java.
Java promised the build once, run anywhere idealogy.
With that choice, I inherited along with it, the entire ecosystem, techniques and platforms that supported the JVM stack for then-'modern' web applications like Apache Httpd, Apache Struts, XML, Oracle DB with JDBC, SOAP APIs (later replaced with Jersey for RESTful APIs), Apache Tomcat, etc.

* Programming Language
* Operating System / Processor Architecture
* Database

Choosing a programming language, operating system or database, very early on in the product development phase, anchors and determines the eco-system you have to work, deploy and debug/support in. It also dovetail you towards the available pool of talent you would hire, and the mindset of those talent pool that comes along, etc.

### Here be dragons...
Below are a bunch of strong personal opinions on techincal choices that had shaped my recent thoughts and opinions that I would like to share, which I hope, can help you alleviate the risk and cost of choosing a technology stack

### Defer choice, maintain flexibility

Compare how much complexities you would need to tackle and hire to work with the first stack of options

[Erlang](https://www.erlang.org/) developed by Joe Irving, Robert Virding and company in [Ericsson](https://en.wikipedia.org/wiki/Ericsson), to support the high demands of a robust, fault tolerant and hot deploy requirements of the Telecom market/industry. It has been measured with 9 9s ([99.9999999%](https://pragprog.com/articles/erlang)) uptime, with hot deploy capability without any downtime.

### Homogenous $\rightarrow$ Simplicity

* Erlang/Elixir gives you the option to implement initial solution exclusively with a homogenous Erlang stack
* You can then switch to alternatives, more full blown solution when enough knowledge has been acquired (e.g. through real user feedback or after data characterisation has been conducted, etc). e.g. Introduce [traefik](https://traefik.io) or [caddy](https://caddyserver.com) to replace (cowboy2)[https://ninenines.eu/articles/cowboy-2.0.0/], or replace [Mnesia](https://hackernoon.com/erlang-mnesia-the-art-of-remembering-5f6b975c7094) with [CockroachDB](https://www.cockroachlabs.com), or replace [Erlang Term Storage (ets)](http://erlang.org/doc/man/persistent_term.html) with [Aerospike](https://www.aerospike.com/products/aerospike-database-platform/).
* Homogeneous stack, single repo systems are easier to develop, test and maintain.

### Responsive interactivity w/o JS

[Phoenix Framework](https://phoenixframework.org/) with [Phoenix LiveView](https://dockyard.com/blog/2019/05/16/phoenix-liveview-leave-regular-web-development-behind), no more complex javascript frameworks needed to build an interactive and responsive web application. ([ElixirConf2019 Phoenix.LiveView Update](https://www.youtube.com/watch?v=XhNv1ikZNLs&feature=youtu.be))

Show me the code: https://github.com/yeongsheng-tan/live_view_todos
* [master branch](https://github.com/yeongsheng-tan/live_view_todos.git) (single phoenix server with Phoenix LiveView)
* [distributed_phx_pubsub branch](https://github.com/yeongsheng-tan/live_view_todos.git) (distributed phoenix server with redis pubsub as Phoenix Channels)

What is needed?

- Production code ->
* Phoenix 1.4.9 and above
* hex package: phoenix.liveview
- Look Ma, No Javascript!!!

## D.A.S.H

### Your Weakest Link

- Traditional RDBMS
- NO SQL
- New SQL (e.g. [CockroachDB](https://www.cockroachlabs.com), [YugabyteDB](https://www.yugabyte.com), [TiDB](https://pingcap.com/en/), etc)

### D.A.S.H

- beyond [12-Factor-App](https://12factor.net/)
- 4 properties of CloudNative Ready DB
* D: Disposability
* A: API symmetry
* S: Shared nothing
* H: Horizontal Scaling

### Enables global, scalable, resilient services

## Multi-master Distributed DB - Can it be trusted?

### CockroachDB passes jepsen testing

[Jepsen testing](https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-jepsen-testing/)

Show me the code: https://github.com/yeongsheng-tan/lunchbox_api
* [master branch](https://github.com/yeongsheng-tan/lunchbox_api.git) (single monolithic PostgreSQL)
* [cockroachdb_postgrex_cdb_api_jwt_auth_e2e_tests branch](https://github.com/yeongsheng-tan/lunchbox_api.git) (multi-master CockroachDB with Cypress.IO e2e tests)

### What is needed?

- Production code ->
* hex package: [postgrex_cdb](https://hexdocs.pm/postgrex_cdb/readme.html)
- Test code (to support DB sandbox and cleanup across tests) ->
* hex package: [ecto_replay_sandbox](https://hexdocs.pm/ecto_replay_sandbox/readme.html)
- No prod code change
- Test code requires some changes to fit into API of ecto_replay_sandbox

From a practical point of view.
Changeability (應變) is the only design metric that matters; code that's easy to change is well - Designed.

###### tags: `MW19`