# Building a Synthesizer in WebAssembly

###### tags: `dedication` `work` `knowledge` `unfinished` `buffer`

> :wave: Welcome! This Buffer's purpose is to document everything related to the Tim's WASM SYNTH project. To understand how to work with this and similar documents, check out [>> the Buffer Standard <<](https://hackmd.io/u9ViiQ7PR9a0_Fb1fbKGEg) for an introduction.

> In this document, I'm summarizing what I've learned from building a software synthesizer using modern web technologies.

## Introduction

Beginning 2018, I've started taking piano lessons. But even long before that, I've been fascinated by creating sounds with a computer. Shortly after my first lessons, I made the decision to buy Ableton Live to explore the world of software-synthesized music. Initially, I was quite confused by all the new terms and concepts that were presented to me.

To further my curiosity and knowledge, I decided to take a deep dive into digital music processing. Not only to learn how to use Ableton Live more effectively, but also to prove to myself that I can work in this field too. As writing a synthesizer proved to be a fairly non-trivial task, I've decided to journal this project in this document.

The next section, will detail the goals I have for creating this project.

## Goals

I have a bachelor's degree in "Computer Science and Media". In the past 5 years of my career, I've mostly focused on Computer Science (more specifically distributed systems). With this project, I'd like to take a deep dive into audio engineering to prove to myself that it's a field I can (skill-wise) and want (motivation-wise) to work in.

Additionally, I'd like to give this project a certain set of constraints such that I execute it in a controlled manner.

Hence, what I specifically would like to achieve is the following:

- I want to create a piece of software ("instrument") that allows a user to create pleasant sounds
- I want to build most of the software in C++ as it's is the "lingua franca" of audio engineering (apart from offering various other benefits...)
- I want to use modern web technologies that allow a user to play the instrument in the browser
- I want the instrument's user interface to be elegant and intuitively explorable
- I want to understand the math and engineering that is behind digital audio processing and make it accessible for a musician/user
- I don't want to use pre-made software or libraries to generate sounds, as I want to learn and understand all techniques and concepts of digital audio processing from the ground up
- Optionally, I'd like to create some weird blockchain/web/audio experiment that shows the benefits of creating instruments with modern web technologies

## Journal

This section contains all journal entries ascending in time.

### Some time in October - Nov 10, 2019

Some time in October, I started the project. I created a [GitHub repository](https://), setup emscripten and did some first experiments with C++ and WebAssembly. I was able to pass a string from C++ to JavaScript and log it in the console. Additionally, I did a lot of reading on frequencies and digital signal processing. The following sub-sections usually feature a little JavaScript demo on jsbin.com.

### Math

#### Sinus

##### Vertical Stretching

- [Good resource](https://www.sparknotes.com/math/trigonometry/graphs/section4/)
- To stretch a sinus function vertically, multiply input times a value e.g. `sin(2x)`
- This increases the amplitude of the sinus, making the output sound louder and "more or less jarring"
- [Interactive example](https://jsbin.com/likituxuji/edit?js,output)

##### Horizontal Stretching

- [Good resource](https://www.sparknotes.com/math/trigonometry/graphs/section4/) - To stretch a sinus horizontally, divide input by value e.g. sin(x/2) (doubling the size of a period)
- This increases the size of a period and hence the frequency. For the human ear, this results in hearing "high" or "low" tones
- [Interactive example](https://jsbin.com/likituxuji/edit?js,output)

##### Fun with Lasers

- By accident, I divided some constant by the input e.g. sin(2/x) - This results in a laser-like sound as the function is compressed heavily initially
- [Interactive example](https://jsbin.com/likituxuji/edit?js,output)

##### Generating Frequencies

- [One sin period is `2*Pi` long](https://www.wolframalpha.com/input/?i=plot+sin%28x%29+x+between+0+and+2*pi)
- When generating a tone, we need to pass a buffer object to JavaScript's AudioContext. Additionally, we need to specify the AudioContext's sampling rate.
- Simply put, the [sampling rate](https://en.wikipedia.org/wiki/Sampling_(signal_processing)#Sampling_rate) is the amount of discrete data points we give to an output device over time (e.g. per seconds)
- In our interactive example, the sampling rate is `96000`
- This means, to e.g. generate the A note below middle C according to [concert pitch](https://en.wikipedia.org/wiki/Concert_pitch) for 1 second, we'll have to generate 96000 data samples from the sinus function at 440 Hertz
- To simplify our problem, let's try to generate a 1 Hertz frequency with 96000 data points
- We already know that `sin(2*Pi*t)` where t = 1s. But since, we need to have 96000 data points, we'll have to cut t into 96000 pieces and compute them individually: `sin(2*Pi*i*t/frameCount)`
- We do this by splitting t into 96000 pieces (`i*t/frameCount`), and by inputting them into our wave function.
- [Interactive example](https://jsbin.com/zasunokodi/edit?html,js,output)

### Nov 11, 2019 - Nov 17, 2019

#### Creating this document

I had documented the project before directly in the GitHub repository. It seemed to me though that documenting it through hackmd.io was more fun to read and write, given that it live-renders and displays images and links directly. I, hence, removed the `/docs` folder from the repo, moved everything in here and complemented some sections that were still missing.

#### Implementing ADSR

Generating a sin wave and outputting through the browser turned out to be fairly straight forward. [A `for`-loop takes a number of relevant parameters and iterates through a buffer of the `size = time * sampleRate`](https://github.com/TimDaub/wasm-synth/commit/e7bd4822d5a73ebb8f0a3f9923be5a8f8b0dc6a0#diff-2f9184557ea427cf9e738e8d75c295deR6-R10). This, of course, creates a very boring and annoying sound of a simple sin wave. An additional issue is that the sound isn't stopping gradually, when the buffer is filled, as the sin wave's amplitude is steadily oscillating between 1 and -1. This makes for an ugly "plop!" at the end of every sound that is played through the instrument.

Naive as I was when I started this task, I thought that this "plop!" sound could be fixed by simply cutting the buffer at the last x-axis intersection of the wave. Of course, that didn't fix the problem as the wave has to become inaudible by gradually decreasing it's amplitude.


Being aware of this, I started to plot various sin waves and multiplied exponentially increasing and decreasing functions. My first tries involved using different logarithms, the Euler function and other stuff I still remembered from math class in university.
After a bit of research on the web and some more study of Ableton Live's Operator instrument, I became aware of "ADSR", which stands for [*Attack, Decay, Sustain and Retain*](https://en.wikipedia.org/wiki/Envelope_(music)#ADSR). This practice essentially describes the shaping of an amplitude according to 4 properties:


- **Attack:** The amount of time it takes for the sound to reach max amplitude
- **Decay:** The amount of time it takes for the amplitude to reach the sustain level
- **Sustain:** The level a sound is sustained for the time the key remains pressed
- **Release:** The amount of time is takes for a sound to become inaudible.

In the case of a synthesizer, these phases are usually mapped to when a key is first pressed (attack) and when it is released (release).

As I was fairly clueless on how to build something like this properly, I took to the internet to find mathematical models. On DSP Stackoverflow, I was lucky to find [this cool model on desmos.com](https://www.desmos.com/calculator/hbuxoqnnzw). It allows the adjustment of all parameters for each phase based on the x value of the sin wave. In [7aaa207672eaa222836cc2c7740a5578c2ebafd2](https://github.com/TimDaub/wasm-synth/commit/7aaa207672eaa222836cc2c7740a5578c2ebafd2#diff-2f9184557ea427cf9e738e8d75c295deR9-R47), I implemented a simple version of this by modulating the sin wave's amplitude. Using the performant JavaScript plotting library [dygraphs](http://dygraphs.com/), I was able to verify that my implementation was correct.


A next step could be to make these parameters available to the JavaScript part of the application to not only modulate amplitude, but also pitch using ADSR.

#### Implementing real-time parameter modification

Most synthesizer's allow tones to be changed while playing. This is particularly important when adjusting a synth to find a specific sound for a composition. As of commit [7aaa207672eaa222836cc2c7740a5578c2ebafd2](https://github.com/TimDaub/wasm-synth/commit/7aaa207672eaa222836cc2c7740a5578c2ebafd2), however, only statical sounds with a really huge buffer size were generated. This not only makes audio generation non-interactive, it also doesn't allow a user to find sounds easily. Hence, I'm planning the following modifications:

- Make the buffer size configurable by the user
- Implement a `render` and `update` loop that periodically fetches a buffer from C++ and outputs it to the `audioContext`
- For each call of the `render` function, similarly to how browser games work, allow the user to adjust the synthesizer's parameters by calling the `update` function
- This work most likely entails building a simple state machine within C++ to manage the parameters and on-going sound generation

#### Continuous Deployment with Travis-CI and GitHub

Currently the GitHub repository has to be downloaded manually, emscripten has to be installed and a user has to have very specific knowledge about npm. To make the synth easily playable by anyone that has a browser, we'd like to host its latest version automatically on a website. Currently, the following process is planned:

1. New push on the master branch
2. Travis-CI build is triggered and e.g. runs tests
3. If tests complete, Travis-CI pushes the build to a GitHub Pages repository that is hosted under `http://<name>.github.io`

## Resources

This section is a collection of links that I've used to build the project.

### WebAssembly

- https://emscripten.org
- https://medium.com/@tdeniffel/pragmatic-compiling-from-c-to-webassembly-a-guide-a496cc5954b8
- [How to manage memory in WebAssembly](https://marcoselvatici.github.io/WASM_tutorial/)
- [Nice example on passing a `Float32Array` back and forth between JS and
C++](https://gist.github.com/automata/5832104)
- [Cheerp is an emscripten alternative](https://www.leaningtech.com/cheerp/)
- [nbind is an alternative to embind](https://github.com/charto/nbind)

### Continous Deployment

- https://emscripten.org/docs/compiling/Travis.html

## how audio is perceived

- https://www.audiocheck.net/soundtests_nonlinear.php

### Digital Audio

#### Math and Physics

- https://en.wikipedia.org/wiki/Piano_key_frequencies
- https://www.sparknotes.com/math/trigonometry/graphs/section4/
- https://www.logicprohelp.com/forum/viewtopic.php?t=79553
- https://stackoverflow.com/questions/1073606/is-there-a-one-line-function-that-generates-a-triangle-wave/1073634#1073634
- https://www.codeproject.com/Articles/30180/Simple-Signal-Generator
- [Summing up frequencies to play chords?](http://shallowsky.com/blog/programming/python-play-chords.html)
- [Article on why we hear a clicking sound when suddenly stop outputting a sound wave](https://alemangui.github.io/blog//2015/12/26/ramp-to-value.html)
- https://www.dsprelated.com
- https://docs.cycling74.com/max5/tutorials/msp-tut/mspdigitalaudio.html
- https://www.reddit.com/r/synthesizers/comments/544gfd/lets_say_i_want_to_design_and_build_synths_as_a/
- https://docs.juce.com/master/tutorial_sine_synth.html
- [How to calculate chords (Gleichstufige Stimmung)](http://www-math.bgsu.edu/~zirbel/sound/Trigonometric%20functions%20and%20sound.pdf)
- [Gleichstufige Stimmung Wikipedia](https://de.m.wikipedia.org/wiki/Gleichstufige_Stimmung)
- [What exactly are harmonics and where do they appear?](https://electronics.stackexchange.com/questions/32310/what-exactly-are-harmonics-and-how-do-they-appear)

##### Envelopes
- https://issuu.com/petergoldsborough/docs/thesis
- https://en.wikipedia.org/wiki/Damped_sine_wave
- https://www.researchgate.net/publication/2889345_Envelope_Model_Of_Isolated_Musical_Sounds
- https://web.eecs.umich.edu/~fessler/course/100/l/l10-synth.pdf
- https://dsp.stackexchange.com/questions/2555/help-with-equations-for-exponential-adsr-envelope
- https://www.desmos.com/calculator/nduy9l2pez

#### MIDI

- https://www.midi.org/specifications/item/table-1-summary-of-midi-message
- http://www.personal.kent.edu/~sbirch/Music_Production/MP-II/MIDI/midi_protocol.htm
- [It's possible to send MIDI from Ableton Live to the Web MIDI API in OS X](https://stackoverflow.com/questions/43544357/how-to-connect-web-midi-api-to-native-application-like-ableton-live)
- [Demo of using MIDI in the browser](https://codepen.io/peteranglea/pen/KZrGxo?editors=1111)

#### Oscillators

- [How does Polyphony work](http://www.martin-finke.de/blog/articles/audio-plugins-016-polyphony/)

#### Misc

- https://www.szynalski.com/tone-generator/

#### OpenAL
- [Output sin wave in OpenAL](https://stackoverflow.com/questions/21238543/c-al-altypes-h-no-such-file-or-directory)

#### Web Audio API

- https://marcgg.com/blog/2016/11/01/javascript-audio/
- https://github.com/cwilso/midi-synth
- https://webaudiodemos.appspot.com/slides/index.html#/23
- https://www.webaudiomodules.org/
- http://zachberry.com/talks/2014/lets-build-a-synth-with-js/#24
- Using [`createBufferSource`](https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/createBufferSource)
- [How to build an oscillator in C++](http://teropa.info/blog/2016/08/04/sine-waves.html)
- https://www.keithmcmillen.com/blog/making-music-in-the-browser-web-audio-api-part-1/
- [WebAudio API implementation for Node.js detailing gain up and down ramping](https://github.com/audiojs/web-audio-api/blob/f80aaa6bfaaa41d418f3316d767ac0f88f6bc4e0/lib/AudioParam.js#L172)
- [Explaination on the relationship of Gain and Decibels in the WebAudio API](http://teropa.info/blog/2016/08/30/amplitude-and-loudness.html)
- [People are having problems with the Web Audio API](https://blog.mecheye.net/2017/09/i-dont-know-who-the-web-audio-api-is-designed-for/)