# Tuning Your Instrument With Google Teachable Machine | Lesson Plan

Created by Ganning Xu ([e-mail])

<iframe width="100%" height="400" src="https://www.youtube.com/embed/8Ib7MGjTpvM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<!-- {%youtube 8Ib7MGjTpvM %} -->

## Overview
The Tuning Your Instrument with Google Teachable Machine workshop is designed to be a gentle introduction to fundamental machine learning concepts. This lesson is best used in a 45 or 90 minute lesson block. No programming experience is required to follow along and understand the workshop. A breakdown of the workshop is as follows:
1. Overview of Machine Learning (0:00)
2. Finished App Demo (2:05)
4. Training & testing the machine learning model (12:50)
5. Overview of Tools & Technologies to deploy model (24:57)
6. Building a website for everyone to use our model (31:45)
7. Conclusion (44:35)

*Students only need their computers for this workshop. They'll enter as programming novices, and walk out having built a machine learning model to tune musical instruments.*

## 90 Minute Lesson Plan
| Minutes | Plan |
| -------- | -------- |
| 5 | Introduce the workshop and demo the final product |
| 45 | Play the workshop video! |
| 30 | Questions & allow students to create extensions for the project created in the workshop (possible extension ideas shown below) |
| 10 | Students share the project extensions they've added |

## 45 Minute Lesson Plan
Play the workshop video, it's just under 45 minutes long :slightly_smiling_face:

## Important Links
- [Youtube Workshop (embedded above)](https://www.youtube.com/watch?v=8Ib7MGjTpvM)
- [Slide Deck (shown in workshop)](https://www.canva.com/design/DAFKpeC_WVI/q7dbAoBRVIJOQWdCqd4xQw/edit?utm_content=DAFKpeC_WVI&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton)
- [Live Demo](https://ganning127.github.io/Ryden-AI-TA/teachable-machine-tuning/)
- [Completed Code (replit)](https://replit.com/@GanningXu1/TuningInstrumentsML#script.js)

## Ideas for extensions on the project
1. **Train the model to be able to detect notes from specific instruments**. Currently, our model only detects notes from a generic youtube video (used for tuning). If you're able to train the model on notes specific to pianos, violins, or guitars, that would make the app more specified for a target audience
2. **Incorporate sharps and flats for each note**. Don't be limited by just A, B, C, D, and E. Go ahead and include sharps and flats to make the model more useful
3. **Incorporate differet sets of training data**. Remember how in the workshop we only trained each class on a data from a single youtube video? Well, try to find more videos for each class and add them to your model to make it more accurate in detecting!

## If students already have programming experience
- If you already have experience with Google Teachable Machine:
- skip to 24:57 to deploy our model using Replit!
- figure out how to export your model into an app using Expo or Android Studio
- If you already have `HTML` or `JS` experience:
- utilize a CSS framework like Bootstrap or Tailwind CSS to make your webpage more visually appealing!
- vary the hue of the color to corrospond with the probability of the predicted class (hint: you can do this using `rgba()` colors and varying the opacity!)

## Resources
The resources below are mentioned in the video for students to use when following along! We recommend sharing a document with students (such as on Canvas) with the `index.html` and `script.js` template below.

### `index.html` Template
Paste the following code into `index.html`

<div>Teachable Machine Audio Model</div>
<button type="button" onclick="init()">Start</button>
<div id="label-container"></div>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.3.1/dist/tf.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/speech-commands@0.4.0/dist/speech-commands.min.js"></script>

### `script.js` Template
Paste the following code into `script.js`

// more documentation available at
// https://github.com/tensorflow/tfjs-models/tree/master/speech-commands

// the link to your model provided by Teachable Machine export panel
const URL = "<Paste your model URL here>"; // for example, mine is: https://teachablemachine.withgoogle.com/models/JN32mbtrH/

async function createModel() {
    const checkpointURL = URL + "model.json"; // model topology
    const metadataURL = URL + "metadata.json"; // model metadata

    const recognizer = speechCommands.create(
        "BROWSER_FFT", // fourier transform type, not useful to change
        undefined, // speech commands vocabulary feature, not useful for your models
        checkpointURL,
        metadataURL);

    // check that model and metadata are loaded via HTTPS requests.
    await recognizer.ensureModelLoaded();

    return recognizer;
}

async function init() {
    const recognizer = await createModel();
    const classLabels = recognizer.wordLabels(); // get class labels
    const labelContainer = document.getElementById("label-container");
    for (let i = 0; i < classLabels.length; i++) {
        labelContainer.appendChild(document.createElement("div"));
    }

    // listen() takes two arguments:
    // 1. A callback function that is invoked anytime a word is recognized.
    // 2. A configuration object with adjustable fields
    recognizer.listen(result => {
        const scores = result.scores; // probability of prediction for each class
        // render the probability scores per class
        for (let i = 0; i < classLabels.length; i++) {
            const classPrediction = classLabels[i] + ": " + result.scores[i].toFixed(2);
            labelContainer.childNodes[i].innerHTML = classPrediction;
        }
    }, {
        includeSpectrogram: true, // in case listen should return result.spectrogram
        probabilityThreshold: 0.75,
        invokeCallbackOnNoiseAndUnknown: true,
        overlapFactor: 0.50 // probably want between 0.5 and 0.75. More info in README
    });

    // Stop the recognition in 5 seconds.
    // setTimeout(() => recognizer.stopListening(), 5000);
}

## If you want to use code exported by Teachable Machine
The code above was generate by Teachable Machine, but if you want to code that your model generate (should be the exact same, except for the model URL), you'll need to click the `Copy` button in the code block of JavaScript at 24:04.