# HOW TO BUILD A LED FINGERBOARD AID FOR 5str BANJO
**Applied IoT 1DT305**
Freddy Tönnesen (Ft222de)
Email: [e-mail]
---
This project will display tablature for banjo songs on the banjo neck, using a addressable LED strip, and microcontroller (ESP32). It will store and send the training data via MQTT and display it on an aidafruit dashboard

Estimated time to build this project:
* Code: ca 20h
* Hardware: ca 20h

Total 40h effective work

## Objective

Clawhammer (or frailing) is a 5-string banjo picking style that originates from Africa. When Americans brought over slaves, they brought along the banjo and the style of clawhammer.
When learning to play clawhammer there are certain skill-steps that need to be conquered in order to learn to play the banjo in clawhammer style.

**1. the Bum-Diddy rhythm:** many prospecting banjo players already fail this step in frustration, because the playing-style is very special, where you pick down on the every beat with one of the finger nails in a knocking motion and up pick up, every other half beat with your thumb.

**2. the chord patterns:** normally a banjo is tuned in open G, however old-time music where often re-tuned to fit the song and the voice of the player. Therefore, there is as many different chord patterns and scales, as there is tunings. However, today most banjo players can cope with three to five different tunings (G, G-modal, C, Double C, and D tuning). Still, it is a lot to learn compared to a guitar where there is only one standard tuning.

**3. Advanced picking techniques:** this gives the player a set of tools to vary and spice up songs. This include things like slide, hammer on, pull off, drop thumb etc.

**4. The scales:** What differs a beginner banjo player from an advanced is understanding the scales up and down the neck of the banjo. This is also related to the fifth and final skill step.

**5. playing by ear:** To be able to play any song by ear and reproduce the song with nuances and improvisation, is a great tool to have when playing banjo. This is almost a must for any musician that want to perform on a stage.

By using light and color as visual aid on the neck, the prospecting banjo player can get a lot of help with everything from rhythm, chord patterns in different tunings, training aid of advanced picking techniques, learn the scales in order to understand the banjo enough to be able to learn play the banjo by ear. The goal is to try out the prototype on a five year old prospecting banjo player and hopefully mitigate some of the frustration I had, when I tried to learn clawhammer (I actually gave up the first time and picked up clawhammer many years after my first try).

## Limitations
Due to time limitations, this tutorial will only show how to build a proof of concept.

## Material

| Device | price (SEK) | Specs | Buy here: |
| -------- | -------- | -------- | -------- |
| ESP32 | 190 (3 units) | [Data sheet](https://www.electrokit.com/uploads/productfile/41015/esp32_datasheet_en.pdf) | [Link](https://www.amazon.se/XTVTX-utvecklingskort-Bluetooth-klockfrekvens-ESP32-modul/dp/B09FJZQDVW/ref=sr_1_6?crid=20M0JT45F7CKO&keywords=ESP32&qid=1656409192&sprefix=esp32%2Caps%2C90&sr=8-6)|
| OLED Screen | 84,99 | 128 x 64 px [Data sheet](https://cdn.shopify.com/s/files/1/1509/1638/files/0_96_Zoll_Display_Datenblatt_AZ-Delivery_Vertriebs_GmbH_241c4223-c03f-4530-a8c0-f9ef2575872f.pdf?v=1622442722) | [Link](https://www.amazon.se/AZDelivery-I2C-Sk%C3%A4rm-kompatibel-Raspberry-inklusive/dp/B01L9GC470/ref=pd_rhf_ee_s_pd_crcd_sccl_1_4/260-9391719-8440005?pd_rd_w=cNP9R&content-id=amzn1.sym.6ed0b2dc-5081-4711-8f72-847f56bafc2a&pf_rd_p=6ed0b2dc-5081-4711-8f72-847f56bafc2a&pf_rd_r=ZG6SS3XYE4M658SYKSZ0&pd_rd_wg=8HFL3&pd_rd_r=1ecc5b48-e861-4656-9073-750106756478&pd_rd_i=B01L9GC470&psc=1)|
| Addressable LED strip | 220,73 | 7 mm between each LED (important) | [Link](https://www.amazon.se/gp/product/B08JJ77F1S/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1)|
| 2 push buttons | --- | reused | --- |
|Battery holder | 44,90 | 4 AA batteries | [Link](https://www.kjell.com/se/produkter/el-verktyg/batterier/batterihallare/batterihallare-for-4x-aa-med-batterikontakt-p39755) |
| Banjo of your choice | --- | Discarded | --- |
| **Total price:** | **540,62** | --- | --- |

In this project an ESP32 was chosen because I already had one at home. ESP32 works like a charm with micropython, is very cheap and has all the wireless function we need for the project (WIFI + Bluetooth).
**Figure 1.** ESP32 with pinout chart

The OLED screen is perfect for chosing songs on the device and show other information.
**Figure 2.** OLED screen

For displaying the tabs, we use a addressable LED strip. The reason this specific LED strip was chosen, was because the distance between each LED is 7mm. It is the same distance as between each string, making it a perfect match for this project without a need to cut up each LED and solder them together. Instead the LED-strip will be cut up every 4th pixel.
**Figure 3.** Addressable LED strip

The last thing we need is a mock up of a 5 string banjo neck.
**Figure 4.** Banjo neck dummy

## Computer setup

This is a step by step guide for setting up the computer for the project.

**Step 1**
As stated before, we are going to code this project with micropython. We are going to use Thonny as our IDE. Follow this [link](https://thonny.org/) and click download and install the program. Thonny is compatible with Mac, Linux, and Windows.

**Step 2**
Next, we must download the micropython firmware from the micropython homepage.
Click this [link](https://micropython.org/download/) and download the latest version for the kind of microcontroller you have. Personally, I have a generic ESP32 which can be found [here](https://micropython.org/download/esp32/) and it is a recommendation to always have the latest version which at the moment of writing this is v1.19.1 (2022-06-18) .bin
Save it to a folder that you can find with ease later.

**Step 3**
We also need the driver for the USB port. We find that in the silicon labs homepage
Follow this [link](https://www.silabs.com/developers/usb-to-uart-bridge-vcp-drivers), download and install the drivers

**Step 4**
Now when everything is downloaded and installed we need to install the firmware for the microcontroller.

Connect the microcontroller to the computer, open Thonny, Run --> Select Interpreter.
Chose port, and click on install or update firmware.
Once again chose the port and select the .bin file downloaded from the micropython homepage.
Now, important to notice is that if this is done on a mac, you need to press the boot button on the microcontroller when installing the firmware.
**Figure 5.** Boot button on ESP32

## Putting everything together
This is the electrical circit for the prototype:

**Figure 6.** electrical circit for the prototype

| ESP32 pin | Display pin | LED strip | Start button | Select button |Color |
| -------- | -------- | -------- | -------- | -------- |-------- |
| 5V | VCC | Vin | | | Red |
| GND | GND | Gnd | Gnd | Gnd | Black |
| 26 | | Data in | | | Green |
| 19 | SDA | | | | Dark yellow |
| 22 | SLC | | | | Gray |
| 12 | | | | in | Yellow |
| 32 | | | in | | Orange |

The push button is just a temporary thing for prototyping. However, in the final version the ESP32 will be built into a cigarr box, containing a battery and a small LED screen for choosing songs.

The LED strip will be cut up in pieces of 4 and mounted on the banjo neck:
**Figure 7.** Wiring for the banjo. Black = ground, green = data, red = power

## Plattform

For plattform we will use Aidafruit.
Why? Because it is free, simple to use and not much to code. This saves a lot time.

Create an acount at [Aidafruit](https://adafruit.com/)

Adafruit IO supports the MQTT protocol for device communication, often known as message queue telemetry transmission. You may publish and subscribe to a feed to transmit and receive feed data using a MQTT library or client.

You can download MQTT explorer [Here](http://mqtt-explorer.com/)

The following connection details should work:
**Host:** io.adafruit.com
**Port:** 1883 or 8883 (for SSL encrypted connection)
**Username:** your Adafruit account username (see the [accounts.adafruit.com](https://accounts.adafruit.com/) page here to find yours)
**Password:** your Adafruit IO key (click the AIO Key button on a dashboard to find the key)
**Figure 8.** Here you can find your password

## The code
To better understand what we want to do, I made a flow chart of the banjo:

**Figure 9.** Flow chart of banjo training aid (not exactly as the project ended up)

To begin with the ESP32 need to import a couple of libraries.

We need to import machine, pin, Neopixel and timer.

from machine import Pin
from neopixel import NeoPixel
from machine import Timer
import machine
import Wifi
import time
import Display
import MQTT

We connected the datacable for the neopixel on pin 26 and use 73 neopixels in total. Also we want to use a start button, a control button for the menu. and start of with setting the menu parameters to zero.

#Menu parameters
in_menu = False
song_pos = 0

# Define IO
np = NeoPixel(Pin(26), 73)
start_button = Pin(32, Pin.IN)

# IO States
start_button_state = False

Next we want to define the timer intereups. When the song is loaded into the main program and the machine is ready to start the song it will notify the user by a certain ready state.

# Define Timer interrupts
beats_timer = Timer(0) #Timer
ready_timer = Timer(1) #timer for ready state
check_IO_timer = Timer(2)

Next we set the global variables

# Global variables
trainingTime = 0
songlength = 0
notesheat = []
bpm = 120
period = 31
song_playing = False
# LED colors
off = (0, 0, 0)
white = (255, 255, 255)
red = (255, 0, 0)
orange = (255, 60, 0)
yellow = (255, 255, 0)
green = (0, 255, 0)
cyan = (0, 255, 255)
blue = (0, 0, 255)
purple = (102, 0, 102)
colors = [off, white, red, orange, yellow, green, cyan, blue, purple]

Note: the colors will be defines as a array from 0 to 8 but you can add as many colors as you want. This is the colors I want to use for my project. To change color the neopixel is defining it as a number from 0 to 255 and a mix of the amount of red, green, and blue.
For example:
color = (red, green, blue)

We start the program by turning of all the pixels and defining the swiching of colors and start the program when startbutton is pressed and stop the song if it is pressed again. Also we want to check for IO changes

# Turns off all LED 
def reset_np(): 
    for i in range(73):
        np[i] = (0, 0, 0)
        np.write()

# Switch color of LED
def swich_state(led,ledcolor):
   if led < 73:
       np[led] = colors[ledcolor]


# Start button pressed
def start_button_pressed():
  if not  song_playing:
    start_music()
  else:
    print('Stopping music')
    stop_music()
    
    
    # Check for IO changes
def check_IO(timer):
  global start_button_state
  if not start_button_state and start_button.value():
      start_button_pressed()
  start_button_state = start_button.value()

Next we will build the ready state timer. The diods will play an animation when the program is idle.

# Ready state timer
readyled = 0
ready_reverse = False
def ready_state(timer): 
  global readyled
  global ready_reverse
  np[readyled] = orange

  if ready_reverse:
    np[readyled +1] = blue
  else:
    np[readyled -1] = blue
  
  if readyled == 72:
    ready_reverse = True 
    readyled -= 1    
  elif readyled == 0:
    ready_reverse = False 
    readyled += 1
  else:
    if ready_reverse:
      readyled -= 1    
    else:
      readyled += 1
  np.write()
  
def start_ready_timer():
  for i in range(73):
    np[i] = blue
  np[0] = orange
  np.write()
  ready_timer.init(period = 10, mode = Timer.PERIODIC, callback = ready_state)

Next is the song timer for every period, we check what note is going to be played and light the matching LED.

# Song timer
beat = 0 
def beats_loop(timer): 
    global beat
    leds_on = []
    for n in notesheat:
        if n[0] == beat:
            swich_state(n[1], n[2])
            leds_on.append(n[1]) 
    if len(leds_on) >0: 
        for i in range(74):
            if i not in leds_on:
                swich_state(i, 0) 
    np.write() 
    beat += 1 
    if beat > songlength: 
        beats_timer.deinit()
        global song_playing 
        song_playing = False
        reset_np()
        start_ready_timer()
        print('song ended after', songlength*period/1000, 'sek') 
        #Period = 31,25 milsec 

Now we load the song file.
The song files will be semicolon separated

#Load song file 
def load_song():
  global notesheat
  global songlength
  f = open('cripple_creek.csv')
  filecontent = f.readlines()

  for l in filecontent: 
    note = l.split(";") #Semicolon separated
    note = [int(s) for s in note]
    notesheat.append(note)

  for u in notesheat:
    if int(u[0]) > songlength:
      songlength = int(u[0])

Now we want to define in what pace the song is going to be played in. A songs pace is defined in beats per minute (bpm) and for our program every beat is devided in 16 "periods" for 120 bpm.

One period is 31,25 milliseconds long (at 120 bpm). (See chapter about converting tabs to readable format for ESP32, for more information). Therefore, the song will be compressed by 1 millisec for every 4th beat. This failure is negligible but worth notice because it is possible to code a sleep function of one millisec for every 4 beat.

def set_bpm():
  global bpm
  global period
  bpm = int(input('bpm: ')) #specify what bpm you want to train in
  period = int((60000/bpm)/16) #60000 millisec per minute 
  print(period, ' ms')

### Display
The display will concist of several screens. However, for the moment the display only consists of two screens.
Song practice and song menu.
**Figure 10.** Song practice screen
This shows when idle.

**Figure 11.** Song menu screen

This screen shows the songs that are available for practice.

def init_display():
  oled.fill(0)
  oled.text('Song practice', 10, 27)

  
  for i in range(64):
    #Draw line vertical (2 pixels wide)
    oled.pixel(0, i, 1)
    oled.pixel(1, i, 1)
    oled.pixel(126, i, 1)
    oled.pixel(127, i, 1)
    
    #Draw line horisontal (2 pixels wide)
    oled.pixel(i, 0, 1)
    oled.pixel(i, 1, 1)
    oled.pixel(i, 62, 1)
    oled.pixel(i, 63, 1)
    oled.pixel(i+64, 0, 1)
    oled.pixel(i+64, 1, 1)
    oled.pixel(i+64, 62, 1)
    oled.pixel(i+64, 63, 1)
  oled.show()

def list_songs(songs):
  oled.fill(0)
  oled.text('Songs ', 0, 0)
  posY = 15
  for song in songs:
    oled.text(song, 7, posY)
    posY += 15
  oled.show()

### Now it's time to play the song (if we pushed the start button)

def start_music():
  global song_playing 
  song_playing = True
  ready_timer.deinit()
#  check_IO_timer.deinit
  global beat
  reset_np() 
  beat = 0 
  beats_timer.init(period = period, mode = Timer.PERIODIC, callback = beats_loop) 
  print('Music starting')
load_song()    

def stop_music():
    global song_playing
    beats_timer.deinit()
    song_playing = False
    start_ready_timer()

print('To start playing music press button')
print('To change tempo press 1')
start_ready_timer()
check_IO_timer.init(period = 50, mode = Timer.PERIODIC, callback = check_IO)

Finally we will also give the option to change bpm (This is a beta version and will only be possible to if it is connected to the computer)

while not song_playing:
  if (input('') == '1'):
    set_bpm()

Next subchapter is about converting tabs into readeble code for the program to read.

## How to convert tabs into readable format for the ESP32
**Step 1. Convert tabs into .txt file**
**Figure 12.** How to go from .tef to .txt

Tablature is often downloadable in .tef format. However to get the timing right we need to convert the .tef edit into .txt format. This can be done by the program TablEdit which can be downloaded [here ](https://tabledit.com/)

Open tab in TablEdit
File --> Export --> Export ASCII --> export with the following settings
**Figure 13.** Export format

**Step 2. Convert .txt tabs into readable code**
**Figure 14.** How to convert from .txt to readable code

Now, before we start converting the tabs to readable code it is important to understand how tabs work in .txt format. Below is a image that explains the most important parts:
**Figure 15.** Explanation of tablature

**Time units**
The most important part is the time units. Every beat have 16 time units. For a song played in 120bpm (Beats per minute) this would mean that every beat is halv a second (60/120=0,5).
half a second is 500 millisec. that gives us 31,25 milliseconds per time unit (500/16). Because 1 millisec is the minimum value the code can read, we have to round off to whole millisec and therefore loose 1 millisec for every 4th beat. This can be compensated later in the code with a sleep function but for now it's such a small error that we don't need to lookin to that right now.

**Position number**
Every note on the banjo must have a number for a position. This number represent what pixel is going to be lit (because every position will have a pixel).
To simplify we use this figure:
**Figure 16.** number for each position

What differs a banjo from other instrument is the fifth string that works as a “drone string”. Therefore, you never note any thread at the 5ft string. Therefor we can give this the number 0.

Now the tablature can be translated to a readable program.
The program will have 3 values:
1. Time unit
2. position
3. color

The easiest (for now) is just to make an excel spread sheet and make time unit in first column (from 1 to infinity) and add position in column 2 and a color in column 3 (colors is explained later in the code chapter). When the song is added (a lot can be copy pasted) just erase those time units where there were no notes.

It should look something like this when done:
**Figure 17.** Excel spread sheet

Once the song is translated, save the excel as a .csv file (semicolon separated)

Once it is in a .csv format you can open it as a .txt file and it should look something like this:
**Figure 18.** Song program

This will serv as one of our song programs.

## Transmitting and presenting the data

Every time you finnish a song, we add that training time to the total training time. Traing time data is stored in a file on the ESP32 and this serves as our database. Sence we only logging one value, this will be sufficient. The training data is then sent to adafruit MQTT broker.

def save_data(data):
  with open('trainingdb.txt','w') as f:
    f.write(str(data))
    f.close()
  MQTT.send_data(str(data/60))

The dashboard is built using aidafruit.io.

**Figure 19.** Dashboard aidafruit
The data will be stored until the it is erased or the memory is full, but in the future this will be improved.

## Finalizing the design

Over all a beta mock up is done, however I will keep develop this until it is a functional banjo with functional app in "Guitarr hero style".

The project was much more complexed than the initial thought however the LED on the mock up worked out very good.

For a demonstration of the banjo neck in action see:
https://youtube.com/shorts/C1aabGLytiU?feature=share

And if you want to hear the how the song sounds for a reference, I play the song here on a regular banjo:
https://youtube.com/shorts/0ZajWHuQVRg?feature=share

In the final version, the display will be built into a banjo. However, for now it is mounted on a bread board.
**Figure 20.** Display

## Limitations

Because of time limitation, the banjo neck is only a mock-up.

In the next version the bpm will be adjusted through the display (on this version it is pre-set to 120 bpm and can only be changed by connecting the ESP32 to the computer).

Today there is only 3 songs available, but in the next version it will be possible to upload new songs through FTP.

I have not considered the power consumption because it is a practice tool and the final version will have a charger.