## Link
[Resources](https://www.spaceappschallenge.org/2023/challenges/immersed-in-the-sounds-of-space/?tab=resources&fbclid=IwAR2BdQDow15oXtKKDvgX2bHasW4lNILzj1ahPkjBbjnHNg6t22VFBfGswRY)
[Mixture2Music](https://github.com/sjhan91/Mixture2Music_Official?fbclid=IwAR27vurfw3iuuFzJDsPCAhbypFQhOFEd7ZFRiWwp1GtWycRRxCClIzgJTPI)
[Test Videos](https://drive.google.com/drive/folders/1n3Go1i1mvLaP7_7aTwe-blXgL9kOdkI7?usp=sharing)
* 有興趣說怎麼想要這樣做的

[Web testing](https://github.com/tim901231/web_testing)

[Tone schedule](https://codesandbox.io/s/tonejs-forked-37r54l?file=/src/index.js:878-1110)

## Progress

### frontend

- [ ] beautiful page (jerry) (50%)
- [ ] Circle detect (晚點)
- [ ] Synthesize basic sound
- [ ] Left/ right turn detect
- [ ] multi-channel (drawing+detect)
- [ ] tone.js playing using manual generated object array

### backend （X）

- [x] collect, analyze data format (hank)
- [ ] ffmpeg process video, send back tone.js command
- [ ] algorithm

### 方向
* 多channel: 靜態 圈圈放大，filter開啟不同的wavelength
* fly through: 3d音效，固定圈圈位置，用detect。upload板應該用同樣的alg

* 後端算影片，前端播音樂
* RGB的應該要非靜態

## sound
* [Piano keys sound](https://github.com/plemaster01/PythonPiano/tree/main/assets/notes)
* [Overtones](https://towardsdatascience.com/music-in-python-2f054deb41f4)
* [Holst: Planets (For asthetics)](https://www.youtube.com/watch?v=TyMVf5LWn7Y)
* 左邊左聲道 右邊右聲道
* 用照片偵測速度
* Finetune MusicGen? condition on music
* 先用鋼琴Key合成一段然後再加強音樂
* 多維怎麼變低維
* 用星星撞到圈圈的事件放入鋼琴音 用速度amplify
* radial sound visualization:
* p5.js

## TODO
* collect testcases (Ken)
* sensing circle, detecting incoming brightness / other wavelength's feature (賀)
* Convert the brightness/color into sound/percussion via astronify (Jerry)
* https://astronify.readthedocs.io/en/latest/astronify/Intro_Astronify_Series.html
* https://en.m.wikipedia.org/wiki/Additive_synthesis
* Synthesize the result (Ken)
* Decorate the result (maybe run on colab)(蔡、(賀))
* high dimension color representation
* [Web deploy](https://www.youtube.com/watch?v=bnCOyGaSe84)
* check model api can be called
* [js audio generation](https://tonejs.github.io/)

## Test case
https://www.youtube.com/watch?v=24cV8GYe-iw

Simple case 0:13-0:18 (星星)
Hard case 1:57-2:02 (有星雲)

## QA
* 這個任務的重點應該是將多個波長的圖片轉成聲音的技術 / video的sonification技術，還是應該專注在生成聲音的品質(音質timbre)上?
* 目前看到的資料有fly through的影片還有紅外線(多個波長)的圖片，我們應該同時展現我們的方法可以做出fly through的效果，以及應用在多波長圖片上的能力嗎?又或是則一
* 成果展現需要網站?還是做成colab能夠讓user自己方影片進來生就好

### 諮詢重點
* 互動性(使用者可以有主動)(那個圈圈已讓使用者條位置)
* 資料有雜訊
* 切換波段可以立即有明顯的不同
* Accessible to 大眾deploy出來(應用程式)
* 立體聲

## Reference

[Sonification](https://www.youtube.com/watch?v=g4lRij_xcHg)
[MusicGEN](https://huggingface.co/spaces/facebook/MusicGen)
[MusicGEN example](https://github.com/sjhan91/Mixture2Music_Official?fbclid=IwAR3B6kuxdx1Vq1cWUIVUsGvZw2qBbaulRdQPtz_wqs5-2fVaSDnZRPFJMZM)
[Music Synthesis](https://www.youtube.com/watch?v=ITk2QMjCMao)
[MixtureToMusic](https://github.com/sjhan91/Mixture2Music_Official?fbclid=IwAR3B6kuxdx1Vq1cWUIVUsGvZw2qBbaulRdQPtz_wqs5-2fVaSDnZRPFJMZM)