---
title: Project Takaki group - Initial issue
tags: IoT, machine learning, music recommendation system
disqus: hackmd
---
# Overleaf
[論文LaTeX](https://www.overleaf.com/2857141848xwcbpmbgmxhc)

# PPT
[Presentation ppt](https://docs.google.com/presentation/d/1QroHUh1rdTWXDrRaDmVR08iQPRmZWRhBtLlz6CZx_1k/edit?usp=sharing)

# Business model
[business](https://www.strategyzer.com/business-model-examples/spotify-business-model)

[journal 1](https://journals.sagepub.com/doi/full/10.1177/1527476417741200)

Spotify itself has pointed out that ads serve a dual purpose, generating a revenue stream for the company but also prompting advertising-averse users to pay for Spotify Premium

# Data Analysis
### Current muisc recommendation dataset
- Yahoo Music Dataset:[https://www.kaggle.com/c/ee627-F18/overview](https://www.kaggle.com/c/ee627-F18/overview)
- Million Song Dataset:[https://www.kaggle.com/c/msdchallenge](https://www.kaggle.com/c/msdchallenge)
- Spotify Dataset(what we choose):[https://www.kaggle.com/mrmorj/dataset-of-songs-in-spotify](https://www.kaggle.com/mrmorj/dataset-of-songs-in-spotify)

The majority of current music recommendation dataset normally focus on features such as album data, artist data, genre, listening history (**see in Table1**), etc. However, our proposed system uses user's emotion, which is given by valence and arousal(engery) (**see in Table2 and Figure1**). According to our previous surveys, the valence-arousal method can represent user's emotion more precisely, and the dataset we choose is provided by Spotify, so we think this approach is pretty reliable.

| User ID | Song ID | Play times |
|:----------------------------------------:|:------------------:|:----------:|
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SOBONKR12A58A7A7E0 | 1 |
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SOEGIYH12A6D4FC0E3 | 1 |
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SOFLJQZ12A6D4FADA6 | 1 |
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SOHTKMO12AB01843B0 | 1 |
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SODQZCY12A6D4F9D11 | 1 |
| fd50c4007b68a3737fe052d5a4f78ce8aa117f3d | SOXLOQG12AF72A2D55 | 1 |
| d7083f5e1d50c264277d624340edaaf3dc16095b | SOUVUHC12A67020E3B | 1 |
| d7083f5e1d50c264277d624340edaaf3dc16095b | SOUQERE12A58A75633 | 1 |
| d7083f5e1d50c264277d624340edaaf3dc16095b | SOIPJAX12A8C141A2D | 1 |
| d7083f5e1d50c264277d624340edaaf3dc16095b | SOEFCDJ12AB0185FA0 | 2 |
| . . . | . . . | . . . |
#### Table1:Million Song Dataset contains user's listening history

| track_name | track_artist | energy | valence |
|:-----------------------------------------------------:|:----------------:|:------:|:-------:|
| I Don't Care (with Justin Bieber) - Loud Luxury Remix | Ed Sheeran | 0.916 | 0.518 |
| Memories - Dillon Francis Remix | Maroon 5 | 0.815 | 0.693 |
| All the Time - Don Diablo Remix | Zara Larsson | 0.931 | 0.613 |
| Call You Mine - Keanu Silva Remix | The Chainsmokers | 0.93 | 0.277 |
| Someone You Loved - Future Humans Remix | Lewis Capaldi | 0.833 | 0.725 |
| Beautiful People (feat. Khalid) - Jack Wins Remix | Ed Sheeran | 0.919 | 0.585 |
| Never Really Over - R3HAB Remix | Katy Perry | 0.856 | 0.152 |
| Post Malone (feat. RANI) - GATTÜSO Remix | Sam Feldt | 0.903 | 0.367 |
| Tough Love - Tiësto Remix / Radio Edit | Avicii | 0.935 | 0.366 |
| If I Can't Have You - Gryffin Remix | Shawn Mendes | 0.818 | 0.59 |
| ... | ... | ... | ... |

#### Table2:Spotify Songs Dataset contains the valence and the energy of songs.



#### Figure1:Our proposed system can capture user's valence and energy value, fitting in our music dataset and then recommend corresponding songs.

# First Survey and Proposal

> [name=周柏瑄, 劉瀚文, 陳威宇, 張劭群, 李國豪, 陳子新 ]

Important link as follow:
Google Meets (that we'll use in 10/16): https://meet.google.com/htp-zkqz-apj

-
(以上區域請勿亂動)

***

# Project Takaki

After the class at 10/14

From Alexa-liked architecture -> IoT music recommendation system

YHH: 認為智慧家庭太簡單，且音樂控制的居家助理在使用情境站不住腳(得想點更酷的)。

# Individual Survey

## 周柏瑄

*"I'm not the best, but I can share the experience as I can."*

Before we talk about our project, we can take a quick look at this: [Affective computing.Wiki](https://en.wikipedia.org/wiki/Affective_computing)
Another supplement: [情感三維理論](https://www.easyatm.com.tw/wiki/%E6%83%85%E6%84%9F%E4%B8%89%E7%B6%AD%E7%90%86%E8%AB%96)
### Paper #1
### Emotion Based Music Recommendation System Using Wearable Physiological(生理的) Sensors

#### Published in: IEEE Transactions on Consumer Electronics ( Volume: 64, Issue: 2, May 2018)
Link: [Emotion Based Music Recommendation System Using Wearable Physiological Sensors](https://ieeexplore.ieee.org/abstract/document/8374807)

#### Abstract:
Most of the existing music recommendation systems use collaborative(協作的) or content based recommendation engines. However, the music choice of a user is not only dependent to the historical preferences or music contents. ***But also dependent to the mood of that user***. This paper proposes an emotion based music recommendation framework that learns the emotion of a user from the signals obtained via wearable physiological sensors. In particular, the emotion of a user is classified by a wearable computing device which is integrated with a **galvanic skin response (GSR)** and **photo plethysmography (PPG)** physiological sensors. This emotion information is feed to any collaborative or content based recommendation engine as a supplementary data. Thus, existing recommendation engine performances can be increased using these data. Therefore, in this paper emotion recognition problem is considered as arousal(喚醒度) and valence(正負性，指感情方面的) prediction from multi-channel physiological signals. Experimental results are obtained on 32 subjects' GSR and PPG signal data with/out feature fusion using decision tree, random forest, **support vector machine** (See also: [支援向量機(Support Vector Machine)介紹](https://medium.com/jameslearningnote/%E8%B3%87%E6%96%99%E5%88%86%E6%9E%90-%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-%E7%AC%AC3-4%E8%AC%9B-%E6%94%AF%E6%8F%B4%E5%90%91%E9%87%8F%E6%A9%9F-support-vector-machine-%E4%BB%8B%E7%B4%B9-9c6c6925856b)) and **k-nearest neighbors algorithms** (See also: [kNN分類演算法](https://northbei.medium.com/machine-learning-knn%E5%88%86%E9%A1%9E%E6%BC%94%E7%AE%97%E6%B3%95-b3e9b5aea8df)). The results of comprehensive experiments on real data confirm the accuracy of the proposed emotion classification system that can be integrated to any recommendation engine.

### Paper #2

---

## 劉瀚文
### Paper #1

### Emotion perceived and emotion felt: same or different?

Link: [Emotion perceived and emotion felt: Same or different?](https://journals.sagepub.com/doi/abs/10.1177/10298649020050S105?casa_token=ZxI_FgTM07UAAAAA:bitHHj_nZY0p7_5yzmyoWRn4_rga97ERGEVZBlGbhVYOWz8U3BWpkKT88WpF0z0gSlX6C4wP1AXy6yM)

#### Abstract:
A distinction is made between emotion perception, that is, to perceive emotional expression in music without necessarily being affected oneself, and emotion induction, that is, listeners’ emotional response to music. This distinction is not always observed, neither in everyday conversation about emotions, nor in scientific papers. Empirical studies of emotion perception are briefly reviewed with regard to listener agreement concerning expressed emotions, followed by a selective review of empirical studies on emotional response to music. Possible relationships between emotion perception and emotional response are discussed and exemplified: positive relationship, negative relationship, no systematic relationship and no relationship. It is emphasised that both emotion perception and, especially, emotional response are dependent on an interplay between musical, personal, and situational factors. Some methodological questions and suggestions for further research are discussed.

//本文說明的情緒感知或是情緒同步，並沒有辦法明確去區別，但可以了解的是或有強烈正相關。

### Paper #2

### An Interaction-aware Attention Network for Speech Emotion Recognition in Spoken Dialogs

Link: [An Interaction-aware Attention Network for Speech Emotion Recognition in Spoken Dialogs](https://ieeexplore.ieee.org/abstract/document/8683293)

#### Abstract:
Obtaining robust speech emotion recognition (SER) in scenarios of spoken interactions is critical to the developments of next generation human-machine interface. Previous research has largely focused on performing SER by modeling each utterance of the dialog in isolation without considering the transactional and dependent nature of the human-human conversation. In this work, we propose an interaction-aware attention network (IAAN) that incorporate contextual information in the learned vocal representation through a novel attention mechanism. Our proposed method achieves 66.3% accuracy (7.9% over baseline methods) in four class emotion recognition and is also the current state-of-art recognition rates obtained on the benchmark database.

//關於語音辨識情緒(本文旨在作為人機溝通的需求，但我們可以提取他對於語音情緒辨識的概念)

----

## 11/02
### Paper #3

### Emotion Detection Algorithm Using Frontal Face Image
Link: https://www.koreascience.or.kr/article/CFKO200533239341594.page

#### Abstract:

An emotion detection algorithm using frontal facial image is presented in this paper. The algorithm is composed of three main stages: image processing stage and facial feature extraction stage, and emotion detection stage. In image processing stage, the face region and facial component is extracted by using fuzzy color filter, virtual face model, and histogram analysis method. The features for emotion detection are extracted from facial component in facial feature extraction stage. In emotion detection stage, the fuzzy classifier is adopted to recognize emotion from extracted features. It is shown by experiment results that the proposed algorithm can detect emotion well.

---

## 陳威宇

### Automatic Emotion-Based Music Classification for Supporting Intelligent IoT Applications

Link: [Automatic Emotion-Based Music Classification for Supporting Intelligent IoT Applications](https://www.mdpi.com/2079-9292/8/2/164)

#### Abstract:
With the arrival of the fourth industrial revolution, new technologies that integrate emotional intelligence into existing IoT applications are being studied. Of these technologies, emotional analysis research for providing various music services has received increasing attention in recent years. In this paper, we propose an emotion-based automatic music classification method to classify music with high accuracy according to the emotional range of people. In particular, when the new (unlearned) songs are added to a music-related IoT application, it is necessary to build mechanisms to classify them automatically based on the emotion of humans. This point is one of the practical issues for developing the applications. A survey for collecting emotional data is conducted based on the emotional model. In addition, music features are derived by discussing with the working group in a small and medium-sized enterprise. Emotion classification is carried out using **multiple regression analysis** and **support vector machine**. The experimental results show that the proposed method identifies most of induced emotions felt by music listeners and accordingly classifies music successfully. In addition, comparative analysis is performed with different classification algorithms, such as random forest, deep neural network and K-nearest neighbor, as well as support vector machine.

//還沒仔細看，但應該有相關

---

## 張劭群

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8743473&casa_token=YQ88kfc0FgAAAAAA:PZeqI3Rf6H8fNampMrPUP6Abi8CBE8nbb73iav4YdgXILifjgQ_vwA3Zy2oT-Idu_1fDqRF2ZZ4&tag=1
幫我看看行不行 回家再編輯 3Q

(柏瑄): 我先幫你重新排版了，這篇感覺比較重在人的運動姿態(不同狀態)，再加以推薦音樂

### Music Recommendation System Using Human Activity Recognition From Accelerometer(加速規) Data

#### Published in: IEEE Transactions on Consumer Electronics ( Volume: 65, Issue: 3, Aug. 2019)
Link: [Music Recommendation System Using Human Activity Recognition From Accelerometer Data](https://ieeexplore.ieee.org/document/8743473)

#### Abstract:
Music listening is a very personal and situational behavior for modern people who always carry smartphones in everyday life. Therefore, contextual information, such as user's current activity and mood state could be used to greatly improve music recommendations. In this paper, we develop a **smartphone-based mobile system** that includes two core modules for **recognizing human activities** and then accordingly recommending music. In the proposed method, a deep residual bidirectional gated recurrent neural network is applied to obtain high activity recognition accuracy from accelerometer signals on the smartphone. In order to improve the performance of ***tempo-oriented music classification***, an ensemble of dynamic classification using a long-term modulation spectrum and sequence classification using a short-term spectrogram is used. Music recommendation is performed using the relationship between the recognized human activities and the music files indexed by tempo-oriented music classification that reflects user preference models in order to achieve high user satisfaction. The results of comprehensive experiments on real data confirm the accuracy of the proposed activity-aware music recommendation framework.

***
---

## 李國豪
### Ninaus, Manuel, et al. "Increased emotional engagement in game-based learning–A machine learning approach on facial emotion detection data." Computers & Education 142 (2019): 103641.
//

#### ABSTRACT:
It is often argued that game-based learning is particularly effective because of the emotionally engaging nature of games. **We employed both automatic facial emotion detection as well as subjective ratings to evaluate emotional engagement of adult participants completing either a game-based numerical task or a non-game-based equivalent.** Using a machine learning approach on facial emotion detection data we were able to predict whether individual participants were engaged in the game-based or non-game-based task with classification accuracy significantly above chance level. Moreover, facial emotion detection as well as subjective ratings consistently indicated increased positive as well as negative emotions during game-based learning. These results substantiate that the emotionally engaging nature of games facilitates learning.

//還沒看，想說看他們怎麼分析比對實驗前後的情緒數據，或許也可以拿來當作我們自己的作品的實驗的參考嗎？

***Link:*** https://www.sciencedirect.com/science/article/pii/S0360131519301940?casa_token=7Wj9n8NzSbwAAAAA:Y6ghBlyaylslU4gcxVYlW9_kbRa-HzZrDyLvFgusCQnif5WGYV0ANiv3qv7NX-wZPIexaVO4pMNe

---

## 陳子新
### Implementation of physiological signal based emotion recognition algorithm
#### Abstract:
Emotion recognition plays an important role in depression detection. The proposed system aims at classifying emotions automatically by pre-processing the physiological signals, feature extraction followed by classification and analyzing classification accuracy. Support Vector Machine (SVM) has been used for classification because of its high recognition rates and superior performance compared to Bayesian and Regression-based classifiers. The data corresponding to eight emotion available in databases DEAP, MAHNOB-HCI has been used for training and testing the proposed system. The physiological signals namely Electromyogram (EMG), Blood Volume Pressure (BVP) and Galvanic Skin Response (GSR) from emotion Sentics database are considered. Classification accuracy of 75% has been obtained for five target emotions, namely, Joy, Grief, Anger, Hate and Reverence. An improved recognition rate of 91% has been obtained by using k-fold leave out one cross-validation to reduce the observation dependence of prediction when there is limited training data.

#### Link
[https://ieeexplore.ieee.org/document/9153878](https://ieeexplore.ieee.org/document/9153878)

### Some stuff on GitHub
1. [https://github.com/sunniee/Emotion-Classification-Ravdess](https://github.com/sunniee/Emotion-Classification-Ravdess)
2. [https://github.com/Zju-George/RealtimeFER](https://github.com/Zju-George/RealtimeFER)
3. [EFR資料級](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)
4. [EFR+](https://github.com/microsoft/FERPlus)

---

## 10.23
### 陳子新
> 我找到這個，但還沒有時間細看，或許某些部分可以拿來使用？

#### Scene-Aware Background Music Synthesis

[https://dl.acm.org/doi/10.1145/3394171.3413894](https://dl.acm.org/doi/10.1145/3394171.3413894)


大致上做法：
- input scene
- color and object detect
- emotions feature
- generate music

---
# Record

## 10.16

瀚文: 正確率的判斷法，對於處在特定情緒中的受測者，需要以正向或是負向進行提供音樂類型。(心理研究方面)
主要蒐集關於感測情緒相關論文。

應用情境:

---
//以下施工中
## 附錄
-[]
-[]

---
## Hardware

### GSR

### Camera
1. 近距離
- 書桌 等
- **Facial expression detection**
3. 大範圍
- 天花板 等
- **Body language detection**

## Software
1. user: 2-3人
2. input:
1. 情緒$\times$GSR **(like coefficient)**
2. user preferences:對應到歌曲的屬性(縮小範圍)
4. output: 2d data
5.