---
# System prepended metadata

title: Introduction to Our System NYCUKA

---

# Introduction to Our System NYCUKA
## Basic Idea
- At our school, the **duration between registration and the actual consultation day** can be extended up to **two months**.
- Our NYCUKA is a spoken-dialogue mental health system **deployed during the waiting period**, **enabling self-disclosure** of the user when the consultation takes place.
- NYCUKA also features **a dashboard accessible only to counselors**. This dashboard enables counselors to **monitor the analyzed conditions of each student derived from our system**, offering insights into their well-being.

## Architecture
- Our model was **based on Flask**, which then combine several modules as the following figure:
![image](https://hackmd.io/_uploads/B1Gz5ucvT.png)
- **API Spec** for current system can be found [here](https://hackmd.io/@heizz/rJNUlZJg6)
- **ASR Module**: Bob's **Whisper-small**, refined through **fine-tuning with a code-switch dataset and a monolingual dataset**
- **TTS Module**: Lijen's **Fastspeech 2**, **pretrained on the AISHELL-3** dataset and further **fine-tuned using our own collected dataset**
- **NLU & NLG**: Our current approach utilizing **GPT3.5-turbo** for both NLU and NLG
- **Dialogue Manager**: We employ **Amazon Lex**, with the dialogue flow **"meticulously"** designed by Mahdin♡
- **Database**: We rely on a **MySQL database** to store user information, facilitating **adaptive output generation** based on the user's specific conditions

## Flow
- **Capture user input** in either text or speech format
- If the input is in speech format, transcribe it into text
- **Determine the next module to engage**: GPT, Lex, or a **combination of both**
- **Retrieve the response** from either Lex or GPT based on the determined module
- Utilize the TTS model to **convert the response into audio**
- **Play the generated audio** and simultaneously display the text response on the webpage for user reference.

## Result
![image](https://hackmd.io/_uploads/BJE2hd5Da.png)

![image](https://hackmd.io/_uploads/HyXqn_9DT.png)


## Current Progress & Future Works

### Current Progress
- We are currently **waiting the dataset collected by counselors** for dialogue flow and emotional TTS
- We are also **waiting for the results from external teams** not affiliated with the lab 
    - Animation + Application deployment: Hippo
    - Dialogue flow: Hui Han
- Collaborating with counselors to **enhance prompt design** for more favorable GPT outputs
- **Optimize the overall architecture** to improve user experience (UX)
- **Resolve any bugs** found during the demo process

### Future Works
- **Continuation of ongoing tasks** (Of course)
- Implementation of **multilingual support** (En and Zh)
    - **Dialogue flow**: features has been enabled, so a "little" modification should be fine
    - **TTS**: can utilize pre-trained english TTS
    - **ASR**: no modification needed, current ASR already support code switching
- **Training a dedicated NLU and NLG model** using datasets from counselor to create a counseling-specific Language Model
- **Continued support for external teams** in their system with ours.
- Optimize anything you deem need to be optimized :)

### Final Words
Need someone incharge of:
- ASR
    - ASR development, providing ASR API for system
- TTS
    - TTS development, providing TTS API for system
- Web Development
    - Integrate all of the modules into one whole system, frontend (Counselor dashboard) and backend development, deployment (Backend)
- GPT/Contact window with counselor
    - Designing dialogue flow, anything related to GPT API
- But all in all, we are together in this! let's 加油!
![image](https://hackmd.io/_uploads/SJlfbt5Dp.png)
