> Reference:
> [1. RASA Your Own Website — Websocket Channel & JWT Authentication](https://rasa.com/docs/rasa/connectors/your-own-website/#websocket-channel)
> [2. RASA Open Source HTTP API — JWT Based Auth](https://rasa.com/docs/rasa/http-api/#jwt-based-auth)
> [3. PyJWT](https://pyjwt.readthedocs.io/en/latest/)
> [4. API Reference — python-socketio documentation](https://python-socketio.readthedocs.io/en/latest/api.html#socketio.Client.connect)
> [5. 基于 rasa 的语音助手搭建](https://dustyposa.github.io/posts/d7e97916/)
# Virtual Therapist
## 架構圖
前端透過RASA框架提供的socket io channel，以proto定義的clientMessageRequest傳送資料，包含語音、臉部圖片、臉部關鍵點（facial landmarks）等等數據。
## 指令
* docker ps
`sudo docker ps -a --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}"`
## 啟動
### mh-chat
1. **製作對話系統 images**
```
$ cd /home/wmlab/virtual_therapist/mh-chat
$ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build kafka1 chatbot-actions chatbot-core
```
2. **啟動聊天系統**
```
$ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml up -d
```
* `-d`：run in background
確認 containers 正常執行： (init_kafka 創完 topic 就會 Exited)
```
$ sudo docker ps -a
```
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
70e2def9e357 confluentinc/cp-kafka:6.1.1 "/bin/sh -c '\n# bloc…" 39 minutes ago Exited (0) 38 minutes ago init_kafka
e25f4b8e3dba confluentinc/cp-kafka:6.1.1 "/etc/confluent/dock…" 39 minutes ago Up 39 minutes 9092/tcp, 0.0.0.0:19092->19092/tcp, :::19092->19092/tcp kafka1
6afeaa097cfb confluentinc/cp-zookeeper:6.1.1 "/etc/confluent/dock…" 39 minutes ago Up 39 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp zookeeper
8ef0bd9b31a7 registry.cn-shanghai.aliyuncs.com/goxy/chatbot-actions:ebc656b8953e8041e52a11b274ddede8f08d308a "./entrypoint.sh sta…" 39 minutes ago Up 39 minutes 0.0.0.0:5055->5055/tcp, :::5055->5055/tcp mh-chat_chatbot-actions_1
24ecc20e96a5 rasa/duckling "duckling-example-ex…" 39 minutes ago Up 39 minutes 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp mh-chat_duckling_1
02bf14c81b58 registry.cn-shanghai.aliyuncs.com/goxy/chatbot-core:ebc656b8953e8041e52a11b274ddede8f08d308a "rasa x --domain dom…" 39 minutes ago Up 39 minutes 0.0.0.0:5002->5002/tcp, :::5002->5002/tcp, 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp mh-chat_chatbot-core_1
2e739181b675 mongo:latest "docker-entrypoint.s…" 39 minutes ago Up 39 minutes 0.0.0.0:27018->27017/tcp, :::27018->27017/tcp mongodb
```
確認 topic (messaging & datasync) 成功被建立：
> Reference: [Docker compose create kafka topics](https://stackoverflow.com/questions/64865361/docker-compose-create-kafka-topics)
```
$ sudo docker logs init_kafka
```
會出現類似下面的 log：
```
[2022-05-27 04:29:21,771] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-05-27 04:29:21,881] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-05-27 04:29:22,082] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-05-27 04:29:22,384] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Creating kafka topics
Created topic messaging.
Created topic datasync.
Successfully created the following topics:
datasync
messaging
```
(:warning: 一定要執行！) 確認 kafka 正常運作：
> Reference: [Why Can't I Connect to Kafka? - Confluent](https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/)
```
$ cd /home/wmlab/virtual_therapist/kafka-listeners/python
$ sudo docker build -t python_kafka_test_client .
$ sudo docker run --network=mh_chat_net --rm --name python_kafka_test_client --tty python_kafka_test_client kafka1:9092
```
```
🥾 bootstrap server: kafka1:9092
✅ Connected to bootstrap server(kafka1:9092) and it returned metadata for brokers as follows:
{0: BrokerMetadata(0, kafka1:9092)}
---------------------
ℹ️ This step just confirms that the bootstrap connection was successful.
ℹ️ For the consumer to work your client will also need to be able to resolve the broker(s) returned
in the metadata above.
ℹ️ If the host(s) shown are not accessible from where your client is running you need to change
your advertised.listener configuration on the Kafka broker(s).
<Producing>
✅ 📬 Message delivered: "foo / 2022-05-27 05:39:41" to test_topic [partition 0]
<Consuming>
✅ 💌 Message received: "foo / 2022-05-27 05:39:41" from topic test_topic
```
### 後臺修改
* /home/wmlab/virtual_therapist/mh-chat/Dockerfile
* Dockerfile 41 行 根據官方更改：https://python-poetry.org/docs/
```py=4
RUN chmod 777 /tmp
```
```py=41
# install poetry - respects $POETRY_VERSION & $POETRY_HOME
# RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
# The get-poetry.py installer has been deprecated and removed.
RUN curl -sSL https://install.python-poetry.org | python3 -
```
* /home/wmlab/virtual_therapist/mh-chat/requirements/**senta.txt**
```yml
numpy==1.23.5
```
* virtual_therapist/mh-chat/**docker-compose.yml**
* zoo1 和 kafka1 新增 user（權限問題）
```yml
user: root
```
* mongodb 改寫
```yml
mongodb:
image: mongo:latest
container_name: "mongodb"
# restart: always
user: root
networks:
mh_chat_net:
ipv4_address: 172.27.0.8
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/data/logs
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=ZMJPh5EliBgnM765
volumes:
- type: bind
source: /home/wmlab/HDD_4T/mh_chat_mongo_data
target: /data/db
- type: bind
source: /home/wmlab/HDD_4T/mh_chat_mongo_log
target: /data/logs
ports:
- 27018:27017
command: sh -c 'chmod 777 -R /tmp ; mongod --bind_ip_all --port 27017 --auth '
# 設定 mongo 資料夾權限
security_opt:
- "seccomp:unconfined"
```
### backend
1. **製作 face recognition image**
> Source: [ageitgey/face_recognition](https://github.com/ageitgey/face_recognition)
```
$ cd /home/wmlab/virtual_therapist/face_recognition
$ sudo docker build -t gao_wan/face_recognition .
```
* `-t`：add tag for image
測試 face recognition image：
```
$ sudo docker-compose up
```
```
Creating face_recognition ... done
Attaching to face_recognition
face_recognition | I found 1 face(s) in this photograph.
face_recognition | A face is located at pixel location Top: 235, Left: 428, Bottom: 518, Right: 712
face_recognition exited with code 0
```
2. **製作後端 images**
```
$ cd /home/wmlab/virtual_therapist/backend
$ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver-base
$ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver receiver-grpc
```
3. **啟動後端**
```
$ sudo docker-compose up -d mongodb receiver receiver-grpc
```
確認 containers 正常執行：
```
$ sudo docker ps -a
```
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
02c917fc909f registry.cn-shanghai.aliyuncs.com/goxy/receiver:1.0.0 "/bin/sh -c './docke…" 4 minutes ago Up 3 minutes backend_receiver_1
0de50e1152e5 mongo:latest "docker-entrypoint.s…" 4 minutes ago Up 3 minutes 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp backend_mongodb_1
89328cd138c0 registry.cn-shanghai.aliyuncs.com/goxy/receiver-grpc:1.0.0 "/bin/sh -c './docke…" 4 minutes ago Up 3 minutes 0.0.0.0:7777->7777/tcp, :::7777->7777/tcp backend_receiver-grpc_1
```
確認 backend receiver 與 kafka 連接成功：
```
$ sudo docker logs backend_receiver_1
```
```
Device status cuda
+ƒaµS† v1.10.4+------------------------------------------+
| id | faust-consumer |
| transport | [URL('kafka://kafka1:9092')] |
| store | memory: |
| web | http://localhost:6066/ |
| log | -stderr- (warn) |
| pid | 7 |
| hostname | 5d4df30f9d55 |
| platform | CPython 3.8.10 (Linux x86_64) |
| drivers | |
| transport | aiokafka=1.1.6 |
| web | aiohttp=3.8.1 |
| datadir | /app/faust-consumer-data |
| appdir | /app/faust-consumer-data/v1 |
+-------------+------------------------------------------+
OK ^
```
### 後臺修改
* /home/wmlab/virtual_therapist/**face_recognition**/docker-compose.yml
* 新增 user（權限問題）
`user: root`
* /home/wmlab/virtual_therapist/**face_recognition**/Dockerfile
* 新增
```py=8
RUN chmod 777 /tmp
```
* /home/wmlab/virtual_therapist/**backend**/docker-compose.yml
* mh-chat 提到的 mongodb 改寫
```yml
mongodb:
networks:
mh_chat_net:
ipv4_address: 172.27.0.11
image: mongo:latest
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: rootpassword
ports:
- 27017:27017
command: sh -c 'chmod 777 -R /tmp ; mongod --bind_ip_all --port 27017 --auth '
volumes:
- /home/wmlab/HDD_4T/backend_mongo_data:/data/db
# - mongodb_data:/data/db
```
* /home/wmlab/virtual_therapist/**backend**/Dockerfile
* 新增
```py=8
RUN chmod 777 /tmp
```
* /home/wmlab/virtual_therapist/**backend**/docker/base/Dockerfile
* 改寫
```py=44
# RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
RUN curl -sSL https://install.python-poetry.org | python3 -
RUN chmod 777 /tmp
```
* /home/wmlab/virtual_therapist/**backend**/docker-entrypoint.sh
```py
#!/bin/sh
set -e
# activate our virtual environment here
. $VENV_PATH/bin/activate
# You can put other setup logic here
for dir in /app/pkgs/*/
do
export PYTHONPATH="${PYTHONPATH}:${dir}"
done
# Evaluating passed command:
exec "$@"
```
---
## 資料庫
可以用 [MongoDB Compass](https://www.mongodb.com/products/compass) 存取資料庫
### 對話紀錄
帳密：登入請至實驗室 wiki 查詢
http://140.115.54.26:8787/
#### ==rasa.conversations==
對話紀錄，包含每一題的++意圖識別++、++speech to text結果++與++多模態情緒識別結果++等等資訊
### 受測者資料&多模態數據
帳密：登入請至實驗室 wiki 查詢
http://140.115.54.26:8787/
#### ==therapist.Subject==
受測者資料，前端emit `session_request` 時帶上的資料(字典)
```python
# Python example
data = {
"name": "高丸",
"birthday": "1998-08-13",
"gender": "male",
"depression_level": 5
# 可任意新增欄位
}
sio.emit('session_request', data)
```
後端會帶入時間戳，若有完成對話會有結束時間(end_timestamp)，否則結束時間為null
#### ==therapist.ExpressionMessageList==
臉部、心率與眼動的情緒識別結果
#### ==therapist.HeartRateMessageWithSession==
估測的心率
#### ==therapist.FixationMessage==
Unity ARkit捕捉的眼動數據
#### ==therapist.features==
每個問答提取的五個模態特徵
## 語音&影片
前端傳送的語音、臉部圖片與影片(從圖片轉檔的)會保存在server的4T硬碟
路徑為：`/home/wmlab/HDD_4T/virtualtherapists/storage/{session_id}`
```
.
├── audio # 語音
├── face_cropped # dlib人臉識別切出來的人臉圖片
└── video # 原圖&從圖片轉成的影片
```
## 導出多模態特徵
1. 導出問答的原始數據
```
$ cd /home/wmlab/virtual_therapist/export
$ bash run.sh
```
2. `backend/docker-compose.yml` 設定 `receiver-feature` 的環境變數：`FEATURE_SID`
```yml=31
receiver-feature:
networks:
mh_chat_net:
ipv4_address: 172.27.0.12
image: ${DOCKER_REGISTRY_SHANGHAI}/receiver-feature:${DOCKER_TAG}
runtime: nvidia
volumes:
- '/home/wmlab/HDD_4T/virtualtherapists/storage:/app/feature_extraction/storage/data'
- '/home/wmlab/HDD_4T/virtualtherapists/storage/raw_data:/app/feature_extraction/storage/raw_data'
environment:
RECEIVER_DB: mongodb://root:rootpassword@172.27.0.11:27017/
FEATURE_DATA_PATH: /app/feature_extraction/storage
FEATURE_SID: {{ session_id }}
```
3. 導出五個模態特徵
```
$ cd /home/wmlab/virtual_therapist/backend
$ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver-feature
$ sudo docker-compose up -d receiver-feature
```
4. 查看log
```
$ sudo docker logs backend_receiver-feature_1
```
```
/opt/venv/lib/python3.8/site-packages/scipy/signal/spectral.py:1961: UserWarning: nperseg = 256 is greater than input length = 52, using nperseg = 52
warnings.warn('nperseg = {0:d} is greater than input length '
2022-07-28 05:09:04,938 [INFO] __main__: {'sid': 'a729718a-1e15-4c32-80ac-035d98b13363', 'question_id': 'depression_parents_treat_negative', 'text': '不知道', 'audio_id': 'ee1ba855ecd245448c9e4bb1a1078f39', 'start_time': 1658556995501201152, 'end_time': 1658556998177232128, 'is_user_option': False, 'slot_value': '不知道'}
2022-07-28 05:09:04,961 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 q :depression_parents_treat_negative text : 不知道
/opt/venv/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=1024 is too small for input signal of length=655
warnings.warn(
2022-07-28 05:09:05,139 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 q :depression_parents_treat_negative audio feature
2022-07-28 05:09:05,139 [WARNING] __main__: a729718a-1e15-4c32-80ac-035d98b13363, 1658556995501201152, 1658556998177232128
{'fixation_mean': 0.40900000000000003, 'fixation_std': 0.30507949783621974, 'saccade_mean': 0.04833333333333334, 'saccade_std': 0.04572623851673007, 'fixation_freq': 2.2598870056497176, 'fixation_max': 0.869, 'saccade_freq': 1.694915254237288}
2022-07-28 05:09:05,157 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 eye features: {'fixation_mean': 0.40900000000000003, 'fixation_std': 0.30507949783621974, 'saccade_mean': 0.04833333333333334, 'saccade_std': 0.04572623851673007, 'fixation_freq': 2.2598870056497176, 'fixation_max': 0.869, 'saccade_freq': 1.694915254237288}
None
```
5. 執行完畢，container的STATUS會是**Exited**
```
$ sudo docker ps -a
```
7. 特徵保存在 [mongodb therapist.features](#therapistfeatures)
### 後臺修改
* Error：ZeroDivisionError: float division by zero
* Fix：
* `/home/wmlab/virtual_therapist/backend/pkgs/feature-eye/feature_eye/`
* eye_feature.py
* store.py
* `/home/wmlab/virtual_therapist/backend/pkgs/feature-heartrate/feature_heartrate/`
* heart_rate.py
## RASA 後臺
* [RASA X 對話紀錄](http://140.115.54.18:5002/login)
1. 查詢登入密碼
```
$ sudo docker logs mh-chat_chatbot-core_1
```
2. 會出現一行，XXXXX 就是密碼
```
The server is running at http://localhost:5002/login?username=me&password=XXXXX
```
## 版本切換 (成年人版本 / 青少年版本)
- 修改檔案，其資料夾中有兩個同檔名不同內容的檔案，以 . 跟 .backup 做區分，分別是成年人版本及青少年版本，將其對調即可
- /actions/therapists/depression/intent_exp.yaml
- /actions/therapists/depression/validate_depression_form.py
- /domain/therapists_depression.yml
## 資料
* [修改版收案紀錄](https://hackmd.io/2_avj3z7T82EY4pZyLzFaw?view)
* [正式版收案紀錄](https://docs.google.com/spreadsheets/d/1mnCQnOEHi3CxQzhOgTLKrdh0dKcU00ht5WAdqcNCF9A/edit?usp=share_link)
## 正式版修改紀錄
* [virtual_therapist/mh-chat/actions/therapists/depression/**validate_depression_form.py**](https://hackmd.io/VM-IYL8eRhWJp6DVwcX0ow?view)
Sign in with Wallet
Connect another wallet