---
# System prepended metadata

title: Virtual Therapist 後端

---

> Reference: 
> [1. RASA Your Own Website — Websocket Channel & JWT Authentication](https://rasa.com/docs/rasa/connectors/your-own-website/#websocket-channel)
> [2. RASA Open Source HTTP API — JWT Based Auth](https://rasa.com/docs/rasa/http-api/#jwt-based-auth)
> [3. PyJWT](https://pyjwt.readthedocs.io/en/latest/)
> [4. API Reference — python-socketio  documentation](https://python-socketio.readthedocs.io/en/latest/api.html#socketio.Client.connect)
> [5. 基于 rasa 的语音助手搭建](https://dustyposa.github.io/posts/d7e97916/)

# Virtual Therapist
## 架構圖

前端透過RASA框架提供的socket io channel，以proto定義的clientMessageRequest傳送資料，包含語音、臉部圖片、臉部關鍵點（facial landmarks）等等數據。

![](https://i.imgur.com/Alb8FfS.png)

## 指令
* docker ps
    `sudo docker ps -a --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}"`


## 啟動

### mh-chat

1. **製作對話系統 images**
    
    ```
    $ cd /home/wmlab/virtual_therapist/mh-chat
    $ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build kafka1 chatbot-actions chatbot-core
    ```

2. **啟動聊天系統**
    
    ```
    $ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml up -d
    ```
    
    * `-d`：run in background
    
    確認 containers 正常執行： (init_kafka 創完 topic 就會 Exited)
    
    ```
    $ sudo docker ps -a
    ```
    
    ```
    CONTAINER ID   IMAGE                                                                                             COMMAND                   CREATED          STATUS                      PORTS                                                                                  NAMES
    70e2def9e357   confluentinc/cp-kafka:6.1.1                                                                       "/bin/sh -c '\n# bloc…"   39 minutes ago   Exited (0) 38 minutes ago                                                                                          init_kafka
    e25f4b8e3dba   confluentinc/cp-kafka:6.1.1                                                                       "/etc/confluent/dock…"    39 minutes ago   Up 39 minutes               9092/tcp, 0.0.0.0:19092->19092/tcp, :::19092->19092/tcp                                kafka1
    6afeaa097cfb   confluentinc/cp-zookeeper:6.1.1                                                                   "/etc/confluent/dock…"    39 minutes ago   Up 39 minutes               2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp                          zookeeper
    8ef0bd9b31a7   registry.cn-shanghai.aliyuncs.com/goxy/chatbot-actions:ebc656b8953e8041e52a11b274ddede8f08d308a   "./entrypoint.sh sta…"    39 minutes ago   Up 39 minutes               0.0.0.0:5055->5055/tcp, :::5055->5055/tcp                                              mh-chat_chatbot-actions_1
    24ecc20e96a5   rasa/duckling                                                                                     "duckling-example-ex…"    39 minutes ago   Up 39 minutes               0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                              mh-chat_duckling_1
    02bf14c81b58   registry.cn-shanghai.aliyuncs.com/goxy/chatbot-core:ebc656b8953e8041e52a11b274ddede8f08d308a      "rasa x --domain dom…"    39 minutes ago   Up 39 minutes               0.0.0.0:5002->5002/tcp, :::5002->5002/tcp, 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp   mh-chat_chatbot-core_1
    2e739181b675   mongo:latest                                                                                      "docker-entrypoint.s…"    39 minutes ago   Up 39 minutes               0.0.0.0:27018->27017/tcp, :::27018->27017/tcp                                          mongodb
    ```

    確認 topic (messaging & datasync) 成功被建立：
    
    > Reference: [Docker compose create kafka topics](https://stackoverflow.com/questions/64865361/docker-compose-create-kafka-topics)
    
    ```
    $ sudo docker logs init_kafka
    ```
    
    會出現類似下面的 log：
    
    ```
    [2022-05-27 04:29:21,771] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
    [2022-05-27 04:29:21,881] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
    [2022-05-27 04:29:22,082] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
    [2022-05-27 04:29:22,384] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka1/172.21.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

    Creating kafka topics
    Created topic messaging.
    Created topic datasync.
    Successfully created the following topics:
    datasync
    messaging
    ```
    
    (:warning: 一定要執行！) 確認 kafka 正常運作：
    > Reference: [Why Can't I Connect to Kafka? - Confluent](https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/)
    ```
    $ cd /home/wmlab/virtual_therapist/kafka-listeners/python
    $ sudo docker build -t python_kafka_test_client .
    $ sudo docker run --network=mh_chat_net --rm --name python_kafka_test_client --tty python_kafka_test_client kafka1:9092
    ```
    
    ```
    🥾 bootstrap server: kafka1:9092

    ✅ Connected to bootstrap server(kafka1:9092) and it returned metadata for brokers as follows:

    {0: BrokerMetadata(0, kafka1:9092)}
        ---------------------
        ℹ️  This step just confirms that the bootstrap connection was successful. 
        ℹ️  For the consumer to work your client will also need to be able to resolve the broker(s) returned
            in the metadata above.
        ℹ️  If the host(s) shown are not accessible from where your client is running you need to change 
            your advertised.listener configuration on the Kafka broker(s).
    

    <Producing>
    ✅  📬  Message delivered: "foo / 2022-05-27 05:39:41" to test_topic [partition 0]

    <Consuming>
    ✅  💌  Message received:  "foo / 2022-05-27 05:39:41" from topic test_topic
    ```
    
### 後臺修改
* /home/wmlab/virtual_therapist/mh-chat/Dockerfile
    * Dockerfile 41 行 根據官方更改：https://python-poetry.org/docs/ 
   ```py=4
   RUN chmod 777 /tmp
   ```
   ```py=41
    # install poetry - respects $POETRY_VERSION & $POETRY_HOME
    # RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
    # The get-poetry.py installer has been deprecated and removed.
    RUN curl -sSL https://install.python-poetry.org | python3 -
    ```

* /home/wmlab/virtual_therapist/mh-chat/requirements/**senta.txt**
   ```yml
   numpy==1.23.5
   ```
* virtual_therapist/mh-chat/**docker-compose.yml**
    * zoo1 和 kafka1 新增 user（權限問題）
	```yml
	user: root
	```
    * mongodb 改寫
    ```yml
    mongodb:
    	image: mongo:latest
    	container_name: "mongodb"
    	# restart: always
    	user: root
    	networks:
    	  mh_chat_net:
    	    ipv4_address: 172.27.0.8
    	environment:
    	  - MONGO_DATA_DIR=/data/db
    	  - MONGO_LOG_DIR=/data/logs
    	  - MONGO_INITDB_ROOT_USERNAME=admin
    	  - MONGO_INITDB_ROOT_PASSWORD=ZMJPh5EliBgnM765
    	volumes:
    	  - type: bind
    	    source: /home/wmlab/HDD_4T/mh_chat_mongo_data
    	    target: /data/db
    	  - type: bind
    	    source: /home/wmlab/HDD_4T/mh_chat_mongo_log
    	    target: /data/logs
	
    	ports:
    	  - 27018:27017
    	command: sh -c 'chmod 777 -R /tmp ; mongod --bind_ip_all --port 27017 --auth '
	
    	# 設定 mongo 資料夾權限
    	security_opt:
    	  - "seccomp:unconfined"
    ```

### backend

1. **製作 face recognition image**
    
    > Source: [ageitgey/face_recognition](https://github.com/ageitgey/face_recognition)

    ```
    $ cd /home/wmlab/virtual_therapist/face_recognition
    $ sudo docker build -t gao_wan/face_recognition .
    ```
    * `-t`：add tag for image
    
    測試 face recognition image：
    ```
    $ sudo docker-compose up
    ```
    
    ```
    Creating face_recognition ... done
    Attaching to face_recognition
    face_recognition    | I found 1 face(s) in this photograph.
    face_recognition    | A face is located at pixel location Top: 235, Left: 428, Bottom: 518, Right: 712
    face_recognition exited with code 0
    ```

2. **製作後端 images**
    ```
    $ cd /home/wmlab/virtual_therapist/backend
    $ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver-base
    $ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver receiver-grpc
    ```

3. **啟動後端**
    ```
    $ sudo docker-compose up -d mongodb receiver receiver-grpc
    ```
    
    確認 containers 正常執行：
    ```
    $ sudo docker ps -a
    ```
    
    ```
    CONTAINER ID   IMAGE                                                                                             COMMAND                   CREATED             STATUS                     PORTS                                           NAMES
    02c917fc909f   registry.cn-shanghai.aliyuncs.com/goxy/receiver:1.0.0                                             "/bin/sh -c './docke…"    4 minutes ago       Up 3 minutes                                                               backend_receiver_1
    0de50e1152e5   mongo:latest                                                                                      "docker-entrypoint.s…"    4 minutes ago       Up 3 minutes               0.0.0.0:27017->27017/tcp, :::27017->27017/tcp   backend_mongodb_1
    89328cd138c0   registry.cn-shanghai.aliyuncs.com/goxy/receiver-grpc:1.0.0                                        "/bin/sh -c './docke…"    4 minutes ago       Up 3 minutes               0.0.0.0:7777->7777/tcp, :::7777->7777/tcp       backend_receiver-grpc_1
    ```
    
    確認 backend receiver 與 kafka 連接成功：
    ```
    $ sudo docker logs backend_receiver_1
    ```
    
    ```
    Device status cuda
    +ƒaµS† v1.10.4+------------------------------------------+
    | id          | faust-consumer                           |
    | transport   | [URL('kafka://kafka1:9092')]             |
    | store       | memory:                                  |
    | web         | http://localhost:6066/                   |
    | log         | -stderr- (warn)                          |
    | pid         | 7                                        |
    | hostname    | 5d4df30f9d55                             |
    | platform    | CPython 3.8.10 (Linux x86_64)            |
    | drivers     |                                          |
    |   transport | aiokafka=1.1.6                           |
    |   web       | aiohttp=3.8.1                            |
    | datadir     | /app/faust-consumer-data                 |
    | appdir      | /app/faust-consumer-data/v1              |
    +-------------+------------------------------------------+
     OK ^
    ```

### 後臺修改
* /home/wmlab/virtual_therapist/**face_recognition**/docker-compose.yml
    * 新增 user（權限問題）
    `user: root`
* /home/wmlab/virtual_therapist/**face_recognition**/Dockerfile
    * 新增
    ```py=8
    RUN chmod 777 /tmp
    ```
* /home/wmlab/virtual_therapist/**backend**/docker-compose.yml
    * mh-chat 提到的 mongodb 改寫
    ```yml
        mongodb:
        networks: 
            mh_chat_net:
                ipv4_address: 172.27.0.11
        image: mongo:latest
        environment:
            MONGO_INITDB_ROOT_USERNAME: root
            MONGO_INITDB_ROOT_PASSWORD: rootpassword
        ports:
            - 27017:27017
        command: sh -c 'chmod 777 -R /tmp ; mongod --bind_ip_all --port 27017 --auth '
        volumes:
            - /home/wmlab/HDD_4T/backend_mongo_data:/data/db
            # - mongodb_data:/data/db
    ```
* /home/wmlab/virtual_therapist/**backend**/Dockerfile
    * 新增
    ```py=8
    RUN chmod 777 /tmp
    ```

* /home/wmlab/virtual_therapist/**backend**/docker/base/Dockerfile
    * 改寫
    ```py=44
    # RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
    RUN curl -sSL https://install.python-poetry.org | python3 -
    RUN chmod 777 /tmp
    ```

* /home/wmlab/virtual_therapist/**backend**/docker-entrypoint.sh
    ```py
    #!/bin/sh

    set -e

    # activate our virtual environment here
    . $VENV_PATH/bin/activate

    # You can put other setup logic here
    for dir in /app/pkgs/*/
    do
        export PYTHONPATH="${PYTHONPATH}:${dir}"
    done
    # Evaluating passed command:
    exec "$@"
    ```

---

## 資料庫

![123](https://i.imgur.com/2gswmOv.png)

可以用 [MongoDB Compass](https://www.mongodb.com/products/compass) 存取資料庫

![](https://i.imgur.com/2qOm0je.png)


### 對話紀錄
帳密：登入請至實驗室 wiki 查詢
http://140.115.54.26:8787/

#### ==rasa.conversations==

對話紀錄，包含每一題的++意圖識別++、++speech to text結果++與++多模態情緒識別結果++等等資訊

![](https://i.imgur.com/7C6AZAB.png)

### 受測者資料&多模態數據
帳密：登入請至實驗室 wiki 查詢
http://140.115.54.26:8787/

#### ==therapist.Subject==

受測者資料，前端emit `session_request` 時帶上的資料(字典)

```python
# Python example
data = {
    "name": "高丸",
    "birthday": "1998-08-13",
    "gender": "male",
    "depression_level": 5
    # 可任意新增欄位
}

sio.emit('session_request', data)
```

後端會帶入時間戳，若有完成對話會有結束時間(end_timestamp)，否則結束時間為null

![](https://i.imgur.com/CViculr.png)


#### ==therapist.ExpressionMessageList==

臉部、心率與眼動的情緒識別結果

![](https://i.imgur.com/PD6ERbS.png)

#### ==therapist.HeartRateMessageWithSession==

估測的心率

![](https://i.imgur.com/sUx3b7s.png)

#### ==therapist.FixationMessage==

Unity ARkit捕捉的眼動數據

![](https://i.imgur.com/w6jODfq.png)

#### ==therapist.features==

每個問答提取的五個模態特徵

![](https://i.imgur.com/FmiPD1d.png)


## 語音&影片

前端傳送的語音、臉部圖片與影片(從圖片轉檔的)會保存在server的4T硬碟
路徑為：`/home/wmlab/HDD_4T/virtualtherapists/storage/{session_id}`

```
.
├── audio          # 語音
├── face_cropped   # dlib人臉識別切出來的人臉圖片
└── video          # 原圖&從圖片轉成的影片
```

## 導出多模態特徵

1. 導出問答的原始數據

    ```
    $ cd /home/wmlab/virtual_therapist/export
    $ bash run.sh
    ```

2. `backend/docker-compose.yml` 設定 `receiver-feature` 的環境變數：`FEATURE_SID`

    ```yml=31
    receiver-feature:
        networks: 
            mh_chat_net:
                ipv4_address: 172.27.0.12
        image: ${DOCKER_REGISTRY_SHANGHAI}/receiver-feature:${DOCKER_TAG}
        runtime: nvidia
        volumes:
            - '/home/wmlab/HDD_4T/virtualtherapists/storage:/app/feature_extraction/storage/data'
            - '/home/wmlab/HDD_4T/virtualtherapists/storage/raw_data:/app/feature_extraction/storage/raw_data'
        environment:
            RECEIVER_DB: mongodb://root:rootpassword@172.27.0.11:27017/
            FEATURE_DATA_PATH: /app/feature_extraction/storage
            FEATURE_SID: {{ session_id }}
    ```

3. 導出五個模態特徵

    ```
    $ cd /home/wmlab/virtual_therapist/backend
    $ sudo docker-compose -f docker-compose.yml -f docker-compose.development.yml build receiver-feature
    $ sudo docker-compose up -d receiver-feature
    ```
    
4. 查看log
    ```
    $ sudo docker logs backend_receiver-feature_1
    ```
    ```
    /opt/venv/lib/python3.8/site-packages/scipy/signal/spectral.py:1961: UserWarning: nperseg = 256 is greater than input length  = 52, using nperseg = 52
      warnings.warn('nperseg = {0:d} is greater than input length '
    2022-07-28 05:09:04,938 [INFO] __main__: {'sid': 'a729718a-1e15-4c32-80ac-035d98b13363', 'question_id': 'depression_parents_treat_negative', 'text': '不知道', 'audio_id': 'ee1ba855ecd245448c9e4bb1a1078f39', 'start_time': 1658556995501201152, 'end_time': 1658556998177232128, 'is_user_option': False, 'slot_value': '不知道'}
    2022-07-28 05:09:04,961 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 q :depression_parents_treat_negative text : 不知道
    /opt/venv/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=1024 is too small for input signal of length=655
      warnings.warn(
    2022-07-28 05:09:05,139 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 q :depression_parents_treat_negative audio feature
    2022-07-28 05:09:05,139 [WARNING] __main__: a729718a-1e15-4c32-80ac-035d98b13363, 1658556995501201152, 1658556998177232128
    {'fixation_mean': 0.40900000000000003, 'fixation_std': 0.30507949783621974, 'saccade_mean': 0.04833333333333334, 'saccade_std': 0.04572623851673007, 'fixation_freq': 2.2598870056497176, 'fixation_max': 0.869, 'saccade_freq': 1.694915254237288}
    2022-07-28 05:09:05,157 [INFO] __main__: sid: a729718a-1e15-4c32-80ac-035d98b13363 eye features: {'fixation_mean': 0.40900000000000003, 'fixation_std': 0.30507949783621974, 'saccade_mean': 0.04833333333333334, 'saccade_std': 0.04572623851673007, 'fixation_freq': 2.2598870056497176, 'fixation_max': 0.869, 'saccade_freq': 1.694915254237288}
    None
    ```
    
5. 執行完畢，container的STATUS會是**Exited**
    ```
    $ sudo docker ps -a
    ```
7. 特徵保存在 [mongodb therapist.features](#therapistfeatures)

### 後臺修改

* Error：ZeroDivisionError: float division by zero
* Fix：
    * `/home/wmlab/virtual_therapist/backend/pkgs/feature-eye/feature_eye/`
        * eye_feature.py
        * store.py
    * `/home/wmlab/virtual_therapist/backend/pkgs/feature-heartrate/feature_heartrate/`
        * heart_rate.py

## RASA 後臺
 * [RASA X 對話紀錄](http://140.115.54.18:5002/login)
1. 查詢登入密碼
```
$ sudo docker logs mh-chat_chatbot-core_1
```
2. 會出現一行，XXXXX 就是密碼
```
The server is running at http://localhost:5002/login?username=me&password=XXXXX
```

## 版本切換 (成年人版本 / 青少年版本)
- 修改檔案，其資料夾中有兩個同檔名不同內容的檔案，以 . 跟 .backup 做區分，分別是成年人版本及青少年版本，將其對調即可
    - /actions/therapists/depression/intent_exp.yaml
    - /actions/therapists/depression/validate_depression_form.py
    - /domain/therapists_depression.yml

## 資料
* [修改版收案紀錄](https://hackmd.io/2_avj3z7T82EY4pZyLzFaw?view)
* [正式版收案紀錄](https://docs.google.com/spreadsheets/d/1mnCQnOEHi3CxQzhOgTLKrdh0dKcU00ht5WAdqcNCF9A/edit?usp=share_link)

## 正式版修改紀錄
* [virtual_therapist/mh-chat/actions/therapists/depression/**validate_depression_form.py**](https://hackmd.io/VM-IYL8eRhWJp6DVwcX0ow?view)