# AI Companion Healthcare Aid Robot
> Due to rapid aging, dealing with the shortage of healthcare resources and helping the elderly manage their daily life independently and build confidence gradually have been the focus of common concerns of technology and the medical industry. Unlike western countries, most elderly in Taiwan rely on home care. The early changes in the elderly's physical and mental health, however, are difficult for family caregivers to perceive because they lack specialised training in elder care, leading to failure of timely medical treatment. **CHARM - Companion Healthcare Aid Robot Manager** - is a social companion robot for the elderly suffering from chronic diseases, cognitive impairments, emotional disorders, and the general population. Based on [ASUS Zenbo Junior](https://zenbo.asus.com/product/zenbojunior/overview/), CHARM provides an autonomous service framework that focuses on the daily-life communication of the elderly. In this framework, we implement the following functions:
> - For the general group, it provides various chat functions, multimedia services, proactive scheduling, etc. Furthermore, it optimises personalised services based on reactions from users.
> - For cognitive health, it provides a brief assessment of levels of cognitive impairment through a natural dialogue to evaluate the risk of Alzheimer's Disease (AD) and Mild Cognitive Impairment (MCI).
> - For emotional disorders, it provides emotional recognition and suppressing based on natural dialogue to conduct positive emotional guidance.
> - For chronic disease care, it provides an early warning by detecting abnormal events in physiological data. Meanwhile, it equips routine life analysis and medical knowledge database to offer recommendations for health management.
>
> This document focuses on the **fall warning system** and the **reminiscence chatting system**.

###### tags: `healthcare robot` `social robot` `human–robot interaction` `natural language processing` `computer vision`

## Fall Warning System
According to the World Health Organization (WHO), adults over 60 have the highest rate of fatal falls. The system aims to detect abnormal posture and prevent falls due to losing balance in the process of getting up in the elderly.
Code and dataset are available [here](https://drive.google.com/drive/folders/1GYAKOV_ofviHfs7T9SSIQtPdZdrU1nVc?usp=sharing).
### Collect and preprocess data
1. Utilizing the built-in RGB camera, the clips for four pose categories, "lying", "sitting", "sitting on bed edge", and "standing", are captured and processed by [Google ML Kit Pose Detection API](https://developers.google.com/ml-kit/vision/pose-detection).
2. Every `*_p.txt` file contains `[index], [x location], [y location], [confidence]` of each landmark in one clip.
3. Shuffle and seperate the data into training set and validation set.
### Training
For posture recognition, Train a random forest classifier by [sklearn.ensemble.RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) with `max_depth=5, random_state=0`.

### Application
The trained model is applied to the CHARM system. The robot monitors their behaviour when the elderly get up from bed. The standard motion order will be “lying” → “sitting” → “sitting on bed edge” → “standing”. If the robot detects any abnormal order, such as falling, it will automatically alert the caregiver.

## Reminiscence Chatting System
Reminiscence therapy is a treatment that can help individuals with dementia recall their memories, that is, to remember events, people and places from their past lives. The therapy aims to assist patients in finding purpose in their lives, boosting their self-esteem, and enhancing their level of pleasure.
This system can proactively ask questions based on the patient’s past photos. We focus on how a robot can associate concepts relevant to the content in the photos and evoke patient's memories by asking relatable and engaging questions. All questions are in Mandarin.

### Image understanding and question generation
These functions are based on [Interactive Question-Posing System for Robot-Assisted Reminiscence From Personal Photographs](https://ieeexplore.ieee.org/document/8716707)[^Reminiscence]. In this paper, the design aims to generate a suitable question based on the given photo. However, the system does not process the meaning of user utterances.
[^Reminiscence]: Y. -L. Wu, E. Gamborino and L. -C. Fu, "Interactive Question-Posing System for Robot-Assisted Reminiscence From Personal Photographs," in *IEEE Transactions on Cognitive and Developmental Systems*, vol. 12, no. 3, pp. 439-450, Sept. 2020, doi: 10.1109/TCDS.2019.2917030.

### User utterance processing and response generation
In a conversation, giving responses is crucial for continuing the topic. Generating appropriate responses can encourage the elderly to share their story and thoughts. Therefore, the user utterance is processed by [jieba](https://pypi.org/project/jieba/) and [SnowNLP](https://github.com/isnowfy/snownlp) to extract the parts of speech and sentiment.
A user utterance is defined as "informative" if the sentence contains a noun and the length is more than five characters. When the user utterance is informative, the system will generate a response from [Blender Bot 2.0](https://ai.facebook.com/blog/blender-bot-2-an-open-source-chatbot-that-builds-long-term-memory-and-searches-the-internet/). Otherwise, the system will generate a general response based on the sentiment analysis score.

### Application
The photos are uploaded to a tablet and processed by image understanding to extract the labels of people, places, and events. After generating the labels, the robot will greet the user and start the conversation. One question is followed by two responses, and there will be five questions for one photo. The flowchart is shown below, where blue boxes represent the robot's statements. The user can change the photo or stop the conversation at any time.

graph TD;
Greeting:::robot-->Question;
Question:::robot-->FirstUserUtterance;
FirstUserUtterance-->FirstResponse;
FirstResponse:::robot-->SecondUserUtterance;
SecondUserUtterance-->SecondResponse;
SecondResponse:::robot-->Question;
classDef robot fill:#48d;

<img src="https://i.imgur.com/7eU1HKx.jpg" width="500"/>

The picture above is the practical test of the Reminiscence Chatting System.