{%hackmd theme-dark %}

# Haru Notes

###### tags: `HRI`
[TOC]

---
## [ToDo](https://hackmd.io/@jitesh/todo-haru)

[Docs](https://drive.google.com/drive/folders/17Ud5RXh2ido0hXm_HY10VIrtKXWg_V8p)

## 1) Small talk

### 1.1) Usage
:::spoiler

cd haru-smalltalk
roslaunch haru_gazebo haru_simulator.launch
roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch
roslaunch ./launch/backbone-advanced.launch
roslaunch ./launch/test_bt.launch 

:::

roslaunch haru_unity unity_app_launcher.launch
roscd haru_projector
./run.sh smalltalk
cdhst
roslaunch ./launch/backbone-advanced-noisefix.launch
roslaunch haru_smalltalk_utils haru_asr_filter.launch
roslaunch haru_nlp_ros ssml_service.launch
roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch
roslaunch ./launch/test_bt_latest.launch

### 1.2) Profiling

#### 1.2.1) CPU

- 19GB RAM

#### 1.2.2) GPU

| Process | Memory (MiB) |
| ------- | ------------ |
| Face | 3405 |
| Kinect | 2463 |
| Rviz | 1103 |

---

## 2) Storytelling

### 2.1) Installation
#### For first time, setup the --- repo.

git clone [e-mail]:underspecified/hri_storytelling_server.git

#### Make virtual env, I choose miniconda for that,

conda create --name ENV_NAME python=3.8
conda activate ENV_NAME

#### Install necessary packages

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
cd hri_storytelling_server
pip install -r requirements.txt
python insert_prompts.py

#### Download the model

bash update_models.sh get

unzip models.zip -d models  # Optional
bash update_models.sh put  # Optional

tar -xvf hri-writingprompts2-774M-pytorch.tar.gz
# Change model-name in config.py
# Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0)

#### Optional
- Change model-name in config.py
- Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0)
- Put the config.rviz file somewhere on your computer and specify the path in line 5 of the testbed.launch
- Unzip [both packages](https://strawberry-jp.slack.com/archives/D02TKFCM355/p1642574069000500) in your catkin folder, catkin_make and source the bash.
- Softlink python path `sudo ln -s /usr/bin/python3 /usr/bin/python`

### 2.2) Usage

#### 2.2.1) Run Haru Robot
##### For Real Robot
:::spoiler

echo "<HARU_IP>    <HARU_ID>" | sudo tee -a /etc/hosts
ping HARU_IP
ssh haru@HARU_IP # Optional

- Edit bashrc
- Open .bashrc

    nano .bashrc

- Add/replace ROS_MASTER_HOSTNAME

    export ROS_MASTER_HOSTNAME="haru-29"
    export ROS_MASTER_URI=http://$ROS_MASTER_HOSTNAME:11311/
    export MY_IP=$(ip route get 8.8.8.8 | awk -F"src " 'NR==1{split($2,a," ");print a[1]}')
    export ROS_IP=$MY_IP
    export ROS_HOSTNAME=$ROS_IP

- Check the website

    chrome http://$ROS_IP:7000

:::

##### For simulation

roslaunch haru_gazebo haru_simulator.launch

#### 2.2.2) Run models

python webserver.py
roslaunch ~/haru/haru-storytelling/testbed.launch
roslaunch ~/haru/haru-storytelling/bt.launch

* [Reference](https://strawberry-jp.slack.com/archives/D02TKFCM355/p1642483345004200)
* [Download testbed.launch](https://strawberry-jp.slack.com/files/USWMHKANL/F02U7BKQSJ2/testbed.launch)

### 2.3) IST explanation
- #### 2.3.1) Thinking routine

| | Routine id | Routine name | Base Motion | Sound |
| --- | ---------- | ------------- | ----------- | ----- |
| 1 | 533 | thinking_0 | abs | no |
| 2 | 908 | unsure | none | no |
| 3 | 811 | look_around_c | abs | no |
| 4 | 816 | saccades_d | abs | no |
| 5 | 123 | idle3 | abs | no |
| 6 | 10 | Tracking | none | no |

- #### 2.3.2) [IST with Emotions](https://docs.google.com/presentation/d/1N3Exd4qAOM6D3OyGahy4ftePmgK_NgghKMgF31LNDuU/edit?usp=sharing)

- #### 2.3.3) [IST BT changes](https://hackmd.io/@jitesh/ist-bt-changes)

### 2.4) Profiling

#### 2.4.1) CPU
- 19GB RAM

#### 2.4.2) GPU
- Face: 3405MiB

namespace {

    std::vector<std::string> split_string(const std::string& s, char delimiter=';')
    {
        std::vector<std::string> splits;
        std::string split;
        std::string input = s;
        // Catch cases when input string doesn't end with a delimiter
        if (s.back() != delimiter) {
            input.push_back(delimiter);
        }

        std::istringstream ss(input);
        while (std::getline(ss, split, delimiter))
        {
            splits.push_back(split);
        }
        return splits;
    }
}

template <>
struct AutomaticDeserialization<strawberry_ros_msgs::GetSSML::Request>
{
public:
static BT::PortsList requiredPorts()
{
return {
BT::InputPort<std::string>("criteria_name", "Method name to filter the emotion"),
BT::InputPort<std::string>("routines_anger", "String with routine ids separated by ';' as a delimiter for anger emotion"),
BT::InputPort<std::string>("routines_disgust", "String with routine ids separated by ';' as a delimiter for disgust emotion"),
BT::InputPort<std::string>("routines_fear", "String with routine ids separated by ';' as a delimiter for fear emotion"),
BT::InputPort<std::string>("routines_joy", "String with routine ids separated by ';' as a delimiter for joy emotion"),
BT::InputPort<std::string>("routines_neutral", "String with routine ids separated by ';' as a delimiter for neutral emotion"),
BT::InputPort<std::string>("routines_sadness", "String with routine ids separated by ';' as a delimiter for sadness emotion"),
BT::InputPort<std::string>("routines_surprise", "String with routine ids separated by ';' as a delimiter for surprise emotion"),
BT::InputPort<std::string>("service", "ROS service name"),

BT::OutputPort<std::string>("ssml_text", "ssml text including voice genre for tts"),
BT::OutputPort<std::uint32_t>("routine", "routine id"),
};
}

strawberry_ros_msgs::GetSSML::Request buildMessage(const BT::ActionNodeBase &_tree_node)
{
strawberry_ros_msgs::GetSSML::Request _command{};

const auto &message_field_routines_anger = _tree_node.getInput<std::string>("routines_anger");
const auto &message_field_routines_disgust = _tree_node.getInput<std::string>("routines_disgust");
const auto &message_field_routines_fear = _tree_node.getInput<std::string>("routines_fear");
const auto &message_field_routines_joy = _tree_node.getInput<std::string>("routines_joy");
const auto &message_field_routines_neutral = _tree_node.getInput<std::string>("routines_neutral");
const auto &message_field_routines_sadness = _tree_node.getInput<std::string>("routines_sadness");
const auto &message_field_routines_surprise = _tree_node.getInput<std::string>("routines_surprise");

if (!message_field_routines_anger || !message_field_routines_disgust || !message_field_routines_fear || !message_field_routines_joy || !message_field_routines_neutral || !message_field_routines_sadness || !message_field_routines_surprise)
throw BT::RuntimeError{_tree_node.name() + ": missing message input field."};

_command.routines_anger = message_field_routines_anger.value();
_command.routines_disgust = message_field_routines_disgust.value();
_command.routines_fear = message_field_routines_fear.value();
_command.routines_joy = message_field_routines_joy.value();
_command.routines_neutral = message_field_routines_neutral.value();
_command.routines_sadness = message_field_routines_sadness.value();
_command.routines_surprise = message_field_routines_surprise.value();
return _command;
}
};

![](https://i.imgur.com/KyzjkF6.jpg)
![](https://i.imgur.com/Wu6LHqu.jpg)
![](https://i.imgur.com/x6q06dI.jpg)