Haru Notes

tags: `HRI`

1) Small talk

1.1) Usage





cd haru-smalltalk
roslaunch haru_gazebo haru_simulator.launch
roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch
roslaunch ./launch/backbone-advanced.launch
roslaunch ./launch/test_bt.launch









roslaunch haru_unity unity_app_launcher.launch
roscd haru_projector
./run.sh smalltalk
cdhst
roslaunch ./launch/backbone-advanced-noisefix.launch
roslaunch haru_smalltalk_utils haru_asr_filter.launch
roslaunch haru_nlp_ros ssml_service.launch
roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch
roslaunch ./launch/test_bt_latest.launch

1.2) Profiling

1.2.1) CPU

19GB RAM

1.2.2) GPU

Process	Memory (MiB)
Face	3405
Kinect	2463
Rviz	1103

2) Storytelling

2.1) Installation

For first time, setup the –- repo.

git clone git@github.com:underspecified/hri_storytelling_server.git

Make virtual env, I choose miniconda for that,

conda create --name ENV_NAME python=3.8
conda activate ENV_NAME

Install necessary packages

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
cd hri_storytelling_server
pip install -r requirements.txt
python insert_prompts.py

Download the model

bash update_models.sh get

unzip models.zip -d models  # Optional
bash update_models.sh put  # Optional

tar -xvf hri-writingprompts2-774M-pytorch.tar.gz
# Change model-name in config.py
# Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0)

Optional

Change model-name in config.py
Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0)
Put the config.rviz file somewhere on your computer and specify the path in line 5 of the testbed.launch
Unzip both packages in your catkin folder, catkin_make and source the bash.
Softlink python path sudo ln -s /usr/bin/python3 /usr/bin/python

2.2) Usage

2.2.1) Run Haru Robot

For Real Robot

echo "<HARU_IP>&nbsp; &nbsp; <HARU_ID>" | sudo tee -a /etc/hosts
ping HARU_IP
ssh haru@HARU_IP # Optional

Edit bashrc

Open .bashrc

nano .bashrc

Add/replace ROS_MASTER_HOSTNAME





export ROS_MASTER_HOSTNAME="haru-29"
export ROS_MASTER_URI=http://$ROS_MASTER_HOSTNAME:11311/
export MY_IP=$(ip route get 8.8.8.8 | awk -F"src " 'NR==1{split($2,a," ");print a[1]}')
export ROS_IP=$MY_IP
export ROS_HOSTNAME=$ROS_IP

Check the website
```
chrome http://$ROS_IP:7000
```

For simulation

roslaunch haru_gazebo haru_simulator.launch

2.2.2) Run models

python webserver.py
roslaunch ~/haru/haru-storytelling/testbed.launch
roslaunch ~/haru/haru-storytelling/bt.launch

2.3) IST explanation

2.3.1) Thinking routine

	Routine id	Routine name	Base Motion	Sound
1	533	thinking_0	abs	no
2	908	unsure	none	no
3	811	look_around_c	abs	no
4	816	saccades_d	abs	no
5	123	idle3	abs	no
6	10	Tracking	none	no

2.3.2) IST with Emotions
2.3.3) IST BT changes

2.4) Profiling

2.4.1) CPU

19GB RAM

2.4.2) GPU

Face: 3405MiB


namespace {

    std::vector<std::string> split_string(const std::string& s, char delimiter=';')
    {
        std::vector<std::string> splits;
        std::string split;
        std::string input = s;
        // Catch cases when input string doesn't end with a delimiter
        if (s.back() != delimiter) {
            input.push_back(delimiter);
        }

        std::istringstream ss(input);
        while (std::getline(ss, split, delimiter))
        {
            splits.push_back(split);
        }
        return splits;
    }
}

template <>
struct AutomaticDeserialization<strawberry_ros_msgs::GetSSML::Request>
{
public:
    static BT::PortsList requiredPorts()
    {
        return {
            BT::InputPort<std::string>("criteria_name", "Method name to filter the emotion"),
            BT::InputPort<std::string>("routines_anger", "String with routine ids separated by ';' as a delimiter for anger emotion"),
            BT::InputPort<std::string>("routines_disgust", "String with routine ids separated by ';' as a delimiter for disgust emotion"),
            BT::InputPort<std::string>("routines_fear", "String with routine ids separated by ';' as a delimiter for fear emotion"),
            BT::InputPort<std::string>("routines_joy", "String with routine ids separated by ';' as a delimiter for joy emotion"),
            BT::InputPort<std::string>("routines_neutral", "String with routine ids separated by ';' as a delimiter for neutral emotion"),
            BT::InputPort<std::string>("routines_sadness", "String with routine ids separated by ';' as a delimiter for sadness emotion"),
            BT::InputPort<std::string>("routines_surprise", "String with routine ids separated by ';' as a delimiter for surprise emotion"),
            BT::InputPort<std::string>("service", "ROS service name"),

            BT::OutputPort<std::string>("ssml_text", "ssml text including voice genre for tts"),
            BT::OutputPort<std::uint32_t>("routine", "routine id"),
        };
    }

    strawberry_ros_msgs::GetSSML::Request buildMessage(const BT::ActionNodeBase &_tree_node)
    {
        strawberry_ros_msgs::GetSSML::Request _command{};

        const auto &message_field_routines_anger = _tree_node.getInput<std::string>("routines_anger");
        const auto &message_field_routines_disgust = _tree_node.getInput<std::string>("routines_disgust");
        const auto &message_field_routines_fear = _tree_node.getInput<std::string>("routines_fear");
        const auto &message_field_routines_joy = _tree_node.getInput<std::string>("routines_joy");
        const auto &message_field_routines_neutral = _tree_node.getInput<std::string>("routines_neutral");
        const auto &message_field_routines_sadness = _tree_node.getInput<std::string>("routines_sadness");
        const auto &message_field_routines_surprise = _tree_node.getInput<std::string>("routines_surprise");

        if (!message_field_routines_anger || !message_field_routines_disgust || !message_field_routines_fear || !message_field_routines_joy || !message_field_routines_neutral || !message_field_routines_sadness || !message_field_routines_surprise)
            throw BT::RuntimeError{_tree_node.name() + ": missing message input field."};

        _command.routines_anger =  message_field_routines_anger.value();
        _command.routines_disgust = message_field_routines_disgust.value();
        _command.routines_fear = message_field_routines_fear.value();
        _command.routines_joy = message_field_routines_joy.value();
        _command.routines_neutral = message_field_routines_neutral.value();
        _command.routines_sadness = message_field_routines_sadness.value();
        _command.routines_surprise = message_field_routines_surprise.value();
        return _command;
    }
};



![](https://i.imgur.com/KyzjkF6.jpg)
![](https://i.imgur.com/Wu6LHqu.jpg)
![](https://i.imgur.com/x6q06dI.jpg)

Haru Notes

tags: HRI

ToDo

1) Small talk

1.1) Usage

1.2) Profiling

1.2.1) CPU

1.2.2) GPU

2) Storytelling

2.1) Installation

For first time, setup the –- repo.

Make virtual env, I choose miniconda for that,

Install necessary packages

Download the model

Optional

2.2) Usage

2.2.1) Run Haru Robot

For Real Robot

For simulation

2.2.2) Run models

2.3) IST explanation

2.3.1) Thinking routine

2.3.2) IST with Emotions

2.3.3) IST BT changes

2.4) Profiling

2.4.1) CPU

2.4.2) GPU

Read more

Conda

Untitled

Docs: Handover by Jitesh

Encouraging Mediator

tags: `HRI`