{%hackmd theme-dark %} # Haru Notes ###### tags: `HRI` [TOC] --- ## [ToDo](https://hackmd.io/@jitesh/todo-haru) [Docs](https://drive.google.com/drive/folders/17Ud5RXh2ido0hXm_HY10VIrtKXWg_V8p) ## 1) Small talk ### 1.1) Usage :::spoiler ```shell= cd haru-smalltalk roslaunch haru_gazebo haru_simulator.launch roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch roslaunch ./launch/backbone-advanced.launch roslaunch ./launch/test_bt.launch ``` ::: ```shell= roslaunch haru_unity unity_app_launcher.launch roscd haru_projector ./run.sh smalltalk cdhst roslaunch ./launch/backbone-advanced-noisefix.launch roslaunch haru_smalltalk_utils haru_asr_filter.launch roslaunch haru_nlp_ros ssml_service.launch roslaunch strawberry_ros_dialog strawberry_ros_dialog.launch roslaunch ./launch/test_bt_latest.launch ``` ### 1.2) Profiling #### 1.2.1) CPU - 19GB RAM #### 1.2.2) GPU | Process | Memory (MiB) | | ------- | ------------ | | Face | 3405 | | Kinect | 2463 | | Rviz | 1103 | --- ## 2) Storytelling ### 2.1) Installation #### For first time, setup the --- repo. ```bash git clone git@github.com:underspecified/hri_storytelling_server.git ``` #### Make virtual env, I choose miniconda for that, ```bash conda create --name ENV_NAME python=3.8 conda activate ENV_NAME ``` #### Install necessary packages ```shell conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch cd hri_storytelling_server pip install -r requirements.txt python insert_prompts.py ``` #### Download the model ```shell bash update_models.sh get unzip models.zip -d models # Optional bash update_models.sh put # Optional tar -xvf hri-writingprompts2-774M-pytorch.tar.gz # Change model-name in config.py # Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0) ``` #### Optional - Change model-name in config.py - Either comment out the lines related to emotion-model or upgrade transformer to the latest version (preferably 4.15.0) - Put the config.rviz file somewhere on your computer and specify the path in line 5 of the testbed.launch - Unzip [both packages](https://strawberry-jp.slack.com/archives/D02TKFCM355/p1642574069000500) in your catkin folder, catkin_make and source the bash. - Softlink python path `sudo ln -s /usr/bin/python3 /usr/bin/python` ### 2.2) Usage #### 2.2.1) Run Haru Robot ##### For Real Robot :::spoiler ```bash echo "<HARU_IP>    <HARU_ID>" | sudo tee -a /etc/hosts ping HARU_IP ssh haru@HARU_IP # Optional ``` - Edit bashrc - Open .bashrc ```bash nano .bashrc ``` - Add/replace ROS_MASTER_HOSTNAME ```bash= export ROS_MASTER_HOSTNAME="haru-29" export ROS_MASTER_URI=http://$ROS_MASTER_HOSTNAME:11311/ export MY_IP=$(ip route get 8.8.8.8 | awk -F"src " 'NR==1{split($2,a," ");print a[1]}') export ROS_IP=$MY_IP export ROS_HOSTNAME=$ROS_IP ``` - Check the website ``` chrome http://$ROS_IP:7000 ``` ::: ##### For simulation ```shell roslaunch haru_gazebo haru_simulator.launch ``` #### 2.2.2) Run models ```shell python webserver.py roslaunch ~/haru/haru-storytelling/testbed.launch roslaunch ~/haru/haru-storytelling/bt.launch ``` * [Reference](https://strawberry-jp.slack.com/archives/D02TKFCM355/p1642483345004200) * [Download testbed.launch](https://strawberry-jp.slack.com/files/USWMHKANL/F02U7BKQSJ2/testbed.launch) ### 2.3) IST explanation - #### 2.3.1) Thinking routine | | Routine id | Routine name | Base Motion | Sound | | --- | ---------- | ------------- | ----------- | ----- | | 1 | 533 | thinking_0 | abs | no | | 2 | 908 | unsure | none | no | | 3 | 811 | look_around_c | abs | no | | 4 | 816 | saccades_d | abs | no | | 5 | 123 | idle3 | abs | no | | 6 | 10 | Tracking | none | no | - #### 2.3.2) [IST with Emotions](https://docs.google.com/presentation/d/1N3Exd4qAOM6D3OyGahy4ftePmgK_NgghKMgF31LNDuU/edit?usp=sharing) - #### 2.3.3) [IST BT changes](https://hackmd.io/@jitesh/ist-bt-changes) ### 2.4) Profiling #### 2.4.1) CPU - 19GB RAM #### 2.4.2) GPU - Face: 3405MiB ``` namespace { std::vector<std::string> split_string(const std::string& s, char delimiter=';') { std::vector<std::string> splits; std::string split; std::string input = s; // Catch cases when input string doesn't end with a delimiter if (s.back() != delimiter) { input.push_back(delimiter); } std::istringstream ss(input); while (std::getline(ss, split, delimiter)) { splits.push_back(split); } return splits; } } ``` template <> struct AutomaticDeserialization<strawberry_ros_msgs::GetSSML::Request> { public: static BT::PortsList requiredPorts() { return { BT::InputPort<std::string>("criteria_name", "Method name to filter the emotion"), BT::InputPort<std::string>("routines_anger", "String with routine ids separated by ';' as a delimiter for anger emotion"), BT::InputPort<std::string>("routines_disgust", "String with routine ids separated by ';' as a delimiter for disgust emotion"), BT::InputPort<std::string>("routines_fear", "String with routine ids separated by ';' as a delimiter for fear emotion"), BT::InputPort<std::string>("routines_joy", "String with routine ids separated by ';' as a delimiter for joy emotion"), BT::InputPort<std::string>("routines_neutral", "String with routine ids separated by ';' as a delimiter for neutral emotion"), BT::InputPort<std::string>("routines_sadness", "String with routine ids separated by ';' as a delimiter for sadness emotion"), BT::InputPort<std::string>("routines_surprise", "String with routine ids separated by ';' as a delimiter for surprise emotion"), BT::InputPort<std::string>("service", "ROS service name"), BT::OutputPort<std::string>("ssml_text", "ssml text including voice genre for tts"), BT::OutputPort<std::uint32_t>("routine", "routine id"), }; } strawberry_ros_msgs::GetSSML::Request buildMessage(const BT::ActionNodeBase &_tree_node) { strawberry_ros_msgs::GetSSML::Request _command{}; const auto &message_field_routines_anger = _tree_node.getInput<std::string>("routines_anger"); const auto &message_field_routines_disgust = _tree_node.getInput<std::string>("routines_disgust"); const auto &message_field_routines_fear = _tree_node.getInput<std::string>("routines_fear"); const auto &message_field_routines_joy = _tree_node.getInput<std::string>("routines_joy"); const auto &message_field_routines_neutral = _tree_node.getInput<std::string>("routines_neutral"); const auto &message_field_routines_sadness = _tree_node.getInput<std::string>("routines_sadness"); const auto &message_field_routines_surprise = _tree_node.getInput<std::string>("routines_surprise"); if (!message_field_routines_anger || !message_field_routines_disgust || !message_field_routines_fear || !message_field_routines_joy || !message_field_routines_neutral || !message_field_routines_sadness || !message_field_routines_surprise) throw BT::RuntimeError{_tree_node.name() + ": missing message input field."}; _command.routines_anger = message_field_routines_anger.value(); _command.routines_disgust = message_field_routines_disgust.value(); _command.routines_fear = message_field_routines_fear.value(); _command.routines_joy = message_field_routines_joy.value(); _command.routines_neutral = message_field_routines_neutral.value(); _command.routines_sadness = message_field_routines_sadness.value(); _command.routines_surprise = message_field_routines_surprise.value(); return _command; } }; ``` ![](https://i.imgur.com/KyzjkF6.jpg) ![](https://i.imgur.com/Wu6LHqu.jpg) ![](https://i.imgur.com/x6q06dI.jpg)
Last changed by  
 
Jitesh Gosar
219

Read more

Conda

Activate Conda for the Current Session: Use the command provided in the installation message, but replace YOUR_SHELL_NAME with the name of your shell (e.g., bash, zsh). For example, if you're using bash:

3/5/2024
Untitled

Health and Safety

12/21/2023
Docs: Handover by Jitesh

Much of the work I have completed can be found documented in the README files within the respective repositories or on this webpage. If the Person in charge for overseeing the project (excluding myself) is unavailable for clarification, please do not hesitate to reach out to me at gosar95@gmail.com. 1. Projector (for IST and Smalltalk Demo) 3 parts are used together. 1.1 Python strawberry-ros-projector Person in charge: Lithin, Matija Description: Here we are subscribing to the dialog_result or other rostopic with converation information, convert that data to a json output and post on localhost with port 1706.http://localhost:1706/ For json data http://localhost:3000/ For Frontend app that shows chat info You can show the Frontend on Unity by Press CTRL + w keys or by publishing the above URL on rostopic /unity_projector/conversationApp/set

5/18/2023
Encouraging Mediator

Demo steps information Unity Doc spreadsheets Run Commands Setup rosrun haru_encouraging_mediator setup.sh Run the app roslaunch haru_encouraging_mediator master.launch number_of_ipads:=3

5/11/2023
Read more from Jitesh Gosar
Published on HackMD