# Encouraging Mediator * [Demo steps information](https://docs.google.com/spreadsheets/d/13tUHjGnjESW9kcTUPgptH4PUCOCwH42rmsdwFcTgOBA/edit?usp=sharing) * [Unity](https://www.dropbox.com/s/dhquzubiawzoaw1/test_encouraging_messages.zip?dl=0) * [Doc](https://docs.google.com/document/d/1iK73WO-o5BN1JNPcMj8eHCm4wtiuC6LbBf2jzwwOsXc/view#) * [spreadsheets](https://docs.google.com/spreadsheets/d/1OEW5JdYU7L78y0oz1V_xd_WWo-O8CqgqvvvzAyP13Xk/edit#gid=0) ## Run Commands ### Setup ```shell rosrun haru_encouraging_mediator setup.sh ``` ### Run the app ```shell= roslaunch haru_encouraging_mediator master.launch number_of_ipads:=3 rosrun haru_encouraging_mediator check_setup_and_connectivity_asr_ipad.sh ``` ### Run the BT for Japan ``` roslaunch haru_encouraging_mediator initiation.launch this_country:=japan episode:=2 ``` ### Run the BT for Australia ``` roslaunch haru_encouraging_mediator initiation.launch this_country:=australia episode:=2 ``` ## Setup config ### Japan side * local-ext-ip: `192.168.8.222` * local-robot-ip: `192.168.8.247` ### Japan - Australia ``` JAPAN External PC hostname haru-remote-ext VPN IP VPN ID 9 10.0.0.37 Robot/Avatar hostname haru-47 VPN IP VPN ID 10 10.0.0.41 Remote External PC hostname ext-pc VPN IP VPN ID 7 10.0.0.29 Remote Robot/Avatar hostname haru-0 VPN IP VPN ID 8 10.0.0.33 AUSTRALIA External PC hostname ext-pc VPN IP VPN ID 7 10.0.0.29 Robot/Avatar hostname haru-0 VPN IP VPN ID 8 10.0.0.33 Remote External PC hostname haru-remote-ext VPN IP VPN ID 9 10.0.0.37 Remote Robot/Avatar hostname haru-47 VPN IP VPN ID 10 10.0.0.41 ``` ### Japan - Spain :::spoiler ``` JAPAN External PC hostname haru-remote-ext VPN IP VPN ID 13 10.0.0.53 Robot/Avatar hostname haru-25 VPN IP VPN ID 14 10.0.0.57 Remote External PC hostname ext-pc-one (haru-local-ext) VPN IP VPN ID 11 10.0.0.45 Remote Robot/Avatar hostname haru-0 (haru-local-avatar) VPN IP VPN ID 12 10.0.0.49 SPAIN (aus) External PC hostname ext-pc-one VPN IP VPN ID 11 10.0.0.45 Robot/Avatar hostname haru-0 VPN IP VPN ID 12 10.0.0.49 Remote External PC hostname haru-remote-ext VPN IP VPN ID 13 10.0.0.53 Remote Robot/Avatar hostname haru-25 VPN IP VPN ID 14 10.0.0.57 ``` ::: ### Branches :::spoiler * haru-encouraging-mediator -> master * haru-encouraging-mediator-msgs -> master * behavior-tree-plugins -> master * haru-unity -> testing-..... ::: ## Tests ### Test on 2023-03-13 In Today's demo-tests, everything worked fine other than the following, | Test no. | Japan side issues | Australia side issues | | -------- | ------------------------------------- | -------------------------------- | | 1 | Dialog about the photos did not start | Only the right screen worked but image proportions were wrong ![](https://i.imgur.com/wafvDX1.png) | | 2 | same as above | Only the right screen worked perfectly fine | | 3 | same as above | Both screen worked but left screen was zoomed in ![](https://i.imgur.com/JVQ7PZe.png) | | 4 | same as above | Both screen worked but left screen was zoomed in ![](https://i.imgur.com/73Dkwk4.png) | ### Test on 2023-03-23 In Today's demo-tests, 1. In the first test we had an error `RL execption` after launching the `master.sh` file. The error went away after running the `setup.sh` script. I think the problem came because the ip of the external-pc changed since last time and the router did not have a fixed ip for external-pc. 1. we had only 1 complete test, which showed same result as test on `2023-03-13`, where everything worked except * On Japan side, Dialog about the photos did not start * On Australia side, left screen projector was zoomed in ### Test on 2023-03-24 In Today's demo-tests, | Test no. | Japan side issues | Australia side issues | | -------- | ------------------- | ----------------------------------------------- | | 1 | fkie failed 4 times | fkie failed 4 times | | 2 | fkie failed 4 times | fkie failed 4 times | | 3 | did not continue the conversation about sports (aus tree) | one screen projector was zoomed in | Points to discuss: 1. japan side is mute in the beginning, we should unmute all side for testing 1. videoconference screen changes from left to right, it should be consistent and not change sides 1. the 3-3 user profile screen on the projector does not look symmetrical enough 2. Dialog does not start on the japan side, should we update any package? ### Test on 2023-03-27 ### Test on 2023-03-28 ### Test on 2023-03-29 * Japan side robot's eyes are not working today * The Haru website is not working | Test no. | Japan side issues | Australia side issues | | -------- | ------------------- | ----------------------------------------------- | | 1 | Dialog: Haru asked about famous _sports_ player twice | - | ### Test on 2023-03-30 ### Test on 2023-03-31 | Test no. | ROS bag File name | Description | Review | | -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | ------ | | 1 | encouraging_japan_2023-03-31-10-44-23.bag | Test; connection error while Haru was talking about the 2nd or 3rd picture on Japan side ![](https://i.imgur.com/GTzoST3.jpg) | fail | | 2 | encouraging_japan_2023-03-31-11-05-25.bag.active | Test | pass | | 3 | encouraging_japan_2023-03-31-13-51-58.bag | Test | pass | | 4 | encouraging_japan_2023-03-31-15-40-12.bag | **Demo** round 1 | pass | | 5 | encouraging_japan_2023-03-31-16-24-23.bag | Demo round 2; 1 ipad on japan side did not show page to select profile pics; | fail | | 6 | encouraging_japan_2023-03-31-16-24-34.bag | Demo round 2; Australia side could not connect with the "Whereby room"; we waited for a minute before restarting | fail | | 7 | encouraging_japan_2023-03-31-16-35-42.bag | **Demo** round 2 | pass | #### Suggestions from the kids: 1. replace mute icon with text like "Japan turn", "Australia turn", "mute" and "Talking Room" 1. If possible, don't mute single side. 1. Add translation for Japanese kids as subtitle 1. Have some English keywords text display on the screen to help kids construct a sentence to say 1. Connect Haru with other Honda products like AC and Cars; IOT 1. Give kids simulation or some way to improve Haru features, one kid really want to make Haru do stuff with his program 1. Have Haru teach English #### Other suggestions: 1. When Haru says something on other side, that side is mute... 2. Pick up some keywords and haru have some routine 3. Wait for kids to stop talking. Haru kept interrupting the kids. We can use ASR to detect when kids stopped talking for 5 seconds, then make Haru say something 4. Launch everything with one command smoothly 5. Reduce the volume of privacy-video while Haru talks. There is already a rostopic that we can use to resuce it. ### Test OLD :::spoiler | Test type | Details | | --------- | ------------------------------ | | 1 | 8 MB; uncompressed images both sides | | 2 | uncompressed images on Australia, compressed images on Japan side | | 3 | 1.1 MB; compressed images both sides | | 4 | 50kb images both sides | | Test No. | Test type | Result | | -------- | -------- | -------- | | 1 | 1 | fail on japan side | | 2 | 2 | Success | | 3 | 2 | fail on Australia side | | 4 | 2 | Success | | 5 | 2 | fail on Australia side | | 6 | 2 | Success | | 7 | 2 | Success | | 8 | 4 | Success | | 9 | 4 | Success | | 10 | 4 | Success | | 11 | 4 | Success | ::: --- ## Appendix ### TTS To run TTS on external PC, add the following lines in your testbed. ``` <include file="$(find idmind_tabletop)/launch/idmind_tabletop_audio_controller.launch"> <arg name="robot_namespace" value="" /> <arg name="sounds_filepath" value="$(find idmind_tabletop)/resources/audio/" /> <arg name="device_id" value="-1" /> <arg name="get_volume_method" value="1" /> </include> <include file="$(find idmind_tabletop)/launch/idmind_tabletop_tts.launch"> <arg name="robot_namespace" value=""/> <arg name="play_audio" value="streamed"/> <arg name="voices_path" default="/opt/ros/noetic/share/idmind_tabletop/resources/voice/" /> <arg name="voices" default="andy_cerewave.voice;sara.voice;sarah.voice;haru.voice;yuki.voice;jordan.voice;haruchild.voice" /> <arg name="license" default="/opt/ros/noetic/share/idmind_tabletop/1326C40F-9F8B-44B2-99A6-EE88381E3988_ASH_license.lic" /> <arg name="default_language" default="en-US" /> <arg name="file_out" default="/tmp/tts_audio.wav" /> </include> ```
Last changed by  
 
Jitesh Gosar
336

Read more

Conda

Activate Conda for the Current Session: Use the command provided in the installation message, but replace YOUR_SHELL_NAME with the name of your shell (e.g., bash, zsh). For example, if you're using bash:

3/5/2024
Untitled

Health and Safety

12/21/2023
Docs: Handover by Jitesh

Much of the work I have completed can be found documented in the README files within the respective repositories or on this webpage. If the Person in charge for overseeing the project (excluding myself) is unavailable for clarification, please do not hesitate to reach out to me at gosar95@gmail.com. 1. Projector (for IST and Smalltalk Demo) 3 parts are used together. 1.1 Python strawberry-ros-projector Person in charge: Lithin, Matija Description: Here we are subscribing to the dialog_result or other rostopic with converation information, convert that data to a json output and post on localhost with port 1706.http://localhost:1706/ For json data http://localhost:3000/ For Frontend app that shows chat info You can show the Frontend on Unity by Press CTRL + w keys or by publishing the above URL on rostopic /unity_projector/conversationApp/set

5/18/2023
All Haru Demos

Status Report Demo Name HRI UPO Fix needed Feature needed Overall :bangbang: :bangbang:

4/20/2023
Read more from Jitesh Gosar
Published on HackMD