Try   HackMD

Encouraging Mediator

Run Commands

Setup

rosrun haru_encouraging_mediator setup.sh

Run the app



roslaunch haru_encouraging_mediator master.launch number_of_ipads:=3
rosrun haru_encouraging_mediator check_setup_and_connectivity_asr_ipad.sh

Run the BT for Japan

roslaunch haru_encouraging_mediator initiation.launch this_country:=japan episode:=2

Run the BT for Australia

roslaunch haru_encouraging_mediator initiation.launch this_country:=australia episode:=2

Setup config

Japan side

  • local-ext-ip: 192.168.8.222
  • local-robot-ip: 192.168.8.247

Japan - Australia

JAPAN			
External PC			
	hostname	haru-remote-ext	VPN IP
	VPN ID	9	10.0.0.37
Robot/Avatar			
	hostname	haru-47	VPN IP
	VPN ID	10	10.0.0.41
Remote External PC			
	hostname	ext-pc	VPN IP
	VPN ID	7	10.0.0.29
Remote Robot/Avatar			
	hostname	haru-0	VPN IP
	VPN ID	8	10.0.0.33
			
			
AUSTRALIA			
External PC			
	hostname	ext-pc	VPN IP
	VPN ID	7	10.0.0.29
Robot/Avatar			
	hostname	haru-0	VPN IP
	VPN ID	8	10.0.0.33
Remote External PC			
	hostname	haru-remote-ext	VPN IP
	VPN ID	9	10.0.0.37
Remote Robot/Avatar			
	hostname	haru-47	VPN IP
	VPN ID	10	10.0.0.41

Japan - Spain

JAPAN			
External PC			
	hostname	haru-remote-ext	VPN IP
	VPN ID	13	10.0.0.53
Robot/Avatar			
	hostname	haru-25	VPN IP
	VPN ID	14	10.0.0.57
Remote External PC			
	hostname	ext-pc-one (haru-local-ext)	VPN IP
	VPN ID	11	10.0.0.45
Remote Robot/Avatar			
	hostname	haru-0 (haru-local-avatar)	VPN IP
	VPN ID	12	10.0.0.49
			
			
SPAIN (aus)			
External PC			
	hostname	ext-pc-one	VPN IP
	VPN ID	11	10.0.0.45
Robot/Avatar			
	hostname	haru-0	VPN IP
	VPN ID	12	10.0.0.49
Remote External PC			
	hostname	haru-remote-ext	VPN IP
	VPN ID	13	10.0.0.53
Remote Robot/Avatar			
	hostname	haru-25	VPN IP
	VPN ID	14	10.0.0.57

Branches

  • haru-encouraging-mediator -> master
  • haru-encouraging-mediator-msgs -> master
  • behavior-tree-plugins -> master
  • haru-unity -> testing-…

Tests

Test on 2023-03-13

In Today's demo-tests, everything worked fine other than the following,

Test no. Japan side issues Australia side issues
1 Dialog about the photos did not start Only the right screen worked but image proportions were wrong
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
2 same as above Only the right screen worked perfectly fine
3 same as above Both screen worked but left screen was zoomed in
4 same as above Both screen worked but left screen was zoomed in

Test on 2023-03-23

In Today's demo-tests,

  1. In the first test we had an error RL execption after launching the master.sh file. The error went away after running the setup.sh script. I think the problem came because the ip of the external-pc changed since last time and the router did not have a fixed ip for external-pc.
  2. we had only 1 complete test, which showed same result as test on 2023-03-13, where everything worked except
    • On Japan side, Dialog about the photos did not start
    • On Australia side, left screen projector was zoomed in

Test on 2023-03-24

In Today's demo-tests,

Test no. Japan side issues Australia side issues
1 fkie failed 4 times fkie failed 4 times
2 fkie failed 4 times fkie failed 4 times
3 did not continue the conversation about sports (aus tree) one screen projector was zoomed in

Points to discuss:

  1. japan side is mute in the beginning, we should unmute all side for testing
  2. videoconference screen changes from left to right, it should be consistent and not change sides
  3. the 3-3 user profile screen on the projector does not look symmetrical enough
  4. Dialog does not start on the japan side, should we update any package?

Test on 2023-03-27

Test on 2023-03-28

Test on 2023-03-29

  • Japan side robot's eyes are not working today
  • The Haru website is not working
Test no. Japan side issues Australia side issues
1 Dialog: Haru asked about famous sports player twice -

Test on 2023-03-30

Test on 2023-03-31

Test no. ROS bag File name Description Review
1 encouraging_japan_2023-03-31-10-44-23.bag Test; connection error while Haru was talking about the 2nd or 3rd picture on Japan side
fail
2 encouraging_japan_2023-03-31-11-05-25.bag.active Test pass
3 encouraging_japan_2023-03-31-13-51-58.bag Test pass
4 encouraging_japan_2023-03-31-15-40-12.bag Demo round 1 pass
5 encouraging_japan_2023-03-31-16-24-23.bag Demo round 2; 1 ipad on japan side did not show page to select profile pics; fail
6 encouraging_japan_2023-03-31-16-24-34.bag Demo round 2; Australia side could not connect with the "Whereby room"; we waited for a minute before restarting fail
7 encouraging_japan_2023-03-31-16-35-42.bag Demo round 2 pass

Suggestions from the kids:

  1. replace mute icon with text like "Japan turn", "Australia turn", "mute" and "Talking Room"
  2. If possible, don't mute single side.
  3. Add translation for Japanese kids as subtitle
  4. Have some English keywords text display on the screen to help kids construct a sentence to say
  5. Connect Haru with other Honda products like AC and Cars; IOT
  6. Give kids simulation or some way to improve Haru features, one kid really want to make Haru do stuff with his program
  7. Have Haru teach English

Other suggestions:

  1. When Haru says something on other side, that side is mute…
  2. Pick up some keywords and haru have some routine
  3. Wait for kids to stop talking. Haru kept interrupting the kids. We can use ASR to detect when kids stopped talking for 5 seconds, then make Haru say something
  4. Launch everything with one command smoothly
  5. Reduce the volume of privacy-video while Haru talks. There is already a rostopic that we can use to resuce it.

Test OLD

Test type Details
1 8 MB; uncompressed images both sides
2 uncompressed images on Australia, compressed images on Japan side
3 1.1 MB; compressed images both sides
4 50kb images both sides
Test No. Test type Result
1 1 fail on japan side
2 2 Success
3 2 fail on Australia side
4 2 Success
5 2 fail on Australia side
6 2 Success
7 2 Success
8 4 Success
9 4 Success
10 4 Success
11 4 Success

Appendix

TTS

To run TTS on external PC, add the following lines in your testbed.


    <include file="$(find idmind_tabletop)/launch/idmind_tabletop_audio_controller.launch">
        <arg name="robot_namespace"             value="" />
        <arg name="sounds_filepath"             value="$(find idmind_tabletop)/resources/audio/" />
        <arg name="device_id"                   value="-1" />
        <arg name="get_volume_method"           value="1" />
    </include>
    <include file="$(find idmind_tabletop)/launch/idmind_tabletop_tts.launch">
        <arg name="robot_namespace"             value=""/>
        <arg name="play_audio"                  value="streamed"/>
        <arg name="voices_path"      default="/opt/ros/noetic/share/idmind_tabletop/resources/voice/" />
        <arg name="voices"           default="andy_cerewave.voice;sara.voice;sarah.voice;haru.voice;yuki.voice;jordan.voice;haruchild.voice"  />
        <arg name="license"          default="/opt/ros/noetic/share/idmind_tabletop/1326C40F-9F8B-44B2-99A6-EE88381E3988_ASH_license.lic" />
        <arg name="default_language" default="en-US" />
        <arg name="file_out"         default="/tmp/tts_audio.wav" />
    </include>
Last changed by 
 
Jitesh Gosar
0
401

Read more

Conda

Activate Conda for the Current Session: Use the command provided in the installation message, but replace YOUR_SHELL_NAME with the name of your shell (e.g., bash, zsh). For example, if you&#39;re using bash:

Mar 5, 2024
Untitled

Health and Safety

Dec 21, 2023
Docs: Handover by Jitesh

Much of the work I have completed can be found documented in the README files within the respective repositories or on this webpage. If the Person in charge for overseeing the project (excluding myself) is unavailable for clarification, please do not hesitate to reach out to me at gosar95@gmail.com. 1. Projector (for IST and Smalltalk Demo) 3 parts are used together. 1.1 Python strawberry-ros-projector Person in charge: Lithin, Matija Description: Here we are subscribing to the dialog_result or other rostopic with converation information, convert that data to a json output and post on localhost with port 1706.http://localhost:1706/ For json data http://localhost:3000/ For Frontend app that shows chat info You can show the Frontend on Unity by Press CTRL + w keys or by publishing the above URL on rostopic /unity_projector/conversationApp/set

May 18, 2023
All Haru Demos

Status Report Demo Name HRI UPO Fix needed Feature needed Overall :bangbang: :bangbang:

Apr 20, 2023
Read more from Jitesh Gosar
Published on HackMD