Unity and ML-Agents

• Unity and ML-Agents
  • Install
    • Install Unity
    • Install ML Agents
  • Make your Environment in Unity
  • Train
    • Config
    • Training command
  • Test Examples - 3D Ball
    • Setup
    • Test the pretrained model
    • Train
    • Test your trained model
  • BallJump
    • Objective
    • Setup
    • Unity Environment
    • Agent Logic Script
    • Observations (9)
    • Actions (2; discrete)
    • Rewards
    • Training
    • Tensorboard result
  • Humingbird
    • Project Setup
    • C# scripts
    • RL
    • Scene Setup
    • Tensorboard result
  • 3D Ball
    • Observations (8)
    • Actions (2; continuous)
    • Rewards
  • Basic
    • Observations (3)
    • Actions (1; discrete)
    • Rewards
  • Bouncer
    • Observations (6)
    • Actions (3; continuous)
    • Rewards
  • Crawler
    • Observations (126)
    • Actions (20; continuous)
    • Rewards
  • ToDo
    • Observations (6)
    • Actions (3; continuous)
    • Rewards

Install

Install Unity

• Ref
• Download and install Unity Hub and open it
• Go to the install tab and install Unity (preferably choose LTS version)

Install ML Agents

• Go to the github link
• Download the "Verified Package 1.0.7" from the following,

  

Make your Environment in Unity

Show details

Open Unity Hub and click "NEW"

Select 3D and enter project name and location. Then click CREATE button.

Train

Config

Ref

Training command

After opening Unity scene, check if the behavior name is same as the name inside config ".yaml" file.
Check if Behaviour type is set to default.

Test Examples - 3D Ball

Setup

• Open Unity and install MLAgents from Window -> Package Manager -> MLAgents -> Install
  
• Open 3DBall Scene from the assets folder.
  
• The screen should look like this
  

Test the pretrained model

• Click on the play button to see pretrained model's inference result balancing the ball on the cube.

Train

• Train the 3Dcube by entering the following comand in the console.
  mlagents-learn config/trainer_config.yaml --run_id ANY_NAME --train
• It will create a folder named 'ANY_NAME' and store training results like tensorboard log info, weight file (.nn) in this folder.

Test your trained model

• To test the training output, drag the .nn file to Model input in Behavior Parameters component of Agent in Unity.
  
• Finally click on play button to check the inference result.

BallJump

Objective

Move Ball to the target box in 3D space.

Setup

Install MLAgents and ProBuilder package into Unity from Window -> Package Manager

Unity Environment

Your environment should include an agent with components:

• Collider component (box, shpere or mesh)
• Rigid body: To get affected by forces of the environment
• Decision Requester component
• Agent Logic Script component (Behaviour Parameters Component is added automatically with this component)
  The screenshot of the BallJump environment I created is shown below.
  

Agent Logic Script

• The complete script of Agent Logic is here.
  BallAgentLogic.cs

  ​​​​using UnityEngine;
  ​​​​using Unity.MLAgents;
  ​​​​using Unity.MLAgents.Sensors;
  
  ​​​​public class BallAgentLogic : Agent
  ​​​​{
  
  ​​​​    Rigidbody rBody;
  
  ​​​​    // // Start is called before the first frame update
  ​​​​    void Start()
  ​​​​    {
  ​​​​        rBody = GetComponent<Rigidbody>();
  ​​​​    }
  
  ​​​​    public Transform target;
  
  ​​​​    // Update is called once per frame
  ​​​​    // void Update()
  ​​​​    // {
  
  ​​​​    // }
  
  ​​​​    public override void OnEpisodeBegin()
  ​​​​    {
  ​​​​        this.rBody.angularVelocity = Vector3.zero;
  ​​​​        this.rBody.velocity = Vector3.zero;
  ​​​​        this.transform.localPosition = new Vector3(-9, 0.5f, 0);
  
  ​​​​        target.localPosition = new Vector3(12 + Random.value * 8, Random.value * 3, Random.value * 10 - 5);
  
  ​​​​    }
  
  ​​​​    public override void CollectObservations(VectorSensor sensor)
  ​​​​    {
  ​​​​        sensor.AddObservation(target.localPosition);
  ​​​​        sensor.AddObservation(this.transform.localPosition);
  ​​​​        sensor.AddObservation(rBody.velocity);
  
  ​​​​    }
  
  ​​​​    public float speed = 20;
  ​​​​    public override void OnActionReceived(float[] vectorAction)
  ​​​​    {
  ​​​​        Vector3 controlSignal = Vector3.zero;
  ​​​​        controlSignal.x = vectorAction[0];
  
  ​​​​        if (vectorAction[1] == 2)
  ​​​​        {
  ​​​​            controlSignal.z=1;
  ​​​​        }
  ​​​​        else
  ​​​​        {
  ​​​​            controlSignal.z = -vectorAction[1];
  ​​​​        }
  
  ​​​​        if (this.transform.localPosition.x < 8.5)
  ​​​​        {
  ​​​​            rBody.AddForce(controlSignal * speed);
  ​​​​        }
  
  ​​​​        float distanceToTarget = Vector3.Distance(this.transform.localPosition, target.localPosition);
  
  ​​​​        if (distanceToTarget < 1.42f)
  ​​​​        {
  ​​​​            SetReward(1.0f);
  ​​​​            EndEpisode();
  ​​​​        }
  
  ​​​​        if (this.transform.localPosition.y < 0)
  ​​​​        {
  ​​​​            EndEpisode();
  ​​​​        }
  ​​​​    }
  
  ​​​​    public override void Heuristic(float[] actionsOut)
  ​​​​    {
  ​​​​        actionsOut[0] = Input.GetAxis("Vertical");
  ​​​​        actionsOut[1] = Input.GetAxis("Horizontal");
  ​​​​    }
  ​​​​}
  
  

• We need to import this 3 classes,
  ​​​​using UnityEngine;
  ​​​​using Unity.MLAgents;
  ​​​​using Unity.MLAgents.Sensors;
  

• Make a custom class by inheriting Agents class from MLAgents.
• Overwrite the following 3 methods from the Agents class
  • OnEpisodeBegin
  • CollectObservations
  • OnActionReceived
  • Heuristic (Not necessary to write): This function allows us to control the agents using keyboard or any way other than Neural Network.

Observations (9)

1. Target's location (3 observation)
2. Agent's location (3 observation)
3. Agent's velocity (3 observation)

Actions (2; discrete)

1. move vector x (+1 = right, -1 = left)
2. move vector z (+1 = forward, -1 = backward)

Rewards

1. Agent touches the target: +1

Training

Config parameters

BallAgent:
    trainer: ppo
    batch_size: 128
    beta: 1.0e-3
    buffer_size: 5000
    epsilon: 0.15
    hidden_units: 128
    lambd: 0.92
    learning_rate: 3.0e-4
    learning_rate_schedule: linear
    max_steps: 2.0e6
    normalize: true
    num_epoch: 3
    num_layers: 3
    time_horizon: 128
    summary_freq: 50000
    use_recurrent: false
    reward_signals:
        extrinsic:
            strength: 1.0
            gamma: 0.99

Shell command
mlagents-learn PATH_TO_CONFIG --run-id RUN_ID --train

Tensorboard result

See Graphs

Humingbird

Project Setup

• Download
• Create new project in Unity Hub by selecting "Universal Render Pipeline"
• Drag the downloaded Humingbird scene to Asset folder inside Unity
• Double click the Training file inside the folder Humingbird/Scenes
• (Optional) Delete the folders "Example", "Scenes", "Tutorial" and a file named "Readme" inside the Assets folder. (Don't delete anything inside Hummingbird folder)
• Go to "edit/Project Settings/Physics", change Default Contact Offset from 0.01 to 0.001 so that the collision works better in this project.
• Fix lighting by going to
  • "Windows/Rendering/Lighting Settigs", click "Generate Lighting" button.
  • Assets/Materials, drag Skybox_Mat to "Skybox Material" inside Lighting/Environment.

C# scripts

• Reset flower
• Set nectar-extract speed
• Turns from red to purple after nectar is extracted
• Search flower_plant tag
• Get flowerNector
• Find the nearest flower
• Calculate the distance between Bird's beak-tip and the nearest flower
• Bird move to safe random position when episode starts
• freeze the bird
• Triggers if bird enter nectar area
• Heuristic: keybord controls

RL

Observations (10)

1. Bird's rotation (4 observations)
2. Distance between Bird's beak-tip and the nearest flower (3 observation)
3. Check if Bird's beak-tip is in front of flower or not (-1 to 1)
4. Check if Bird's beak-tip is pointing towards the flower or not (-1 to 1)
5. Relative distance from the Bird's beak-tip and the nearest flower to the flower area diameter (0 to 1)

Actions (5; continuous)

1. move vector x (+1 = right, -1 = left)
2. move vector y (+1 = up, -1 = down)
3. move vector z (+1 = forward, -1 = backward)
4. pitch angle (+1 = pitch up, -1 = pitch down)
5. yaw angle (+1 = turn right, -1 = turn left)

Rewards

1. Inside nectar area: 0.1
2. Inside nectar area and drinking it: 0.2
3. Colliding boundary: -0.5

Scene Setup

Load FloatingIsland from Prefabs

• Create 3 tags in Project Settings, then
  • Add boundary tag in the IslandBoundaries
  • Add nectar tag in the FlowerNectarCollider
  • Add flower_plant tag in the FlowerPlant
• Add script components
  • Add FlowerArea script as a component to FloatingIsland
  • Add Flower script as a component to Flower inside FlowerBud
  • Add Hummingbird script as a component to Flower inside FlowerBud
• Add DecisionRequester component; 5 steps
• HummingbirdAgent script and Behaviour component
• Ray perseption
• Heuristic

Tensorboard result

3D Ball

Observations (8)

1. rotation.z
2. rotation.x
3. ball.transform.position - gameObject.transform.position
4. ball.velocity

Actions (2; continuous)

1. rotate along z axis
2. rotate along x axis

Rewards

1. -1
   • |ball.position.y - agent.position.y|
   • |ball.position.y - agent.position.y|
   • |ball.position.y - agent.position.y|
2. 0.1

Basic

Observations (3)

1. Agent position
2. small goal position
3. big goal position

Actions (1; discrete)

1. 1 axis direction

Rewards

1. -0.01: no goal
2. 0.1: small goal
3. 1: big goal

Bouncer

Observations (6)

1. Agent position (3 observation)
2. target position (3 observation)

Actions (3; continuous)

1. x axis force
2. y axis force (scale from 0 to 1)
3. z axis force

Rewards

1. -0.05 x (mean square of actions)
2. -1: Agent.position.y < -1
3. -1: outside the floor
4. +1: Touch the target

Crawler

Observations (126)

1. For each body part (bp)
   1. is touching the ground?
   2. velocityRelativeToLookRotationToTarget
   3. angularVelocityRelativeToLookRotationToTarget
2. For each body part except center-body
   1. localPosRelToBody
   2. bp.currentXNormalizedRot
   3. bp.currentYNormalizedRot
   4. bp.currentZNormalizedRot
   5. strength ratio (current strength/ max strength)

Actions (20; continuous)

Rewards

ToDo

Observations (6)

Actions (3; continuous)

Rewards