---
# System prepended metadata

title: Unity Input for Depth Sensor Data
tags: [MKE]

---

---
Title: Unity Input for Depth Sensor Data
tags: MKE
description: Research the new input system and other methods then try to look if a standard API can be made to take input in unity for depth sensor data 
---


# Unity Input for Depth Sensor Data

Index
[TOC]

## Research the Controller input and other inputs for Unity
There are 2 main types of input system both which can be used in unity and they can both be used simultaneously.

### 1. [Input Manager(Native Input System)](https://docs.unity3d.com/Manual/class-InputManager.html)
This what we have been using natively in unity which can be used as follows
```UnityEngine.Input.GetKey(KeyCode.D)``` for if the key d is pressed
as such that below is how we can add WASD movement
![image](https://hackmd.io/_uploads/B139mUZwR.png)

:::warning
The **Input Manager** uses the following types of controls:
-   **Key** refers to any key on a physical keyboard, such as W, Shift, or the space bar.
-   **Button** refers to any button on a physical controller (for example, gamepads), such as the X button on a remote control.
-   A **virtual axis** (plural: **axes**) is mapped to a control, such as a button or a key. When the user activates the control, the axis receives a value in the range of \[–1..1\]. You can use this value in your **scripts**.
:::

### 2. [Input System(New Input System)](https://docs.unity3d.com/Manual/class-InputManager.html)

This **Input System package** is a newer, more flexible system, which allows you to use any kind of Input Device to control your Unity content. It's intended to be a replacement for Unity's classic Input Manager. To use it, you must [install it into your project using the Package Manager](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/Installation.html).


In this system all we have to do is tell it what type of input do we need and connect it to the function it has call

As you can see above it takes all kinds of input like controllers, joysticks, mouse, keyboards etc.
![Screenshot 2024-07-01 104136](https://hackmd.io/_uploads/B1HDV8WP0.png)


In the image below I have image i have made 5 different actions which are **Reset**,**Grab**, **DrawToMovements**,**DepthMovement**,**Dis&Ang**.


![image](https://hackmd.io/_uploads/H1BsYGaP0.png)

As seen above:-
- **Reset** takes input from **<u>R(Keyboard)</u>** and **<u>Gamepad south button(PS - x,Xbox-a)</u>** 
- **Grab** takes input from **<u>G(Keyboard)</u>** and **<u>Gamepad Right Shoulder Button</u>**
- **DrawToMovements** takes **<u>WASD keyboard keys</u>** and **<u>Gamepad Left and Right Joystick</u>**
- **DepthMovement** takes input from **<u>Z,X(Keyboard)</u>** and **<u>Gamepad Direction buttons</u>** 
- **Dis&Ang** takes input from **<u>T(Keyboard)</u>** and **<u>Gamepad Left Shoulder Button</u>**

These input are taken through the following code:-
```csharp=
public void Fgrab()
{
    //use3DInput = true;
    grab = !grab;
}
private void Awake()
{
    playerInput = GetComponent<PlayerInput>();

    main_Controls = new Main_Controls();
    main_Controls.Enable();
}
public void Handle_v2_composite()
{
    Vector2 wasd = main_Controls.MAIN.DrawToMovement.ReadValue<Vector2>();
    input3DCoordinates = new Vector3(
        input3DCoordinates.x + (wasd.x * slowcontrol),
        input3DCoordinates.y + (wasd.y * slowcontrol),
        input3DCoordinates.z);
}
public void Handle_depth()
{
    Vector2 depth = main_Controls.MAIN.DepthMovement.ReadValue<Vector2>();
    input3DCoordinates = new Vector3(
        input3DCoordinates.x,
        input3DCoordinates.y ,
        input3DCoordinates.z + (depth.y * DepthControl));
}
```

* Important Links :-
    - [How to use Input System with good examples](https://youtu.be/Yjee_e4fICc?si=-SJMixOPqkHHqAiQ)
    - [Unity Input System Workflow](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/Workflows.html)
    - [New Input System documentation](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/index.html)

## Research the standard API method and check it's necessity

[Github - AnythingAsAController](https://github.com/rbrower28/AnythingAsAController/blob/main/controller_out.py)
The above link contains a code to allow to take input from anything and emulates as a Xbox 360 controller

[Gitbub - vgamepad](https://github.com/yannbouteiller/vgamepad?tab=readme-ov-file)
The above link contains a code to allow to take input from anything and emulates as a Xbox controller or PS4 controller

from that i have written the below codes in `PYTHON`
### 1. This script maps the keyboard keys as xbox controller output

```python
import vgamepad as vg
import keyboard

inputs = []

input_map = {
    "w": "LUP",
    "s": "LDOWN",
    "a": "LLEFT",
    "d": "LRIGHT",

    "up": "RUP",
    "down": "RDOWN",
    "left": "RLEFT",
    "right": "RRIGHT",

    "z": "DUP",
    "x": "DDOWN",
    "c": "DLEFT",
    "b": "DRIGHT",

    "r": "A",
    "l": "B",
    "i": "X",
    "o": "Y",

    "t": "L1",
    "g": "R1",
    "u": "L2",
    "p": "R2",

    "f": "L3",
    "v": "R3",

    "e": "START",
    "q": "SELECT"
}

class Controller:
    """ Emulates an Xbox 360 controller.
        Recognized inputs:
        A B X Y
        LUP LDOWN LLEFT LRIGHT
        RUP RDOWN RLEFT RRIGHT
        DUP DDOWN DLEFT DRIGHT
        L1 L2 L3
        R1 R2 R3
        START SELECT
    """
    
    def __init__(self):
        # controller obj
        self.gamepad = vg.VX360Gamepad()

        # current key memory
        self.keys_down = set()

        # left thumb stick
        self.xL = 0.0
        self.yL = 0.0

        # right thumb stick
        self.xR = 0.0
        self.yR = 0.0

        # triggers
        self.L2 = 0.0
        self.R2 = 0.0

        self.button_map = {
            "A": vg.XUSB_BUTTON.XUSB_GAMEPAD_A,
            "B": vg.XUSB_BUTTON.XUSB_GAMEPAD_B,
            "X": vg.XUSB_BUTTON.XUSB_GAMEPAD_X,
            "Y": vg.XUSB_BUTTON.XUSB_GAMEPAD_Y,

            "DUP": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_UP,
            "DDOWN": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_DOWN,
            "DLEFT": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_LEFT,
            "DRIGHT": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_RIGHT,

            "START": vg.XUSB_BUTTON.XUSB_GAMEPAD_START,
            "SELECT": vg.XUSB_BUTTON.XUSB_GAMEPAD_BACK,

            "L1": vg.XUSB_BUTTON.XUSB_GAMEPAD_LEFT_SHOULDER,
            "R1": vg.XUSB_BUTTON.XUSB_GAMEPAD_RIGHT_SHOULDER,

            "L3": vg.XUSB_BUTTON.XUSB_GAMEPAD_LEFT_THUMB,
            "R3": vg.XUSB_BUTTON.XUSB_GAMEPAD_RIGHT_THUMB
        }

        self.stick_map = {
            "LUP":    (0.0, 1.0),
            "LDOWN":  (0.0, -1.0),
            "LLEFT":  (-1.0, 0.0),
            "LRIGHT": (1.0, 0.0),

            "RUP":    (0.0, 1.0),
            "RDOWN":  (0.0, -1.0),
            "RLEFT":  (-1.0, 0.0),
            "RRIGHT": (1.0, 0.0),
        }

    def handle_input(self, controls):

        for item in self.button_map.keys():
            if item in controls:
                if item not in self.keys_down:
                    self.gamepad.press_button(self.button_map[item])
                    self.keys_down.add(item)
            else:
                if item in self.keys_down:
                    self.gamepad.release_button(self.button_map[item])
                    self.keys_down.remove(item)

        for item in self.stick_map.keys():
            if item in controls:
                if item[0] == "L":
                    if item not in self.keys_down:
                        self.xL += self.stick_map[item][0]
                        self.yL += self.stick_map[item][1]
                        self.keys_down.add(item)
                elif item[0] == "R":
                    if item not in self.keys_down:
                        self.xR += self.stick_map[item][0]
                        self.yR += self.stick_map[item][1]
                        self.keys_down.add(item)
            else:
                if item[0] == "L":
                    if item in self.keys_down:
                        self.xL -= self.stick_map[item][0]
                        self.yL -= self.stick_map[item][1]
                        self.keys_down.remove(item)
                elif item[0] == "R":
                    if item in self.keys_down:
                        self.xR -= self.stick_map[item][0]
                        self.yR -= self.stick_map[item][1]
                        self.keys_down.remove(item)

        if "L2" in controls:
            if "L2" not in self.keys_down:
                self.L2 = 1.0
                self.keys_down.add("L2")
        else:
            if "L2" in self.keys_down:
                self.L2 = 0.0
                self.keys_down.remove("L2")

        if "R2" in controls:
            if "R2" not in self.keys_down:
                self.R2 = 1.0
                self.keys_down.add("R2")
        else:
            if "R2" in self.keys_down:
                self.R2 = 0.0
                self.keys_down.remove("R2")

        # apply changes to controller
        self.gamepad.left_joystick_float(x_value_float=self.xL, y_value_float=self.yL)
        self.gamepad.right_joystick_float(x_value_float=self.xR, y_value_float=self.yR)
        self.gamepad.left_trigger_float(value_float=self.L2)
        self.gamepad.right_trigger_float(value_float=self.R2)

        self.gamepad.update()

controller = Controller()

while not keyboard.is_pressed("esc"):
    for item in input_map.keys():
        if keyboard.is_pressed(item):
            inputs.append(input_map[item])
            

    controller.handle_input(inputs)
    inputs.clear()

print(">>>>> Program finished. >>>>>")
```

### 2. This script has the option to take in the input in the form of list [grab, x, y, z] but currently I am creating some random values to test it

```python
import vgamepad as vg
import keyboard
import time
import random

# Create a virtual PS4 controller
gamepad = vg.VDS4Gamepad()

time.sleep(5.0)

def update_controller(toMove, grab_changed):
    # Move the left analog stick
    gamepad.left_joystick(x_value=toMove[1], y_value=toMove[2])

    # Handle D-pad inputs
    if toMove[3] > 0:
        gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_NORTH)
    elif toMove[3] < 0:
        gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_SOUTH)
    else:
        gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_NONE)

    # Handle shoulder button input based on grab state change
    if grab_changed:
        gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_RIGHT)
        gamepad.update()
        time.sleep(0.05)  # Small delay to simulate press-release action
        gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_RIGHT)

    # Handle keyboard inputs
    if keyboard.is_pressed('r'):
        gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_CROSS)
    else:
        gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_CROSS)

    if keyboard.is_pressed('t'):
        gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_LEFT)
    else:
        gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_LEFT)

    # Update the virtual gamepad to apply the changes
    gamepad.update()

def generate_random_input():
    grab = random.choice([True, False])
    x_coord = random.randint(-32768, 32767)
    y_coord = random.randint(-32768, 32767)
    z_coord = random.choice([-1, 0, 1])
    return [grab, x_coord, y_coord, z_coord]

# Initialize initial and current input values
iniVal = generate_random_input()
curVal = generate_random_input()

try:
    while True:
        # Update current values
        curVal = generate_random_input()

        # Determine if grab state has changed
        grab_changed = curVal[0] != iniVal[0]

        # Calculate the difference
        toMove = [
            curVal[0] if grab_changed else iniVal[0],
            curVal[1],
            curVal[2],
            curVal[3]
        ]

        # Handle input based on the difference and grab state change
        update_controller(toMove, grab_changed)
        
        # Update initial values for next iteration
        iniVal = curVal[:]

        time.sleep(0.3)  # Adjust the frequency as needed

except KeyboardInterrupt:
    print("Exiting...")

```

> [name=Shrey Khetan] 
> [time= Started on Mon, Jul 1 2024, 11:12 AM]

