Unity Input for Depth Sensor Data

--- Title: Unity Input for Depth Sensor Data tags: MKE description: Research the new input system and other methods then try to look if a standard API can be made to take input in unity for depth sensor data --- # Unity Input for Depth Sensor Data Index [TOC] ## Research the Controller input and other inputs for Unity There are 2 main types of input system both which can be used in unity and they can both be used simultaneously. ### 1. [Input Manager(Native Input System)](https://docs.unity3d.com/Manual/class-InputManager.html) This what we have been using natively in unity which can be used as follows ```UnityEngine.Input.GetKey(KeyCode.D)``` for if the key d is pressed as such that below is how we can add WASD movement ![image](https://hackmd.io/_uploads/B139mUZwR.png) :::warning The **Input Manager** uses the following types of controls: - **Key** refers to any key on a physical keyboard, such as W, Shift, or the space bar. - **Button** refers to any button on a physical controller (for example, gamepads), such as the X button on a remote control. - A **virtual axis** (plural: **axes**) is mapped to a control, such as a button or a key. When the user activates the control, the axis receives a value in the range of \[–1..1\]. You can use this value in your **scripts**. ::: ### 2. [Input System(New Input System)](https://docs.unity3d.com/Manual/class-InputManager.html) This **Input System package** is a newer, more flexible system, which allows you to use any kind of Input Device to control your Unity content. It's intended to be a replacement for Unity's classic Input Manager. To use it, you must [install it into your project using the Package Manager](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/Installation.html). In this system all we have to do is tell it what type of input do we need and connect it to the function it has call As you can see above it takes all kinds of input like controllers, joysticks, mouse, keyboards etc. ![Screenshot 2024-07-01 104136](https://hackmd.io/_uploads/B1HDV8WP0.png) In the image below I have image i have made 5 different actions which are **Reset**,**Grab**, **DrawToMovements**,**DepthMovement**,**Dis&Ang**. ![image](https://hackmd.io/_uploads/H1BsYGaP0.png) As seen above:- - **Reset** takes input from **R(Keyboard)** and **Gamepad south button(PS - x,Xbox-a)** - **Grab** takes input from **G(Keyboard)** and **Gamepad Right Shoulder Button** - **DrawToMovements** takes **WASD keyboard keys** and **Gamepad Left and Right Joystick** - **DepthMovement** takes input from **Z,X(Keyboard)** and **Gamepad Direction buttons** - **Dis&Ang** takes input from **T(Keyboard)** and **Gamepad Left Shoulder Button** These input are taken through the following code:- ```csharp= public void Fgrab() { //use3DInput = true; grab = !grab; } private void Awake() { playerInput = GetComponent<PlayerInput>(); main_Controls = new Main_Controls(); main_Controls.Enable(); } public void Handle_v2_composite() { Vector2 wasd = main_Controls.MAIN.DrawToMovement.ReadValue<Vector2>(); input3DCoordinates = new Vector3( input3DCoordinates.x + (wasd.x * slowcontrol), input3DCoordinates.y + (wasd.y * slowcontrol), input3DCoordinates.z); } public void Handle_depth() { Vector2 depth = main_Controls.MAIN.DepthMovement.ReadValue<Vector2>(); input3DCoordinates = new Vector3( input3DCoordinates.x, input3DCoordinates.y , input3DCoordinates.z + (depth.y * DepthControl)); } ``` * Important Links :- - [How to use Input System with good examples](https://youtu.be/Yjee_e4fICc?si=-SJMixOPqkHHqAiQ) - [Unity Input System Workflow](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/Workflows.html) - [New Input System documentation](https://docs.unity3d.com/Packages/com.unity.inputsystem@1.8/manual/index.html) ## Research the standard API method and check it's necessity [Github - AnythingAsAController](https://github.com/rbrower28/AnythingAsAController/blob/main/controller_out.py) The above link contains a code to allow to take input from anything and emulates as a Xbox 360 controller [Gitbub - vgamepad](https://github.com/yannbouteiller/vgamepad?tab=readme-ov-file) The above link contains a code to allow to take input from anything and emulates as a Xbox controller or PS4 controller from that i have written the below codes in `PYTHON` ### 1. This script maps the keyboard keys as xbox controller output ```python import vgamepad as vg import keyboard inputs = [] input_map = { "w": "LUP", "s": "LDOWN", "a": "LLEFT", "d": "LRIGHT", "up": "RUP", "down": "RDOWN", "left": "RLEFT", "right": "RRIGHT", "z": "DUP", "x": "DDOWN", "c": "DLEFT", "b": "DRIGHT", "r": "A", "l": "B", "i": "X", "o": "Y", "t": "L1", "g": "R1", "u": "L2", "p": "R2", "f": "L3", "v": "R3", "e": "START", "q": "SELECT" } class Controller: """ Emulates an Xbox 360 controller. Recognized inputs: A B X Y LUP LDOWN LLEFT LRIGHT RUP RDOWN RLEFT RRIGHT DUP DDOWN DLEFT DRIGHT L1 L2 L3 R1 R2 R3 START SELECT """ def __init__(self): # controller obj self.gamepad = vg.VX360Gamepad() # current key memory self.keys_down = set() # left thumb stick self.xL = 0.0 self.yL = 0.0 # right thumb stick self.xR = 0.0 self.yR = 0.0 # triggers self.L2 = 0.0 self.R2 = 0.0 self.button_map = { "A": vg.XUSB_BUTTON.XUSB_GAMEPAD_A, "B": vg.XUSB_BUTTON.XUSB_GAMEPAD_B, "X": vg.XUSB_BUTTON.XUSB_GAMEPAD_X, "Y": vg.XUSB_BUTTON.XUSB_GAMEPAD_Y, "DUP": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_UP, "DDOWN": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_DOWN, "DLEFT": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_LEFT, "DRIGHT": vg.XUSB_BUTTON.XUSB_GAMEPAD_DPAD_RIGHT, "START": vg.XUSB_BUTTON.XUSB_GAMEPAD_START, "SELECT": vg.XUSB_BUTTON.XUSB_GAMEPAD_BACK, "L1": vg.XUSB_BUTTON.XUSB_GAMEPAD_LEFT_SHOULDER, "R1": vg.XUSB_BUTTON.XUSB_GAMEPAD_RIGHT_SHOULDER, "L3": vg.XUSB_BUTTON.XUSB_GAMEPAD_LEFT_THUMB, "R3": vg.XUSB_BUTTON.XUSB_GAMEPAD_RIGHT_THUMB } self.stick_map = { "LUP": (0.0, 1.0), "LDOWN": (0.0, -1.0), "LLEFT": (-1.0, 0.0), "LRIGHT": (1.0, 0.0), "RUP": (0.0, 1.0), "RDOWN": (0.0, -1.0), "RLEFT": (-1.0, 0.0), "RRIGHT": (1.0, 0.0), } def handle_input(self, controls): for item in self.button_map.keys(): if item in controls: if item not in self.keys_down: self.gamepad.press_button(self.button_map[item]) self.keys_down.add(item) else: if item in self.keys_down: self.gamepad.release_button(self.button_map[item]) self.keys_down.remove(item) for item in self.stick_map.keys(): if item in controls: if item[0] == "L": if item not in self.keys_down: self.xL += self.stick_map[item][0] self.yL += self.stick_map[item][1] self.keys_down.add(item) elif item[0] == "R": if item not in self.keys_down: self.xR += self.stick_map[item][0] self.yR += self.stick_map[item][1] self.keys_down.add(item) else: if item[0] == "L": if item in self.keys_down: self.xL -= self.stick_map[item][0] self.yL -= self.stick_map[item][1] self.keys_down.remove(item) elif item[0] == "R": if item in self.keys_down: self.xR -= self.stick_map[item][0] self.yR -= self.stick_map[item][1] self.keys_down.remove(item) if "L2" in controls: if "L2" not in self.keys_down: self.L2 = 1.0 self.keys_down.add("L2") else: if "L2" in self.keys_down: self.L2 = 0.0 self.keys_down.remove("L2") if "R2" in controls: if "R2" not in self.keys_down: self.R2 = 1.0 self.keys_down.add("R2") else: if "R2" in self.keys_down: self.R2 = 0.0 self.keys_down.remove("R2") # apply changes to controller self.gamepad.left_joystick_float(x_value_float=self.xL, y_value_float=self.yL) self.gamepad.right_joystick_float(x_value_float=self.xR, y_value_float=self.yR) self.gamepad.left_trigger_float(value_float=self.L2) self.gamepad.right_trigger_float(value_float=self.R2) self.gamepad.update() controller = Controller() while not keyboard.is_pressed("esc"): for item in input_map.keys(): if keyboard.is_pressed(item): inputs.append(input_map[item]) controller.handle_input(inputs) inputs.clear() print(">>>>> Program finished. >>>>>") ``` ### 2. This script has the option to take in the input in the form of list [grab, x, y, z] but currently I am creating some random values to test it ```python import vgamepad as vg import keyboard import time import random # Create a virtual PS4 controller gamepad = vg.VDS4Gamepad() time.sleep(5.0) def update_controller(toMove, grab_changed): # Move the left analog stick gamepad.left_joystick(x_value=toMove[1], y_value=toMove[2]) # Handle D-pad inputs if toMove[3] > 0: gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_NORTH) elif toMove[3] < 0: gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_SOUTH) else: gamepad.directional_pad(vg.DS4_DPAD_DIRECTIONS.DS4_BUTTON_DPAD_NONE) # Handle shoulder button input based on grab state change if grab_changed: gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_RIGHT) gamepad.update() time.sleep(0.05) # Small delay to simulate press-release action gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_RIGHT) # Handle keyboard inputs if keyboard.is_pressed('r'): gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_CROSS) else: gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_CROSS) if keyboard.is_pressed('t'): gamepad.press_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_LEFT) else: gamepad.release_button(vg.DS4_BUTTONS.DS4_BUTTON_SHOULDER_LEFT) # Update the virtual gamepad to apply the changes gamepad.update() def generate_random_input(): grab = random.choice([True, False]) x_coord = random.randint(-32768, 32767) y_coord = random.randint(-32768, 32767) z_coord = random.choice([-1, 0, 1]) return [grab, x_coord, y_coord, z_coord] # Initialize initial and current input values iniVal = generate_random_input() curVal = generate_random_input() try: while True: # Update current values curVal = generate_random_input() # Determine if grab state has changed grab_changed = curVal[0] != iniVal[0] # Calculate the difference toMove = [ curVal[0] if grab_changed else iniVal[0], curVal[1], curVal[2], curVal[3] ] # Handle input based on the difference and grab state change update_controller(toMove, grab_changed) # Update initial values for next iteration iniVal = curVal[:] time.sleep(0.3) # Adjust the frequency as needed except KeyboardInterrupt: print("Exiting...") ``` > [name=Shrey Khetan] > [time= Started on Mon, Jul 1 2024, 11:12 AM]