###### tags: `sdk-documentation`, `ur`, `ai`

# Ur - Hand Tracker
Ur provides real-time tracking of 3D hand poses from monocular RGB video in mobile phones. Tracking and reconstructing the 3D pose and geometry of multiple hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, hand gesture control, or sign language recognition. In order to address the inherent depth ambiguities in RGB data, we propose a novel lite CNN network that regresses camera pose, 3D hand dense mesh (optional), and 3D landmark positions.

## Obtaining and configuring the Hand Tracker
The `com.aukilabs.unity.ur` Unity SDK plugin that contains the hand tracker can be found at our [npm repository](https://npm.aukiverse.com/-/web/detail/com.aukilabs.unity.ur)
Assuming you followed the *SDK Setup* from the *Auki SDK documentation's User Manual* section, you should be able to find and install the Ur SDK plugin in Unity package manager.

// TODO(JB): Insert the "Please make sure your development environment is set up by following the [User manual](https://documentation.dev.aukiverse.com/user-manual)" Note here.

## Concepts

### Landmark
There are 21 landmarks per hand. A landmark is a 3D vector (x,y,z) that holds the local translation for each hand bone joint.
There is a root landmark that define the palm of the hand joint and another 20 joints that define the tree-like layout of all the hand/fingers joints.


### Translation Landmark
For each hand there is a translation, another 3D vector (x,y,z), with contains the global translation of the hand in camera space. This basically means that all hand landmarks need to displaced by this value.

## Sample code
We start by importing the Auki.Ur package that contains the hand tracker:

`using Auki.Ur;`

and instantiate a new HandTracker:

`HandTracker _handTracker;`

`HandTracker` is a singleton. You can get the instance with:

`_handTracker = HandTracker.GetInstance();`

In order to work, the hand tracker requires access to the currently running `ARSession`.
For an Auki SDK based application, you can get the current `ARSession` like this:

var rig = _auki.GetUnityBridge().gameObject;
_arSession = rig.GetComponentInChildren<ARSession>();

Now the only left to do is to start the hand tracker:

`_handTracker.Start(_arSession);`

If all goes well, this method will return `true` and the hand tracker is now processing frames and detecting hand landmarks.

Whenever you want to stop the hand tracker you call:

`_handTracker.Stop();`

### Displaying an helper hand "mesh"
It is possible to verify that all is working by enabling(disabling) showing the helper hand mesh with:

_handTracker.ShowHandMesh();
_handTracker.HideHandMesh();

### Accessing the hand landmarks
To access directly the current hand landmarks we can call the following methods of `HandTracker` class:

float[] Get3DLandmarks();
float[] GetTranslations();

We can also register a call back that is called whenever new landmarks are detected. This is the recommend way to access the landmarks:

_handTracker.OnUpdate += (landMarks, translations, isRightHand, score) =>  
{  
	// landmarks processing code:
};