Sequence Tracking and the Gesture Engine

Sequence Tracker

The Sequence Tracker is a library I have written that allows for tracking of a sequence of steps over time. It's represented as a 'tree', with branching paths depending on the gestures that are done.

Usage

You start off by creating your root Step. This should be done once only, such as in the Start() or Awake() unity methods.

For example:

public class SomeMono : MonoBehaviour {
    Step root;
    public void Start() {
        root = Step.Start();
    }
}

Step.Start() takes an optional Action parameter - an action that is called whenever a step is successfully completed.

This can be helpful both for letting the player know they've successfully completed a step, or for debugging (as you can e.g. log the current state whenver it changes).

root = Step.Start(() => Player.currentCreature.handRight.HapticTick());

---

Actions, Functions and Lambdas

An aside: if you're not familiar with Action or Func in C#, they are essentially the type of a function.

• An Action takes zero or more arguments and returns nothing
  • Action is an action with no arguments and no return value
  • Action<int> is an action with one int argument and no return value
• A Func takes zero or more arguments and can return something
  • Func<bool> is a function with no arguments that returns a bool
  • Func<string, int> is a function with one string argument that returns an int

These can be represented in a couple ways. You can define a function as normal and reference it like this:

public bool IsEven(int number) {
    if (number % 2 == 0)
        return true;
    return false;
}

public void Start() {
    Func<int, bool> function = IsEven;
    
    Debug.Log(function(1));  // false
    Debug.Log(function(2));  // true
}

Or, you can define an inline function - also known as a "Lambda Function".

public void Start() {
    Func<int, bool> function = (number) => {
        if (number % 2 == 0)
            return true;
        return false;
    };
    
    Debug.Log(function(1));  // false
    Debug.Log(function(2));  // true
}

This can be shortened further:

public void Start() {
    Func<int, bool> function = (number) => number % 2 == 0;
    
    Debug.Log(function(1));  // false
    Debug.Log(function(2));  // true
}

For functions or actions without arguments, the syntax looks like this:

public void Start() {
    Action sayHello = () => Debug.Log("Hello!");
    
    sayHello();  // says Hello!
}

---

Once you've made your root step, you can begin defining your sequence.
This mainly uses two functions: .Then() and .Do().o

There are many overloads of .Then() to make it easy to add in multiple gestures, various parameters, or even sets of sets of gestures. But the most basic looks like this:

public void Start() {
    root = Step.Start();
    root.Then("Condition A", () => conditionA)
        .Do("Action A", () => actionA);
}

The parameters of .Then are as follows:

• name: A human-readable name for the step.
• condition: A Func<bool> that returns true if the condition has been achieved, or false otherwise.

The parameters of .Do are similar:

• name: A human-readable name for the action.
• action: An Action that is triggered when the steps have been completed.

A practical example: let us write the tree for the following sequence involving an Item.

1. Item is held
2. Item is dropped
3. Item is held again

public void Start() {
    root = Step.Start();
    root.Then("Item held", () => item.mainHandler != null)  // held
        .Then("Item dropped", () => item.mainHandler == null)  // not held
        .Then("Item grabbed again", () => item.mainHandler != null)  // held again
        .Do(() => Debug.Log("Gesture complete!"));
}

Note that the name of the steps - the first parameter - is purely for debugging. You can leave it out but I strongly recommend you leave it in. Debugging aside, it helps you to remember which step is which.

Updating a sequence

Once you're done, you must then update the sequence tracker every frame.

public void Update() {
    root.Update();
}

If you don't do this, your sequence will do nothing! Calling Update() tells the tracker to check conditions and traverse its own tree.

Resetting a sequence

The last important note is about root.AtEnd() and root.Reset(). Once a sequence is complete, the tracker will not automatically reset - it will just sit there doing nothing, as there are no more conditions to check. You can test whether the sequence has hit an end point with root.AtEnd(), and you can reset it from the start with root.Reset().

Branching

Let's say we want to do two different things for the previous example depending on whether the creature that picked up the item the second time is a player or an NPC.

We can store the state at the point of being dropped, and then branch off from it depending on what picked it up:

public void Start() {
    root = Step.Start();
    
    var dropped = root
        .Then("Held", () => item.mainHandler != null)
        .Then("Dropped", () => item.mainHandler == null)

    dropped
        .Then("Held by player", () => item.mainHandler?.ragdoll.isPlayer == true)
        .Do("Log player hold", () => Debug.Log("Item held by player!"));

    dropped
        .Then("Held by NPC", () => item.mainHandler?.ragdoll.isPlayer == false)
        .Do("Log NPC hold", () => Debug.Log("Item held by an NPC!"));
}

Debugging

There are a couple tools available to debug. One common pattern I find myself using is this:

public void Start() {
    root = Step.Start(() => Debug.Log(root.GetCurrentPath()));
}

Whenever a step is complete, this will log the current path like this:

Held > Dropped > Held by player

The other useful tool allows you to visualise the entire tree:

Debug.Log(root.DisplayTree());

This will dump the entire tree to the console in a format such as this:

- Held
- Dropped
  - Held by Player
    - Action: Log player hold
  - Held by NPC
    - Action: Log NPC hold

Other Usage

Condition Sets

Sometimes you want multiple steps to be done under one name, or you want to have a function that dynamically generates a set of steps.

This can be done in the following way:

public NamedConditionSet ComplicatedSequence() {
    return Tuple.Create("Name of sequence", new Func<bool>[] {
        () => someConditionA,
        () => someConditionB,
        () => someConditionC,
        () => someConditionD
    });
}

NamedConditionSet is an alias for a tuple of a name and a list of condition functions. You can alias this as such:

using NamedConditionSet = Tuple<string, Func<bool>[]>;
using NamedCondition = Tuple<string, Func<bool>>;

In-sequence Actions

You don't just have to put a .Do() at the end of a sequence! .Do() can be placed at any step in the process, although note that each step can only have one .Do() attached to it.

root = Step.Start();
root.Then(() => ...)
    .Do(() => ...)
    .Then(() => ...)
    .Do(() => ...)
    .Then(() => ...)
    .Do(() => ...);

Combining Steps

Sometimes it is nicer to have complicated steps written out over several lines. You can do this with .And():

root = step.Start();
root.Then(() => conditionA)
    .And(() => conditionB)
    .And(() => conditionC);

This is functionally identical to the following:

root.Start()
    .Then(() => conditionA && conditionB && conditionC);

Delays

You can use .After(duration) to wait a bit before checking the next step.

root = step.Start();
root.Then(() => conditionA)
    .After(0.5f)
    .Then(() => conditionB)
    .Do(() => ...)

Time-related Steps

Sometimes you want to check if a condition is true for more (or less) than a particular duration.

The following two steps check to see whether a button is tapped or held, by checking whether it was released before or after 0.3 seconds.

root = Step.Start()
    
var buttonWasTapped = root
    .Then(() => buttonPressed, "Button Tapped", 0.3f, mode: DurationMode.Before);

var buttonWasHeld = root
    .Then(() => buttonPressed, "Button Held", 0.3f, mode: DurationMode.After);

This works with the endCondition parameter of Then(), which lets you define a custom check for 'condition complete'. Otherwise it defaults to the inverse of the start condition.

---

Gesture Engine

The Gesture Engine is a library for Blade and Sorcery mods that allows you to detect hand gestures.

Note: you must have your C# version set to 'latest' to use this library.

Usage

public class SomeMono : MonoBehaviour {
    Step root;
    public void Start() {
        var gesture = Gesture.Left
            .Palm(Direction.Inwards)    // palm pointing inwards
            .Moving(Direction.Down)     // hand moving downards
            .Point(Direction.Forwards)  // index direction pointing forwards
            .Fist;                      // hand is making a fist
    }

    public void Update() {
        if (gesture.Test()) {
            Debug.Log("Gesture activated!");
        }
    }
}

You can also use it in concert with the Sequence Tracker. A Gesture or GestureStep implicitly converts itself to a format compatible with the Sequence Tracker, including adding in a generated human-readable name for the gesture.

public class SomeMono : MonoBehaviour {
    Step root;
    public void Start() {
        root = Step.Start();
        root.Then(Gesture.Both                   // both hands
                  .Palm(Direction.Up)            // palm upwards
                  .Point(Direction.Outwards)     // point outwards
                  .At(Position.Face)             // positioned near the face
                  .Offset(Direction.Up, 0.3f));  // offset that position upwards by 0.3m
        .Do(() => Debug.Log("Praise the sun!"));
    }

    public void Update() {
        root.Update();
    }
}

A full list of all the checks you can add to your gesture can be found in the code, which is self-documenting.

The Gesture Engine is set up to allow easy handedness options. Changing Gesture.handedness from Side.Right to Side.Left puts it into left-handed mode, which means that Gesture.Left and Gesture.Right will be switched.