Dev Diary 3: Gesture and Sequence Tracking

I am making a gesture controlled wand, and there's lots to talk about!

This devlog will be more code-heavy than the others, but hopefully it's still enjoyable!

Illustrations done in Figma and Gravity Sketch.

Tracking a Sequence of Things

Each spell requires a series of gestures to perform. For example, the "lift" spell - which allows you to pick up enemies and objects with your wand - requires several steps:

1. A "brandish" gesture - slight flick downwards, then point
2. The wand has to be pointed at an item or enemy
3. The item/enemy becomes targeted, and you flick upwards to lift

So how do you represent this in code? How do you detect multiple paths of gestures?

The Sequence Tracker

I have written a library called the Sequence Tracker. It allows you to register a branching 'tree' of gestures, and add event handlers to each step.

At its most basic usage, the sequence tracker is made of conditions and actions.

root = Step.Start();
root.Then(condition)
    .Do(action);

When condition is true, it will run action (where action is a C# Action type).

root = Step.Start();
root.Then(conditionOne)
    .Then(conditionTwo)
    .Do(action);

Now, conditionOne must be triggered first, then when conditionTwo is triggered it will run action.

You can even put in extra actions in the middle!

root = Step.Start();
root.Then(conditionOne)
    .Do(actionOne)
    .Then(conditionTwo)
    .Do(actionTwo);

For example, this is the full sequence I use for the aforementioned Lift spell:

root = Step.Start();
// Detect the trigger being pressed
root.Then(() => item.mainHandler?.playerHand?.controlHand.usePressed == true)
    // _THEN_ when the wand is 'brandished'
    .Then(Brandish())
    // ...target an entity, setting the `targetEntity` class variable
    .Do(() => TargetEntity(module.targetArgs))
    // _THEN_ when the wand is flicked upwards
    .Then(wand.Flick(AxisDirection.Up, wand.module.gestureVelocityLarge))
    // ...lift the entity specified by `targetEntity`
    .Do(LiftEntity);

Pretty fancy! This links in with the 'spell module' system that is described down below, such that adding a new spell to the wand can be as simple as this:

public class Lift : WandModule {
    public override void OnInit() {
        base.OnInit();
        // wand.targetedItem means that the wand must have first targeted an item
        wand.targetedItem
            .Then("Lifting", wand.Flick(AxisDirection.Up, wand.module.gestureVelocityLarge))
            .Do(LiftEntity);
        
        // wand.targetedEnemy is the same, but for enemies
        wand.targetedEnemy
            .Then("Lifting", wand.Flick(AxisDirection.Up, wand.module.gestureVelocityLarge))
            .Do(LiftEntity);
    }
}

You can also 'branch out' multiple conditions from a single one. For example, the Petrify and Polymorph spell gestures:

• Petrify: Target enemy, then swirl counter-clockwise
• Polymorph: Target enemy, then swirl clockwise

var targetedEnemy = root
    // Trigger pressed
    .Then(() => item.mainHandler?.playerHand?.controlHand.usePressed == true)
    // Wand is 'brandished' to target something
    .Then(Brandish())
    // ...target an entity
    .Do(() => TargetEntity(module.targetArgs))
    // ...and continue only if a creature was targeted
    .Then(() => targetEntity?.creature != null);

// This state is now saved, and can be branched off of.
targetedEnemy
    .Then("Petrify", wand.Swirl(SwirlDirection.CounterClockwise))
    .Do(PetrifyEntity);

targetedEnemy
    .Then("Polymorph", wand.Swirl(SwirlDirection.Clockwise))
    .Do(PolymorphEntity);

As a side note, the "Petrify" and "Polymorph" strings are names given to states to help with debugging.

The 'sequence tree' now looks a bit like this:

You can see and feel this in-game. Every time the tracker transitions from one state to the next, the wand will vibrate in your hand and a little dot will appear on the trail (like the one shown below).

What Even Is a Gesture?

So, we can track sequences of conditions, but the next hard bit is what those conditions should be.

Spaces

The first big concept is the three spaces the wand operates in. When I say that, to trigger a spell, you have to "push the wand forwards", what does "forwards" mean?

From the wand's perspective - the wand's "space" - forwards is stabbing the wand along its axis. From the player's point of view, it's pushing the wand forwards in the direction they are looking. And in Blade and Sorcery there is the concept of a 'global forwards', which is the same no matter which way you are facing.

Wand Space

This is used in gestures such as the lightning zap. You stab the wand forwards, which means that the wand checks to see if its velocity along the 'forward axis' in Wand Space is greater than a certain value.

Player Space

The best example of player space is in the fireball spell. It's intended to be a 'slash' gesture with the wand.

The condition for a fireball is both:

• Low forwards velocity in wand-space - to prevent the gesture from triggering on a 'stab' forwards gesture
• High forwards velocity in player-space

Global Space

Global space is mostly used for gestures that require up or down. Gemini, for example, requires that you target an item, face the tip of the wand downwards and swirl. Grapple requires that you point the wand upwards, then bring it downwards and point.

Twisting and Swirling

These two gestures are far more complex than simple checking a directional swish or flick.

Twisting

For twisting, such as in the Lumos spell, I track the rotation of the wand around its forward axis (in wand-space). The conditions are based on angle - for example, trigger the lumos spell when the wand has been rotated at least 50 degrees.

Swirling

Swirling is hard.

To track a swirl, I need to be able to track the angle that the tip of the wand has travelled around an arbitrary circle in space.

Circles are defined by two things - center, and radius. To test whether a circle is being drawn, I have to have both of these things. Additionally I need to know how 'circular' the circle is - it should be as even as possible, not squshed like an ellipse.

First, I track the position of the wand over time - in an array of vectors. Each frame I add another position into the list. These are represented by the blue lines in the picture above.

Finding the center is easy - just take the average position of the stored points. The radius can be found by getting the average distance each point is from the center. The center and radii are represented by the red dot and yellow lines in the picture above.

For testing circularity, I calculate the average radial deviation - this is a value that represents how much, on average, each points' radius deviates from the above average radius. In a perfect circle each radius would be equal, so the average radial deviation would be equal to zero.

The more circular a shape, the closer that value will be to zero. So I pick a value - 0.02, from memory - and test to see that the deviation is less than that.

Calculating the angle travelled around the circle is done by guessing the circle normal based on the wand's position, and finding the angle between the current radial vector and the one from the last frame.

An Aside: Modularity and Moddable Mods

Lots of my mods are one item or spell with multiple modes, abilities or uses - Shatterblade and the wand are both examples. Making these modes modular really helps with my own development, but it also allows other modders to make addons for these mods.

So: how can we expose a mod such that it can itself be modded? A great implementation to emulate is the one that B&S itself uses - Item Modules!

{
  "$type": "ThunderRoad.ItemData, ThunderRoad",
  "id": "ZiplineWheel01",
  // ...
  "modules": [
    {
      "$type": "ThunderRoad.ItemModuleReturnInInventory, ThunderRoad",
      "returnsWhenNotLookingAtIt": true,
      "returnsAfterReachingMaxDistance": true,
      "maxDistanceBeforeReturning": 20.0,
      "timeAfterWhichItemReturn": 2.0
    }
  ],
  // ...
}

This is part of the JSON catalog definition for the U11 Zipline Wheel. The "modules" key contains a list of item modules. Item modules are classes you can inherit from. In this case, ThunderRoad.ItemModuleReturnInInventory is an item module that ensures an item returns to the inventory if you leave it alone somewhere.

So, I do something similar for my mods - in this case, the wand:

public class ItemModuleWand {
    // ...
    public List<WandModule> spells;
    // ...
}

And in the wand's item JSON:

{
    "$type": "ThunderRoad.ItemData, Assembly-CSharp",
    "id": "Wand",
    // ...
    "modules": [
        {
            "$type": "Wand.ItemModuleWand, WandModule",
            "gestureVelocity": 3,
            "spells": [
                {
                    "$type": "Wand.Lightning, WandModule",
                    "id": "Lightning",
                    "title": "Lightning Arcs",
                    "description": "Trace slicing arcs of lightning through the air.",
                    "iconAddress": "Lyneca.Wand.Tutorial.Icon.Lightning",
                    "videoAddresses": [ "Lyneca.Wand.Tutorial.Video.Arcwire" ],
                    "type": "Button"
                },
                {
                    "$type": "Wand.Spongify, WandModule",
                    "title": "Spongify",
                    "description": "Forces an item to constantly bounce.",
                    "iconAddress": "Lyneca.Wand.Tutorial.Icon.Bounce",
                    "type": "Trigger"
                },
                // ... etc
            ]
        }
    ]
}

That's right - modules within modules! When the wand is loaded, it looks through its spell list and loads all of the spells inside it. That way, modders can use B&S's JSON Array Update system to add their own spells, by adding a new entry to the spells list that references their own code.

This is also helpful here because I can allow modders to very easily add tutorials for their spells to the list, and have them integrated in with all the other spells in the tutorial system!

How do I do that in my mod too?

The list of spells inside an item module will be a single static instance - so make sure you don't simply say wand.spells = module.spells, otherwise each wand you spawn will share the same spell list.

public override void OnItemLoaded(Item item) {
    base.OnItemLoaded(item);
    var wand = item.gameObject.AddComponent<WandBehaviour>();
    wand.module = this;
    
    // For each spell 'template' in our list
    for (var index = 0; index < spells.Count; index++) {
        var spell = spells[index];
        
        // The magic - clone the spell and add it to the wand
        wand.spells.Add(spell.Clone());
    }
    
    // Initialise the wand
    wand.Init();
    
    // Initialise each module
    for (var index = 0; index < spells.Count; index++) {
        var eachModule = spells[index];
        eachModule.Begin(this);
        eachModule.OnInit();
    }
}

The clone method is also simple:

public virtual WandModule Clone() {
    return MemberwiseClone() as WandModule;
}

Although I would recommend making this virtual and overridable, so that inheriting modules can add custom behaviour on cloning (like making sure that list properties get cloned and recreated).

---

The mod is available on my discord, at discord.gg/lyneca. Grab your PC/Nomad role from #roles, read the #rules and download the mod in #open-beta-downloads.

Have fun, and see you next time!