Anata Technical Challenges

• Anata Technical Challenges
  • Shared Male Hair
  • Optimization
  • Base Meshes Male And Female
  • Body + Tattoos
  • Armatures
    • Scaling errors
  • Assembly Pipeline
  • QA
  • Merge Armature C# Code (Close but not working to merge the armatures properly)
• Mesh Transfer Tool Documentation
  • Overview
  • Method Documentation
    • TransformDeepChildExtension Class
    • MeshTransferToolEditor Class
• Fireflies
  • AI meeting summary
  • Action items
  • Outline
  • Notes

---

Shared Male Hair

Still need male shared hair, all the reference pictures are in a 1.5GB zip file containing folders for each: https://wormhole.app/dboap#J85jQIG3FieJuB6A-q_9ig

Note, there's diff styles for same metadata trait name. We just have to go with 1

Optimization

Texture names are very very long, probably need to be done in blender

Base Meshes Male And Female

Finalize Base Mesh UV map, make the male and female normalized 1,1,1 and fix scaling issues at the base.

Body + Tattoos

• Bake body + tattoo + eyes together after getting final base meshes
• We need to create a dictionary that maps the eye color, and skin color trait into the Tatoo trait, so we can use the Tatoo trait to load these new merged textures, since there are no eye triats or skin color traits.

Armatures

Need to apply transforms / scale

Base Mesh Scaling Doesnt Match 1,1,1

There are many Aramtures, and in unity they are coming in like this:

The positions of the master blends are slightly off because of scale, and not aligned to the final base besh

To merge any armatures, into a superArmature, or masterAramture, it needs to be finalized and be the exact same aramture structure, so that the bones can be merged in the right place to the target armature. The "taret armature" is the armature that we are trying to merge with all of the new armatures to make a master armature with the new meshes.

Scaling errors

I tried to normalize the scale and bones but it causes the avatar base mesh to be scaled incorrectly

• Merging armatures (hair/body)
  Need a solution on how to deal with the multiple armatures

Assembly Pipeline

• Making sure metadata matches ingestion

We re-exported the MetaData to match the new trait formatting, but all traits have to match in boom-tools to some object (meaning all items either need to be done, or include an empty placeholder) I got some trait matching errors when trying to text the export.

QA

Merge Armature C# Code (Close but not working to merge the armatures properly)

using System.Collections.Generic;
using UnityEngine;
using UnityEditor;


public static class TransformDeepChildExtension
{
    // Breadth-first search
    public static Transform FindDeepChild(this Transform parent, string name)
    {
        Queue<Transform> queue = new Queue<Transform>();
        queue.Enqueue(parent);
        while (queue.Count > 0)
        {
            Transform current = queue.Dequeue();
            if (current.name == name)
            {
                return current;
            }
            foreach (Transform child in current)
            {
                queue.Enqueue(child);
            }
        }
        return null;
    }
}

[CustomEditor(typeof(MeshTransferTool))]
public class MeshTransferToolEditor : Editor
{
    public Dictionary<string, Transform> masterBoneMap = new Dictionary<string, Transform>();
    public HashSet<string> addedBones = new HashSet<string>();

    void GetBones(Transform pBone)
    {
        foreach (Transform bone in pBone)
        {
            if (!masterBoneMap.ContainsKey(bone.name))
            {
                masterBoneMap[bone.name] = bone;
            }
            GetBones(bone);
        }
    }

    void MergeUniqueBones(Transform targetArmature, Transform otherArmature)
    {
        if (targetArmature == null || otherArmature == null)
        {
            Debug.LogError("Either targetArmature or otherArmature is null. Aborting...");
            return;
        }

        // Here, instead of just looking for "Hips", you can start from the root
        Transform targetRoot = targetArmature.root;
        Transform otherRoot = otherArmature.root;

        // Populate masterBoneMap starting from the root
        GetBones(otherRoot);

        foreach (var kvp in masterBoneMap)
        {
            string boneName = kvp.Key;
            Transform bone = kvp.Value;
            if (bone == null)
            {
                Debug.LogError("A null bone was encountered in masterBoneMap. Skipping...");
                continue;
            }

            if (!addedBones.Contains(boneName))
            {
                // If the bone to be added is a root bone, skip it
                if (bone.parent == otherRoot)
                {
                    continue;
                }

                Transform targetParent = targetRoot.FindDeepChild(bone.parent?.name ?? "");
                if (targetParent == null)
                {
                    Debug.LogError($"Parent bone for {boneName} not found in target armature. Skipping...");
                    continue;
                }

                Transform existingBone = targetParent.Find(boneName);
                if (existingBone == null)
                {
                    Transform newBone = Instantiate(bone, targetParent);
                    newBone.name = boneName;
                    addedBones.Add(boneName);
                }
            }
        }
    }



    void ReparentMeshes(Transform newArmatureRoot, SkinnedMeshRenderer[] meshRenderers)
    {
        foreach (var mesh in meshRenderers)
        {
            if (mesh != null)
            {
                mesh.transform.SetParent(newArmatureRoot);

                // Set the new root bone to match the bone in the new armature
                string oldRootBoneName = mesh.rootBone.name;
                Transform newRootBone = newArmatureRoot.FindDeepChild(oldRootBoneName);
                if (newRootBone != null)
                {
                    mesh.rootBone = newRootBone;
                }
                else
                {
                    Debug.LogError($"Couldn't find new root bone matching {oldRootBoneName}");
                }
            }
        }
    }



    Vector3 CalculateAverageScale(Transform armature1, Transform armature2)
    {
        Vector3 scale1 = armature1.lossyScale;
        Vector3 scale2 = armature2.lossyScale;

        Vector3 averageScale = new Vector3(
            (scale1.x + scale2.x) / 2,
            (scale1.y + scale2.y) / 2,
            (scale1.z + scale2.z) / 2
        );

        return averageScale;
    }

    void RescaleToAverage(Transform armature, Vector3 averageScale)
    {
        if (armature.parent != null)
        {
            Vector3 parentScale = armature.parent.lossyScale;
            armature.localScale = new Vector3(
                averageScale.x / parentScale.x,
                averageScale.y / parentScale.y,
                averageScale.z / parentScale.z
            );
        }
        else
        {
            armature.localScale = averageScale;
        }
    }


    Transform FindCorrespondingBone(string name, Transform armature)
    {
        if (armature.name == name)
        {
            return armature;
        }

        foreach (Transform child in armature)
        {
            Transform result = FindCorrespondingBone(name, child);
            if (result != null)
            {
                return result;
            }
        }

        return null;
    }

    void BoneRemap(SkinnedMeshRenderer mesh)
    {
        Transform[] newBonesList = new Transform[mesh.bones.Length];
        bool isAnyBoneNotMapped = false;

        for (int j = 0; j < mesh.bones.Length; ++j)
        {
            string boneName = mesh.bones[j].name;
            if (masterBoneMap.TryGetValue(boneName, out newBonesList[j]))
            {
                Debug.Log("Successfully mapped bone " + boneName);
            }
            else
            {
                Debug.Log("Unable to map bone \"" + boneName + "\" to target skeleton.");
                isAnyBoneNotMapped = true;
            }
        }

        if (!isAnyBoneNotMapped)
        {
            mesh.bones = newBonesList;

            // Update the root bone
            string rootBoneName = mesh.rootBone.name;
            Transform newRootBone = FindCorrespondingBone(rootBoneName, masterBoneMap[rootBoneName].root);
            if (newRootBone != null)
            {
                mesh.rootBone = newRootBone;
            }

            mesh.localBounds = mesh.GetComponent<Renderer>().bounds;  // Update bounding box
        }
    }

    public override void OnInspectorGUI()
    {
        serializedObject.Update();

        SerializedProperty targetArmatureProperty = serializedObject.FindProperty("TargetArmature");
        SerializedProperty armaturesProperty = serializedObject.FindProperty("Armatures");
        SerializedProperty meshRenderersProperty = serializedObject.FindProperty("Mesh_Renderers");

        EditorGUILayout.PropertyField(targetArmatureProperty, new GUIContent("Target Body Avatar Armature"));
        EditorGUILayout.PropertyField(armaturesProperty, new GUIContent("Armatures to Merge"), true);
        EditorGUILayout.PropertyField(meshRenderersProperty, new GUIContent("Meshes to Transfer"), true);

        GUILayout.Space(10f);

        if (GUILayout.Button("Merge to Master Armature"))
        {
            MeshTransferTool meshTransferTool = (MeshTransferTool)target;
            Transform targetArmature = meshTransferTool.TargetArmature;
            Transform rootObject = targetArmature.parent; // The object one level higher

            // Normalize scale for targetArmature and each otherArmature
            foreach (Transform otherArmature in meshTransferTool.Armatures)
            {
                Vector3 averageScale = CalculateAverageScale(targetArmature, otherArmature);

                // Rescale both armatures to this average scale
                RescaleToAverage(targetArmature, averageScale);
                RescaleToAverage(otherArmature, averageScale);
            }

            addedBones.Clear();
            masterBoneMap.Clear();
            GetBones(targetArmature);  // Populate with target armature bones

            foreach (Transform otherArmature in meshTransferTool.Armatures)
            {
                MergeUniqueBones(targetArmature, otherArmature);
            }

            // Update masterBoneMap again after merging
            masterBoneMap.Clear();
            GetBones(targetArmature);

            ReparentMeshes(rootObject, meshTransferTool.Mesh_Renderers);

            // Remove old armatures
            foreach (Transform oldArmature in meshTransferTool.Armatures)
            {
                DestroyImmediate(oldArmature.gameObject);
            }
        }


        serializedObject.ApplyModifiedProperties();
    }

   
}

Mesh Transfer Tool Documentation

Overview

The script serves to merge multiple armatures (skeletons) into a single "master" armature and transfer associated skinned meshes to the new "master" armature. It is implemented as a custom Unity Editor, providing a GUI to execute these tasks. The overall flow includes:

1. Scale Normalization: Both the target and other armatures are normalized to an average scale.
2. Bone Merging: Unique bones from other armatures are merged into the target (master) armature.
3. Mesh Transfer: The mesh renderers are reparented to the master armature.
4. Bone Mapping: Bones of the transferred meshes are mapped to those of the master armature.
5. Armature Removal: The old armatures are deleted.

---

Method Documentation

TransformDeepChildExtension Class

FindDeepChild(this Transform parent, string name)

• Description: Finds a child transform within a parent transform using breadth-first search (BFS).
• Parameters:
  • parent: The parent Transform object.
  • name: The name of the child Transform to find.
• Returns: The found Transform, or null if not found.

---

MeshTransferToolEditor Class

GetBones(Transform pBone)

• Description: Recursively populates masterBoneMap with all bones in a given armature.
• Parameters:
  • pBone: The parent bone Transform from which to start collecting bones.

MergeUniqueBones(Transform targetArmature, Transform otherArmature)

• Description: Merges unique bones from one armature into another.
• Parameters:
  • targetArmature: The target armature.
  • otherArmature: The armature whose bones are to be merged into targetArmature.

ReparentMeshes(Transform newArmatureRoot, SkinnedMeshRenderer[] meshRenderers)

• Description: Reparents given skinned meshes to a new armature.
• Parameters:
  • newArmatureRoot: The new parent armature Transform.
  • meshRenderers: Array of SkinnedMeshRenderer components.

CalculateAverageScale(Transform armature1, Transform armature2)

• Description: Calculates the average scale between two armatures.
• Parameters:
  • armature1, armature2: The two armatures to consider.
• Returns: Average scale as a Vector3.

RescaleToAverage(Transform armature, Vector3 averageScale)

• Description: Rescales an armature to an average scale.
• Parameters:
  • armature: The armature to rescale.
  • averageScale: The average scale to apply.

FindCorrespondingBone(string name, Transform armature)

• Description: Finds a corresponding bone in a given armature by name.
• Parameters:
  • name: The name of the bone to find.
  • armature: The armature where the bone should be searched for.

BoneRemap(SkinnedMeshRenderer mesh)

• Description: Remaps the bones of a SkinnedMeshRenderer to those in the masterBoneMap.
• Parameters:
  • mesh: The SkinnedMeshRenderer whose bones are to be remapped.

OnInspectorGUI()

• Description: The main GUI logic for the Unity Editor window.
• Actions:
  1. Displays and updates serialized properties.
  2. Provides a button to execute the armature merging and mesh transferring logic.

---

Fireflies

AI meeting summary

The team discussed various technical challenges they are facing. They mentioned issues with shared mail hair and the need to clean up texture names. They also talked about merging base meshes, creating a dictionary for mapping traits, and armature scaling errors. The team explored different approaches for merging armatures and discussed the assembly pipeline. Finally, they briefly touched on a mesh transfer tool and options for handling bones in the master armature.

Action items

Based on the transcript, here are the follow-ups and action items:

• Follow-up: Still working on the shared mail hair.
• Action Item: Need to complete the modeling of the hair.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Action Item: Write a script to clean up the texture names.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Action Item: Create a Reuv map and unpack the female base mesh to make it asymmetrical.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Action Item: Develop a dictionary to map the traits and load the merge texture.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Action Item: Finalize the armature structure for hips, legs, arms, and chest to enable merging of armatures correctly.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Action Item: Address scaling errors and find a solution to handle normalization properly.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Action Item: Test the attachment of hair to the head bone using unibrm.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Follow-up: Resolve humanoid error on model import.
• Action Item: Investigate and resolve the humanoid error during model import.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Follow-up: Resolve humanoid error on model import.
• Follow-up: Armature merging process needs improvement.
• Action Item: Explore alternative methods for merging armatures, such as using cats merging.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Follow-up: Resolve humanoid error on model import.
• Follow-up: Armature merging process needs improvement.
• Follow-up: Consider merging all armatures into one object in Blender.
• Action Item: Explore the option of merging all armatures into one object in Blender to avoid discrepancies.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Follow-up: Resolve humanoid error on model import.
• Follow-up: Armature merging process needs improvement.
• Follow-up: Consider merging all armatures into one object in Blender.
• Follow-up: Strip extra bones after generating the master armature.
• Action Item: Remove unnecessary bones from the master armature after generation.
• Follow-up: Still working on the shared mail hair.
• Follow-up: Texture names are too long.
• Follow-up: Female base mesh needs a Reuv map and unpacking.
• Follow-up: Create a dictionary to map eye color and skin color traits to tattoo traits.
• Follow-up: Armature merging is causing issues.
• Follow-up: Scaling errors are occurring.
• Follow-up: Test attaching the hair to the head bone for proper cascade.
• Follow-up: Resolve humanoid error on model import.
• Follow-up: Armature merging process needs improvement.
• Follow-up: Consider merging all armatures into one object in Blender.
• Follow-up: Strip extra bones after generating the master armature.

These are the follow-ups and action items mentioned in the transcript.

Outline

Based on the transcript, here is an outline with chapters and corresponding timestamps:

• Chapter 1: Introduction and Overview
  • 12:18-12:22: Introduction to the technical challenges
  • 12:23-12:27: Mention of ongoing work on shared mail hair
  • 12:30-12:33: Reference to a large zip file containing folders with reference pictures
  • 12:39-12:45: Discussion about cleaning up texture names and finishing modeling with hair
• Chapter 2: Base Meshes and Tattoo Traits
  • 13:03-13:07: Mention of the need to create a Reuv map and unpacking for the female base mesh
  • 13:23-13:34: Explanation of the process of merging body, tattoos, and eyes into one texture using a dictionary and tattoo traits
• Chapter 3: Armature Challenges
  • 14:00-14:02: Mention of issues with the current armature merging process
  • 14:07-14:35: Discussion about the number of armatures and the need for a finalized structure for correct merging
  • 15:04-15:10: Explanation of the target armature and the need for all armatures to have the same structure
  • 15:35-15:58: Mention of scaling errors and difficulties with merging armatures
• Chapter 4: Hair and Body Merging
  • 16:01-16:11: Proposal to attach hair to the head bone using unibrm
  • 16:24-16:53: Mention of issues with humanoid error on model import and difficulties with the armature merging process
• Chapter 5: Assembly Pipeline and Metadata
  • 16:56-17:20: Discussion about re-exporting metadata and the need for object matching in traits
  • 17:26-17:45: Explanation of the mesh transfer tool and its role in normalizing scales and merging bones
  • 18:01-18:49: Discussion about merging armatures, removing old armatures, and potential solutions for bone merging
• Chapter 6: Conclusion and Technical Difficulties
  • 18:59-19:07: Mention of potential discrepancies on the base mesh and the challenge of merging all bones into one master armature
  • 19:16-19:43: Proposal to use a base mesh master armature with all versions of bones and remove extra bones later
  • 19:52: Conclusion and acknowledgement of technical difficulties

Please note that the chapter titles are based on the content discussed in the transcript and may not fully capture the context or details of each section.

Notes

Sure! Here are some shorthand bullet-point notes summarizing the key points from the transcript:

• Still working on the shared mail hair and need to clean up texture names.
• Need to finish modeling the hair and create a script to clean up texture names.
• Female base mesh needs a UV map and unpacking to merge with tattoos and eyes.
• Need to create a dictionary to map eye and skin color traits to tattoo traits.
• Issues with armatures and scaling causing problems when merging.
• Need to finalize and match the armature structure for merging armatures.
• Scaling errors and normalization process causing issues.
• Planning to attach hair to head bone using Unibrm for proper cascading.
• Troubles with importing model due to humanoid error.
• Issues with armature merging and changing root bone orientation.
• Metadata traits need at least one object to avoid errors.
• Mesh transfer tool normalizes scales and merges unique bones into a target armature.
• Options to merge bones using cats merging or generate bones separately.
• Need to remove old armatures or merge them into one object in Blender.
• Consider creating a master armature with all variations of bones for weighted mesh.
• Strip extra bones after generation to avoid unused bones.

Please note that these bullet points provide a concise summary of the main points discussed in the transcript.