Try   HackMD

Avatar Interoperability

tags: devlog cv vrchat
  • Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    Back to xrdevlog home
  • Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    Part 2: Avatar LOD
  • Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →
    Part 3: NFT 3D avatars

Originally published March 2020

"As Hiro approaches the street, he sees two couples probably using their parent's computer for a double date in the Metaverse. He's not seeing real people, of course. It's all part of a moving illustration created by his computer from specifications coming down the fiber optic cable. These people are pieces of software called avatars. They are the audiovisual bodies that people use to communicate with each other in the Metaverse." - Snow Crash

People seem to own less and less nowadays. From games, movies, and digital assets like skins, these things are more often streamed from the cloud than tangible items in a collection. If we can't atleast have an offline copy of our avatars, do we really own it?

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

I want the things you make or earn in one platform to carry over into the next. One of the most fantasized examples would be the ability to have one's favorite avatar move between different VR experiences. Open standards like glTF and ERC 721/1155 are making such a possibility real.

There are practical reasons why interoperability is an interesting topic. For one, wearables can have more market reach through exposure on other platforms. Second, it makes the digital asset more useful to the collector in the same way that a traveler will want a cell phone that can be unlocked to work in different countries.


VRM

“VRM” is a file format for handling 3D humanoid avatar (3D model) data for VR applications. It is based on the royalty free glTF2.0. Anyone is free to use it.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Links

VRoid

VRoid is a 3D character maker that lets you use your imagination to easily create your own original characters without needing 3D modeling experience. The program is free and the VRM files that export out of it are yours to keep.

Vroid Studio: https://vroid.com/en/studio/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Vroid Hub: https://hub.vroid.com/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

https://hub.vroid.com/characters/8352788162851037719/models/8787943333195890619

Blender plugin: https://github.com/iCyP/VRM_IMPORTER_for_Blender2_8

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

How to add accessories to your VRM file from Blender to Unity

Unity plugin: https://github.com/vrm-c/UniVRM

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Converting to VRM: https://gist.github.com/emilianavt/51d8399987d67544fdebfe2ebd9a5149

Unreal plugin: https://ruyo.github.io/VRM4U/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Babylon.js: https://doc.babylonjs.com/resources/babylonjs_and_vrm

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Exokit avatars: https://avatars.exokit.org/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

NeosVR: https://neosvr.com/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

VRoid -> Neos: https://www.neossearch.com/2022/08/31/最速vroidインポート手順/

VRChat: https://vrchat.com

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Anarchy Arcade: http://anarchyarcade.com/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Cluster: https://cluster.mu

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


Somnium Space: https://somniumspace.com/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


Craftopia: https://store.steampowered.com/app/1307550/Craftopia/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


SPRing Fighter V: https://rc-cobalt.booth.pm/items/1756359

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


Webaverse: https://webaverse.com/

Webaverse is building an interconnected network of virtual worlds and existing community infrastructure to manifest the open Metaverse.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


Substrata: https://substrata.info/

Substrata is a multi-user cyberspace/metaverse. Chat with other users or explore objects and places that other users have created.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


PlutoVR: https://plutovr.com/

PlutoVR is a holographic communications app that's purpose built to work alongside multiple simultaneous running applications.


Kalidoface: https://3d.kalidoface.com/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


Action Drill: https://booth.pm/en/items/1615146

Free to download demo made with 3D Game Kit.

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

https://gyazo.com/cb779743abff5a594188db9013042dc0
https://gyazo.com/0a0194258c5d09eeb1923816cefca972


3D Game Kit: https://hoshimi12.com/?p=20503

Requirements:

Unity3D tutorial on how to replace 3D Game Kit characters with VRoid models.

Note: The links for this seem to of gone offline :(


Incomplete list of applications you can use vroid avatars in: https://hub.vroid.com/apps
Here are apps that can use VRM with: https://vrm.dev/en/vrm_applications/


Cryptovoxels Wearables

Wearables: https://www.cryptovoxels.com/wearables
Wearables config-o-matic: https://www.cryptovoxels.com/hats
Wearables marketplace: https://opensea.io/assets/cryptovoxel-wearables

This experiment is a test on how NFT avatars wearables are ported and treated in other platforms beyond the main client. I used VRChat in this experiment which does not have any blockchain aspect to it whatsoever. Even if the NFT is not honored as a unique digital collectible, it still served a useful purpose as a prop to illustrate a point.

We first begin at the Cryptovoxels wearables config-o-matic which is where a signed-in user can customize their avatar with .vox wearable accessories.

Download every wearable

https://www.cryptovoxels.com/w/5adf72b34c945c23cbb35ce35875fb03d11d5f79/vox - top hat
https://www.cryptovoxels.com/w/1c6504d86afaf0a9cc4fe042f8ab65ffdf7ea4c4/vox - deal with it
https://www.cryptovoxels.com/w/597b5b64b21b0d80cc0a362e25ba96d3ba6a85f0/vox - black tie
https://www.cryptovoxels.com/w/b356afa56bf2541d52d5fe01b8cc552274c40c49/vox - katana
https://www.cryptovoxels.com/w/ecd7d7eff61f8a4807900a09a30c5b96f77fb23f/vox - penguin

Export as obj then import files into Unity, attach to avatar

VRChat

Ported Cryptovoxels avatar in a ported snapshot of Origin City in VRChat!

VRM Cryptovoxels avatar: https://github.com/madjin/cryptovoxels-avatars

Link: https://hub.vroid.com/en/characters/5660465296970164040/models/4325550015911912241

Preview of Cryptovoxels VRM avatar in other programs:

VRM + Cryptovoxels wearables

Code: (coming soon)

NeosVR

coming soon

JanusXR

coming soon


Thoughts

When porting NFTs into other platforms, they will lose their uniqueness properties. Majority of virtual worlds do not have a means of checking or enforcing the provenance of digital assets. The Cryptovoxels wearable ported into VRChat loses its magic and becomes just another 3D model added to your avatar.

Asking the companies developing these other platforms to implement a blockchain solution isn't a very practical strategy for many reasons. We don't need to get very into it, we've been shilling the developers of these platforms about crypto for a long time and listening to their responses. There has to be a better and more permissionless way to implement blockchain / NFTs within a platform with scripting instead of waiting for humans to catch up.

What we need is something like a shazam or Pokedex that can identify a NFT at glance. I believe that cryptoart marketplaces are well positioned to be the first useful data set to train a neural network to detect NFTs with. The MVP could be an app that detects the tokenized art work then displays information about the piece such as name / description / current owner, last sold price.

In the next iteration the same app could be also used for AR animation overlays, similar to what the Artivive app does. If able to port the NFT pokedex to WebXR we could make it into an XR multi-app that can be used in conjuction with any AR or SteamVR program with Metachromium.


LOD Avatars

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Read part 2: Avatar LOD

On New Years Eve 2020 I hung out in VRChat at the Times Square map and celebrated with approximately 39 others in my instance. I recall there were over 5000 people split between all the different instances of this custom designed world made just for new years celebrations. Later that night I went to an underground rave party with a cool crowd of people in real life.

I remember thinking, the biggest difference between the VR and meatspace party is the presence of being in a crowd. There have been hundreds of thousands of visits to that VRChat map but I only ever saw perhaps 100 people tops that entire night.

LOD in game dev means "level of detail", an optimization technique that consists on reducing the number of triangles in an object mesh if the object is far away from the camera.

Imposters

Even further right on the LOD chart are imposters, billboarded flat objects used to represent 3D objects. They're often used when there's large enough distance that the 3D object and image would be indistinguishable.

Maybe imposter avatars can be arranged so that the further away the avatar is, the slower the refresh rate is. For example, if an avatar is standing 20 feet away from you it can only send 1 frame per second or perhaps it is a super low resolution video of the current 'state' the avatar is in.

To get a closer look at what I mean, check out this VRChat world: https://vrchat.com/home/launch?worldId=wrld_a58b67be-a143-4f8a-8177-a2029b14520c

Update: Cryptovoxels is implementing LOD avatars

https://twitter.com/cryptovoxels/status/1242944487422214145

Update: LOD avatar system seen in AltspaceVR during Burning Man event!

Update

Decentraland has implemented avatar imposters as well for scaling the number of concurrent avatars on their unity webgl client, full post here: https://decentraland.org/blog/project-updates/100-avatars-in-a-browser-tab/

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →


CATS Blender Plugin

Website: https://github.com/michaeldegroot/cats-blender-plugin

I think this deserves its own special mention since it does such a good job in taking janky avatars and standardizing them for compatibility with VRChat. The "Fix Model" option is a serious magic wand that fixes all the bones and materials for an avatar super well.

In terms of standardization perhaps a potential reference model could be a y-bot tposing on Mixamo then imported into CATS blender plugin and fixed.


Spritesheets

Spritesheets can be a potential interoperable avatar format. The technique dates all the way back to classic Doom.

What if we can generate something like that, even just to capture the likeness of an avatar. It would be like the virtual equivalent of a 3D body scanner that you sometimes see at studios, conferences, or even print shops.

Creating spritesheets the traditional method is time consuming. Even some of the paid for tools that exist out there have many tedious steps in order to generate sprites from animations or models.

What if there was a more streamlined method of generating a spritesheet from one's avatar, as easy as stepping into a booth and getting a scan like those body scanners at conventions?

This sprite machine is located at The Sandbox Dance Lounge by Splinks (pictured below): https://vrchat.com/home/launch?worldId=wrld_aea5c1d3-7abc-4287-acf8-200279c12700

Perhaps then this sprite can be useful to capture an avatars likeness as a hologram on a "Metaverse Passport". In the future, this data combined with NFT wearables can be useful for a machine learning based provenance tracker. Point the ML augmentation at someones avatar to identify the wearables they're wearing, like shazam for fashion. The privacy implications will need to be thought through with something like this.

One of the limitations with the doom sprite machine is that it's only a static pose. In order to have walk or run animations you would need to simulate your avatar walking in place, as if it is on an omnidirectional treadmill.

Luckily, VRChat has a glitch right now in which you can walk or run while standing in place by pressing the 'Go Home' button on the menu but not selecting any option.

Another way to streamline avatar spritesheet with animations pipeline is by using webglstudio by tamat.

Source: https://twitter.com/tamat/status/1124318984051015680

https://twitter.com/anarchyarcade/status/1248620390223499266


Convert VRM to 2D spritesheet


See avatar LOD dev log: https://hackmd.io/@XR/avatarlod

  1. Import VRM into blender with https://github.com/saturday06/VRM_IMPORTER_for_Blender2_8

  1. Export to .vox model using voxwriter, import into MagicaVoxel by dragging .vox into vox folder in install directory

  1. Export as .obj from Magicavoxel. Zip obj, mtl, and texture file together and upload to Mixamo
  2. Rig the character, find walking animation, check 'walk in place', download animation with skin as collada (.dae) file

  1. Drag .dae into webglstudio then drag texture file also into the window. Go to window -> plugins -> search 'spritesheet' and import it.


Adjust lighting and emissive: https://gyazo.com/90cba48d7df064c7ffe5b82a05072df5

  1. Get rid of viewport UI, adjust model in center. Open window -> Spritesheets and hit generate. You can download the sprite sheet to your hard drive now.


With alpha background: https://i.imgur.com/OLjrxUj.png

This process takes care of the bulk tedious work of pixel art to where you all that's left are detailed touch-ups.


Augmenting Avatars

What if your avatar was a separate program from the main application? This concept is much like the VRoid SDK:

By introducing VRoid SDK, users will be able to use the 3D characters released on VRoid Hub between various services, eliminating the need to have different avatar management systems for each service.

This idea extends to the web as well. Exokit innovated a method to allow a user to run and composite multiple 3D sites at the same time. It does this via Reality Layers, iframes with added ability to do volumetric manipulation. Reality Layers can interact with eachother via postMessage.

Exokit avatars can be integrated into the stack as a reality layer, giving people the ability to bring their avatars in seamlessly. Features include:

  • Loads GLB, FBX, VRChat, VRoid, .unitypackage humanoid, Minecraft skins
  • Auto-detects rigs, height, orientation
  • World scale matching
  • HMD + gamepads input
  • Bones orientation output
  • Arms, legs inverse kinematics
  • Walking kinematics
  • Face animation (visemes) with microphone
  • Hair/clothing animation

This reduces reliance on platform developers to implement their own avatar system so that they can instead focus more on quality gameplay while users get to use their favorite avatars. Win win.


Cryptoavatars

Cryptoavatars is the 'Superrare of avatars' platform where designers can sell 1/1 single edition VRM's on the Ethereum blockchain.

The platform's goal is to help provide a trustworthy proof of ownership solution for avatar creators, so that people can own and keep their valuable digital identity across the Metaverse.

Both VRM and ERC721 are excellent standards and the founder of the project is well known in the space on multiple platforms: VRChat, Decentraland, Cryptovoxels and The Sandbox Game. The team Polygonal Mind that is behind Cryptoavatars created 100 open source avatars for VRChat and other worlds, now open source on Github: https://github.com/polygonalmind/100Avatars.

Prior to launching Cryptoavatars, Polygonal Mind created an HQ in Cryptovoxels, VRChat, and Decentraland. One of the overlooked challenges of interoperability is building relations and we believe this team has set the bar in terms of having skin in each platform and giving back to the community.

Orion

Traveled through space and time to rmain forever on the blockchain. The very first CryptoAvatar ever minted.

The icons on the right side of the card represent the permissions of the file. This is similar to how vroid hub has permissions listed underneath the model:


Here are the platforms that we've been able to import Orion into:

From left to right: VRoid Hub, Unity, and Blender.

From left to right: Cluster, VRChat, NeosVR

From left to right: Somnium Space, Decentraland, VMagicMirror

From left to right: VRAST!, Craftopia, Action Drill

Download

https://hub.vroid.com/en/characters/3525286503410789593/models/7691858106479004081

Orion is CC0 licensed and has a very neutral skin. At first look we thought about how its a perfect base model for remixing, which is why we've made Orion able to be downloaded from vroid hub.

More Thoughts

One may wonder how the record of owning a cryptoavatar matters on platforms that do not have any blockchain integration. This digs right into the issues with how NFT ownership for any type of asset (images, audio, 3D, avatars) works across the multiverse.

Here are a few ideas on how provenance can be honored cross-platform, with or without native integration:

  1. Look up the token on the web to verify who is the owner after learning that it's on the blockchain: https://cryptoavatars.io/gallery

  1. Tag future Cryptoavatars with a discrete watermark such as a hash, QR code, or URL like a clothing tag.

It doesn't have to be big but having a digital barcode or equivalent of tamper proof evidence will help prevent issues of fraud for a multiverse asset.

  1. NFT recognition via computer vision. Perhaps the spritesheets can help train the neural net to recognize a NFT at an angle.

See Thoughts.

BAYC

The founder of cryptoavatars is now working on making 3D VRM versions of all the Bored and Mutant apes in Bored Ape Yacht Club!

Announcement: https://twitter.com/polygonalmind/status/1432081620023001094


Other Platforms

This section is miscellaneous fun stuff about porting / modding.

Super Smash Bros

Super Smash Bros is sorta Metaversey if you think about it. First, it is the ultimate cross-over game in terms of the character selection. Other platforms like Fortnite and Sony wish they could do something like Smash Bros with their IP, Sony has tried and Fortnite continues to do promotions with other companies. Second, the gameplay is more sandbox and free-jazz style than other fighting games like Street Fighter.

The Smash Bros modding scene is still active with new mods and characters uploaded all the time, even for the older Wii title Brawl. This game works with Dolphin emulator as well.

It'd be interesting to have a port of a VROID character into Smash Bros.

1029chris

1029chris is a 3D artist that has been porting their avatar into many platforms. It's a highly manual process but the results are really cool, check it out!

The Witcher

Super Smash Bros

Half Life and GMod

Super Mario 64, Tower Unite, Legend of Zelda: BOTW

Minecraft, Jet Set Radio


End

For the Metaverse, your avatar is a key part of your identity. Consistency is important for people and companies as it relates to branding. Even just having the likeness expressed on different platforms can be enough for most people.

I'd like to experiment more in this area. Some ideas I have are testing with Decentraland avatars and wearables or having a way to check the provenance of a NFT on a non-blockchain VR platform.

Questions or comments? Hit me up on discord (jin#6455) or DM me on twitter.


Platform lock-in sucks, yet it's the way most things are designed. I've dedicated the past 7 years to open Metaverse research and development, the alternative digital dystopia is enough motivation.

I'd like to produce a more refined prototype of an avatar LOD / passport system and video covering the information in this dev log. Here are some ways you can donate NFTs or crypto to support the cause.

Thank you kindly,
jin