Open Metaverse Framework

tags: devlog

Context

This post is inspired by the recent Meta (formally Facebook) keynote where Mark Zuckerberg went over basic concepts for the Metaverse. If you haven't seen the video I highly recommend it. The video is presented as a vision of the future and can somewhat be interpreted as a schelling point for the direction Zuck is taking his company as well as for collaborators across the industry.

This frame stood out to me as something we could reference when talking about the open Metaverse with.

Watch the full video here: https://www.youtube.com/watch?v=Uvufun6xer8

Overall I think these categories provide a solid framework so I want to elaborate on the open and decentralized solutions we've come up with for each.

---

Presence

Presence is tricking the brain into feeling like you're truly teleported somewhere else. It's the magic of VR.

To achieve this effect, VR technology fundamentally relies on a heavy number of sensors to calculate surroundings, body positions, gestures, and soon things like eye tracking for foveated rendering.

Social is the killer feature of VR, so the real-time data about "you" is also often traveling over the network.

There are two important questions to ask: who is running the presence servers and are there options to run my own?

Lets ignore the first question for now, assuming that most platforms providing the experience run their own presence servers. The second question is exponentially more important and filters down the number of projects to focus on.

Which projects have open source presence servers?

• https://madjin.github.io/janus-guide/#/?id=janus-server
• https://github.com/webaverse
• https://github.com/mozilla/hubs-cloud
• https://github.com/XRFoundation/XREngine
• https://github.com/decentraland
• https://github.com/networked-aframe
• https://github.com/vircadia/vircadia

If we're to filter further down to the apps that include custom full body avatars, a feature most people care about, we get down to just four:

• Webaverse: https://twitter.com/webaverse
• JanusXR: https://twitter.com/janusxr
• XREngine: https://twitter.com/xr_engine
• Vircadia: https://github.com/vircadia/vircadia

Four projects that support custom full body avatars and open source presence servers. This small community has been quietly building everything open source, talking 90%+ purity, and using WebXR also to prevent being gatekept by the app stores. They are the unsung heros of the open and decentralized metaverse.

---

Avatars

Mark Zuckerberg announced that Meta is working on an API to enable avatar interop between apps although we're still waiting on exact technical details. I wonder if they only are referring only to their avatars or custom uploads as well?

Imagine signing up to a social network where you can't upload your own profile pic. Freedom of self expression is a feature of paramount importance to users. The best way to enable that is by enabling custom geometry.

VRChat is the most popular social VR platform for a reason. It's one of the few platforms that allows users to upload their own creations. Lately they have been seeing close to 50k concurrent users on a typical weekend, around 40-50% of which are in VR. On NYE 2021 VRChat was getting close to hitting 100k online users!

Also, what does it mean to truly own your avatar? Can you save-as the files? If the platform controls your access to the avatar, I'd say it's not yours to control.

The best option for the user is having the least amount of friction to import / export avatars between experiences without compromising true ownership.

VRM

Based on my research the number one avatar file format for the open metaverse is VRM. Practically any humanoid character model can be made a VRM avatar.

Quick rundown:

• https://hackmd.io/@XR/avatars
  • https://vrm.dev/en/
  • https://vrm.dev/en/vrm_applications/

“VRM” is a file format for handling 3D humanoid avatar (3D model) data for VR applications. It is based on the royalty free glTF2.0. Anyone is free to use it.

Learn more:

• https://hackmd.io/@XR/avatarlod
  • How to scale number of concurrent avatars per space
• https://hackmd.io/@XR/nftavatars
  • Catalog of 3D NFT avatar projects with guideline tips
• https://github.com/madjin/vrm-samples
  • cc0 samples of avatars and wearables to use in your projects

Avatar Suggestions

There's more work to be done to improve the standard and adoption, especially in the west. We need to be part of the VRM consortium, have active working groups working on metaverse interop, and promote better. Many veteran game devs lack awareness VRM and as such have been recently shooting down notions of interop in online discussions.

• Better content production of avatar interop
  • events / field trips / video editing
• Western participation in VRM consortium
• More adoption across platforms
  • Finish implementation in JanusWeb / Cryptovoxels
  • Initiate support in Somnium Space / Decentraland / Hubs / Open Brush
• Improve VRM blender plugin
  • more testing, atleast get UX parity with univrm
• More webxr ready VRM models
  • 100avatars.com
  • 256fes
  • more variety than anime
  • 0N1 Force / other NFT collections
• Animation interop with VRM
  • gltf-extensions working group
  • collaborations with HEAT
• Design MVP of webxrchat
  • demo with multiplayer avatars, mirror, media player
  • make the source code open source + MIT licensed
• Open source avatar builder
  • Create an alternative to VRoid studio in Blender or webgl
• VRM -> PFP / Spritesheet pipeline
  • https://twitter.com/dankvr/status/1451206110489235457
  • https://hackmd.io/@xr/avatarlod

---

Home Space

The home space is a place you start in when booting up a headset, similar to the desktop environment when your computer operating system loads. This is a place where you can store your favorite digital objects or achievements and change your avatar.

From the Meta announcement video

Part 1: https://youtu.be/Uvufun6xer8?t=230

part 2: https://youtu.be/Uvufun6xer8?t=659

As it pertains to Meta, Mark said in the future creators will be able to make their own home space environments as well.

Every headset and VR platform seems to sport their own version of a home space. Meta and SteamVR both have the abiliy to invite guests over. This is a big deal and huge advantage for Meta since social is a killer feature of VR. Here's a look at the Horizon Home on the Oculus Quest platform:

Mark says that pretty soon you'll be able to hang out, watch videos, and jump into apps together.

Note: Apparently you can change the home space environment for Oculus:

• https://creator.oculus.com/blog/introducing-oculus-home-user-created-spaces/
• https://uploadvr.com/oculus-home-custom-environments/
• https://www.youtube.com/watch?v=lyU-ewSqsEA

Dear Meta employees reading this: can you upload some older home space designs to Sketchfab with a permissive license like cc0 for remixing? Those environments are pretty dope.

Open Source Home Space

Below are links to some open source home space projects with a design aimed to be easy to modify, port, and self-host on your own hardware if you choose to. See https://github.com/madjin/home-space for links to the project files.

• https://github.com/madjin/home-space
  • https://hackmd.io/@XR/hhomekit
• https://github.com/madjin/anime-room
• https://github.com/madjin/tuscany

Demos

• tuscany.webxr.eth
• pod.homespace.eth
• home.hackerlab.eth

Personally I believed that home spaces should more closely resemble the format of something more portable for pioneers exploring the open metaverse frontiers. These are the early wild west days afterall.

For this reason I really like the shipping container design for home spaces. Shipping containers are meant to be portable, and they're becoming a very popular building material in the real world as well so virtual designs can influence physical and vice versa.

I've been porting my home space to various platforms for a number of years, studying what each platform has to offer and how much friction there is to import my belongings into it.

View dev log: https://hackmd.io/@XR/hhomekit

They're also modular and stackable so you can create interesting arrangements with your groups.

Note: Seems like Webaverse will be offering floating island home spaces in the future

Layers of the metaverse

Original link: https://hackmd.io/@XR/privacy#Layers-of-the-Metaverse

Physical layer

"The digital equivalent of owning the land is being close to the hardware metal"

Omitted

https://slate.host/jin/cryptostorage

---

Teleporting / Traversal

This category is centered around the ability to walk from one experience to the next. Meta is going to allow it so that you can hop from your Oculus home space into other apps with your friends. "It's going to be like clicking a link on the internet, it's an open standard," says Mark Zuckerberg.

Hyperlinks mention: https://youtu.be/Uvufun6xer8?t=456

The main mode of transportation in VR at the moment is mostly conducted through 2D menus. Apps are greatly silod from one another so you have to exit one, then wait as you download/load the next.

The isolating moment of delay in between loading worlds is a bit of a jarring experience that I think will get eventually smoothed out. It reminds me of the early internet days when every web page took a bit to load.

Within platforms like VRChat anybody can drop a portal from their menu and lead groups into other worlds. The world selection is limited to those within the platform, so if you're in VRChat you can only explore other VRChat worlds etc.

We have been testing traversal between web based virtual worlds made with different frameworks for atleast 5 years now. The gif below is going from Janus to Aframe then back to Janus.

In 2019 M3 conducted a multiverse field trip where as a group we walked through 7 different platforms using in-world navigation only. This research trip was inspired by a 2014 article on navigating between unrelated virtual worlds.

It's possible to navigate between XR experiences in desktop mode, although at times a bumpy ride. The highlight video from that journey is posted below:

Watch on Youtube: https://www.youtube.com/watch?v=_TID6n4kJ9Q
Link to documentation: https://github.com/M3-org/research#traversing-disparate-virtual-worlds

Imagine if something as fundamental to the Internet as hyperlinks don't work the same way across different web pages. We selectively curated applications that had the ability to open URLs, which isn't many. Each platform has their own custom implementation.

During our journey, some apps opened in the same tab, others in a new tab. This becomes an issue when the old program drains resources in the background without any way to automatically close it.

Metaverse traversal in VR mode has its own challenges and overall needs more testing.

---

Portal Experiments

Recently a M3 OG and colleague of the Open Metaverse Interoperability group Koolala shared an awesome video demonstrating some experiments for moving in between disparate VR experiences. You can see him go from Neos VR to Anyland to Mozilla Hubs, and finally to Gorilla Tag with seamless portals.

Full video with commentary from the dev below

Watch on youtube: https://www.youtube.com/watch?v=g50f6Sjle5o
View blog post: https://nwn.blogs.com/nwn/2022/01/metaverse-portals-neosvr-gorilla-tag-anyland-portal.html

How's it work?

Avaer Kazmer is a metaverse hacker legend that created a web browser able to run spatial apps side-by-side with OpenVR.

The browser is called Metachromium and can be downloaded on Steam.

• https://metachromium.com/
• https://store.steampowered.com/app/685110/Metachromium/

It lets you open any WebXR page (or multiple) as a spatial overlay app that runs on top of any other VR game! Sorta feels like equipping a pair of programmable AR glasses while in VR.

Avaer programmed some demos that are actually able to read the game's buffers and interact with them. For example, VRChat depth clipping and awareness of the world you're currently in which allows for sidechannel game-aware multiplayer.

Learn more:

• https://hackmd.io/@XR/crosslink
• https://hackmd.io/@XR/crosslink2

---

Discovery

See: https://hackmd.io/@xr/maps

portal guns
virtual cities
cyberspace galaxies

---

Interoperablity

There were some insightful nuggets of wisdom about interoperability during the live chat with the VAIL VR team and loomdart that

• Open standards / file formats
  • GLTF
    • https://www.khronos.org/gltf/
    • https://github.com/atteneder/glTFast - Unity plugin
    • https://vrm.dev/en - Humanoid avatars
    • OMI has glTF working group, in contact with khronos / industry
• DeFi / web3 space to be inspired in terms of aligned incentives and composable software
• Metaverse Makers
  • https://github.com/m3-org
  • Virtual events, web3 + webxr community
• Open Metaverse Interoperability group
  • https://twitter.com/open_metaverse
  • https://github.com/omigroup
• https://xrdevlog.com/ - Lotta experiments

https://hackmd.io/@XR/avatars: Overview of interoperable avatar systems

What if it's not the perfect representation of our avatars that traverses between worlds but rather more focus on the metadata / traits in which the world server platform can have the ability to reinterpret such in their own stylized way?

See: https://hackmd.io/@xr/lootwars

---

Privacy and Safety

• https://hackmd.io/@XR/privacy

We need to understand how our data is being handled in the metaverse since it's an absolute gold mine for third party advertisers and behavorial scientists who are seeking to own it.

Network visualizer with force directed graph: https://www.youtube.com/watch?v=1tGqdnUizLE

---

Cultural Forces

We need to protect and cherish anonymity. There's simply no freedom without it.

“Anonymity was one of the major perks of the OASIS. Inside the simulation, no one knew who you really were, unless you wanted them to. Much of the OASIS’s popularity and culture were built around this fact.” - Ready Player One

Achieving widespread privacy by default is a challenge that is going to take more than just technology to solve.

VTubing is a cultural force that originated in Japan, where some of the biggest pop stars are completely synthetic like Hatsune Miku, and is quickly spreading across the rest of the Internet.

Avatars are inherently a privacy enhancing technology, but our voices can be fingerprinted and can compromise privacy.

---

Virtual goods

Zuck alludes to NFT avatar merch: https://youtu.be/Uvufun6xer8?t=767

It's hard to actually do what's being portrayed in the below image, although closed platforms like Roblox have been able to achieve such as described in this blog post.

Update

Prada, Balenciaga and Thom Browne partnered with Meta, more on the way apparently.

MetaFactory

• https://www.metafactory.ai/
• https://slate.host/jin/metafactory

---

Metaverse OS

Meta abandoned their plans, what else is in the works?

Linux VR

https://simulavr.com/

XR desktop (openXR support)

https://gitlab.freedesktop.org/xrdesktop/xrdesktop

https://www.collabora.com/news-and-blog/blog/2021/09/10/an-xrdesktop-summer-of-code/

WebOS

• https://github.com/madjin/startpage
• https://github.com/jnmcfly/awesome-startpage
• https://www.reddit.com/r/startpages/

Anarchy Arcade

If you took your shortcuts from desktop / bookmarks into a 3D world, what would it look like?

AA is a frontend for customizing and launching shortcuts in a 3D world.

Imagine if icons to your files weren't just flat previews but had an extra field for a skinnable 3D model, such as an arcade cabinet, TV monitor, or CD case.

All the 2D stuff is going to come with us into the next dimension.

If you look at someone elses desktop it looks like a bunch of crap, but if you saw your desktop you'd recognize it all - has a personal connection to you.

The metaverse is kind of like the web, the web is like the metaverse just not visualized in a 3D way yet.

https://github.com/M3-org/schedule/tree/master/2019-05-02#sithlord-what-is-a-shortcut

Shorcuts have always been there since dawn of computing.

---

Natural Interfaces

Oculus touch

https://en.wikipedia.org/wiki/Oculus_Touch

ctrl labs armband

---

Lynx

https://lynx-r.com/
https://www.youtube.com/c/LynxMR/videos

Hand tracking in WebXR:

• https://github.com/stewdio/handy.js
• https://hackmd.io/@XR/handtracking

---

MVP: FOSS VRChat clone?

ThrillSeeker made a video following the Meta announcement that I noticed featured a lot of VRChat footage when talking about the kind of metaverse he wants to live in.

It's hard to understate how important freedom of expression is for people. If it doesn't support custom geometry, it's not the metaverse. Open read/write is needed.

Here's a few web based open source projects all have custom avatars, worlds, and presence servers that you can self-host on your own hardware.

• Webaverse: https://twitter.com/webaverse
• JanusXR: https://twitter.com/janusxr
• XREngine: https://twitter.com/xr_engine

---

Multi-apps

Pluto VR

Site: https://www.pluto.app/
Must listen podcast: https://voicesofvr.com/966-overlaying-multiple-layers-of-reality-with-pluto-vr-telepresence-standards-driven-multi-app-ecosystem/

---

Metachromium

Site: http://metachromium.com/
Github: https://github.com/search?q=metachromium

---

Notes

Microsoft also did a presentation

https://www.youtube.com/watch?v=PraEcNDGSqY&t=2457s

Troubling trends brewing in the video game industry: https://twitter.com/dankvr/status/1541971496159121408

I believe there'll be a few dominant layers to the Metaverse:

• Local
  • Offline, your desktop, total freedom sandbox
• P2P
  • Ephemeral spaces, shared with friends / groups
• Global
  • Where majority of the virtual economy takes place

version history = liberating
very cautious without it

---

Interop catalysts:

• clonex space pods got many creators to mint 3D NFTs to decorate home spaces with and similarly
• Worldwide_WEB3 got many NFT projects to create animated sprite versions of their collections