WebVRchat

What could VRchat look like with a web client? In what ways can the web and the native program interact with eachother?

Originally published April 2019

tags: devlog janus vrchat

In August 2018 VRchat released an update introducing the Home, a default starting world with a beginner friendly tutorial on their website for people to customize and upload their Homes. Tip: check the comments for a timestamp list.

Homekit

Here is how the default Home looks like when you first open up VRchat.

---

Construction

Software used:

• Unity 2017.4.15f1
• Unity Exporter
• Blender
• FireVR

I first used the Unity Exporter tool to export the geometry from the Unity scene into a folder with JML describing all the positions.

I've now learned of an alternative tool that can relatively achieve the same results. GLTF-Exporter is a Unity3D tool for exporting Unity3D objects to glTF 2.0 format, which blender can import.

Now that the scene was in Blender, custom textures were added by FirefoxG who gave it a southwest flair.

I'm currently working on editing a video game trailer to the webvrchat project. This is a clip I shot from one of the scenes:

---

Merch Shop

The GM3 merch shop is connected to the entire virtual studio but was segmented off to limit the scope of this experiment. I wanted to work with another VRChat creator to collaborate on figuring out the pipeline better from Unity3D -> Blender -> WebVR.

To visit this world, go here: https://vrchat.net/home/launch?worldId=wrld_7621bb7e-883c-495b-9dc6-24d623c925a2

This world is more complicated than the homekit, and the Janus exporter did not really work as planned. Looking around for alternatives, we found some Unity glTF exporter plugins on github and tested a few:

• https://github.com/KhronosGroup/UnityGLTF - Looked promising as its released by Khronos Group, lacks instructions
• https://github.com/sketchfab/UnityGLTF - Seems really promising but always threw an error when exporting
• https://github.com/Plattar/gltf-exporter - The only one that successfully exported geometry

The third option from Plattar worked, mostly. The plugin exported the geometry into a gltf file but the textures were messed up:

This did not turn out to be much of a problem, the files were sufficient enough for my 3D modeling professional friend FirefoxG to retexture and make it even more beautiful than before:

The walls needed decoration. An imgur album containing much of GM3's artwork was created first. Next, I wrote some JML for the glTF file to start decorating the map using the native janus client. Using drag n' drop from imgur into janus and keyboard shortcuts to move things around, this webvr ready scene was created in a day and ready to see in virtual reality.

Next I added some big speakers that 'bounce' to a beat, a video music visualizer, and strobelight into the scene. All that was missing were the avatars! I invited some friends to dance in VRChat with full body tracking and recorded their performance in front of green scren. They danced to original music produced by GM3 which brought the whole scene together.

---

UI / UX

This is what the UI menu looks like in VRchat when you press esc in desktop mode…

And here is a prototype world select developed for Janusweb without looking at VRchat as reference!

---

Avatars

The sbs3d capture of this avatar was done inside of a virtual green screen studio in front of a mirror while OBS recorded from the headset display mirror.

Here's a preview of how that avatar appears in the world, try the demo below if you want to view with a HMD!

Demo

---

Sprites

This system can potentially lead to a streamlined method of converting detailed avatars made in Unity into a spritesheet that can be useful for doom sprite avatars, a format that can be highly cross-compatible with different platforms.

• https://doc.babylonjs.com/babylon101/sprites - Babylon.js documentation for sprites
• https://threejs.org/docs/#api/en/objects/Sprite - Three.js docs on sprites
• https://www.youtube.com/watch?v=oI3T4hxAglk - Simple three.js doom sprite rotation
• https://vrcat.club/threads/retro-sprite-shader.2222/ - Shader for Unity that emulates directional sprites
• https://docs.unrealengine.com/en-US/Engine/Paper2D/Sprites - Unreal engine documentation for creating sprites
• https://docs.unrealengine.com/en-us/UE-Dev-Blog/cid/446515 - Unreal documentation on Octahedral Imposters
• https://forums.highfidelity.com/t/animated-spritesheet-shader-is-now-out/12777 - Spritesheet animation shader for High Fidelit

Here's how something like this looks in-game VRChat:

The 'Avatar Exam Center' is a station built in The Sandbox that takes a picture of your avatar with 11 cameras and generates a doom sprite version spawned into the world.

Watch on Youtube: https://www.youtube.com/watch?v=6e-Hhz0vQe8

Question that I am wondering: If one were to T-pose in that avatar station, would it be possible to export a spritesheet that can streamline the workflow for creating one of these types of avatars?

Apparently, it's a lot of precise setup to get working well right now.

---

Depth Streaming

This avatar was recorded with Anicap. Anicap VR Animation studio lets you use your VR headset as a real motion capture system. It supports full body tracking so you can record your avatar, edit, and use it in VRchat.

Another method to stream with depth is via the Depthfield camera avatar which uses a shader for displaying VRchat in equirectangular projection with a depth map that you can use to stream or record.

Here is what the camera sees when the user is switched into the avatar:

The DepthField world by VoxelKei also features a volumetric video player which can play any mp4 video depthfield camera recording so you can test out how your shots in realtime with others.

It'd be really cool to add these videos in other worlds using something like a crystal ball, like those found in Valve's "The Lab" experience. We could satellite broadcast immersive experiences this way.

---

WebVR depth streaming

If we could stream 360 depth from VRChat to the web browser it would radically improve accessibility and make some of the coolest WebXR content ever! Here are some open source projects I found that could be useful:

• https://juniorxsound.github.io/DepthKit.js/
• https://github.com/ScatterCo/Depthkit.js
• https://vimeo-depth-player-playback.glitch.me/
• https://github.com/vimeo/vimeo-depth-player/

Update: IT WORKS!!

Playing back 360 + Depth video from VRChat using DepthField camera avatars by Voxelkei in the web browser with JanusWeb, a three.js / webgl framework for VR!

Code: https://github.com/janusxr-examples/360-viewer/tree/master/5

---

Threejs Client

Using the metavly API Avaer is able to dynamically decode and render VRChat avatars and worlds in a threejs web client!

https://github.com/webaverse/vrc

First renders

Decode textures with https://github.com/BinomialLLC/crunch

First glimpse of loading Unity VRChat into WebXR

Unity javascript mesh parser: https://gist.github.com/maierfelix/1802f9523dee4090e16dc44a4ac70176